- Follow all steps in Upgrading Bash on macOS to update bash (need bash > 4.0)
- Install bashdb: brew install bashdb
- In VS Code:
- Install extension Bash Debug
- Configure bash debugger by https://marketplace.visualstudio.com/items?itemName=rogalmic.bash-debug
- Additional configure: add "terminalKind": "debugConsole" to launch.json to each configuration
Summary of my Development and QA Automation work experiences and list useful links/articles of development, automation regression, and load test etc.
May 10, 2019
Setup Bash Debugger in VS Code
Jun 16, 2018
API Automation Test
Use Katalon
As I said in earlier post, I'm a fan of Katalon Studio which combines UI and API test in one Eclipse based user friendly IDE with valuable Recorder(record/play) feature! Only missing part is no direct support of performance test.
As shown in Create your first API test with Katalon Studio (manual style without scripting) and Katalon Web Service Test (video with a little bit scripting), Katalon supports API test in powerful user friendly way as other popular API test tools e.g. Postman, Restlet Client; By Parameterize a Web Service object and scripting, we could have all kind of APIs calls combinations.
The important point here is Katalon uses Groovy as scripting language, besides enjoy the beauty of Groovy, we have full control of the scripts flow! We could introduce any other external Java libraries into Katalon Studio to leverage all powerful features on Data-driven, assertion etc. integrating with UI/API testing. Even make up some performance test indirectly with coding around 3rd party libraries e.g. REST Assured.
Use long live JMeter
And still love JMeter if load test API after feature test is a must. JMeter is born for load test!
- JMeter Performance and Load Testing @ DZone (the detailed basics of JMeter)
- Parameterize sample tutorial 1 | tutorial 2
- Advanced data driven: JMeter – REST API Testing – A Complete Data-Driven Approach
Jun 15, 2018
Geb / Spock Gatling vs JMeter
Jun 3, 2018
Selenium WebDriver Automation Best Practices (Refer to testautomationguru)
- Page Objects: Page objects is a well known design pattern, widely accepted by the automation engineers, to create separate class file for each page of the application to group all the elements as properties and their behaviors / business functionalities as methods of the class. But it might not be a great idea always, especially when the page has more / different sets of elements / complex element like a grid / calendar widget / a HTML table etc. ex http://www.testautomationguru.com/keyword-driven-framework-for-localization-testing-using-selenium-webdriver/ (Table/keyword driven + page objects model)
- Selenium WebDriver – Blackbox Automated Testing using Arquillian Framework
- Selenium WebDriver – Advanced Page Object Pattern with Page Fragments using Arquillian Graphene (Single Responsibility Principle)
"Note: Your aim should not be to implement a certain pattern in your framework. Instead, identify a problem in the framework and then recognize the pattern which could be used to solve the problem. Do not use any design pattern where it is not really required!!"
- Creational patterns - Factory Pattern (Driver Manager Factory)
- Behavioral patterns - Strategy Design Pattern (Text or Voice search)
- Behavioral patterns - Template Method Pattern (Car and Hotel reservation)
- Structural Patterns - Proxy Pattern (Run selected test on Production env.)
- Structural Patterns - Decorator Design Pattern (Decorator pattern allows us to add additional responsibilities to an object dynamically. By using decorator pattern, we create page objects with dynamic components based different roles. In future, if the application adds another component or another role type, it is extremely easy to update the test.) (Used Java8 Consumer Functional Interface for better readability)
- Behavioral Patterns - Chain Of Responsibility Design Pattern (Whole online shopping process)
- Execute Around Method Pattern (execute around method pattern hides the complex logic of switching to frames / windows etc. The user does not have to worry about switching and instead just focuses on business logic. It makes both page objects and test classes look clean and neat.)
Refer to https://sourcemaking.com/design_patterns or http://www.oodesign.com/ for design patterns.
4. WebDriver in Fluent Style (java 8 Stream)
Selenium WebDriver – How To Design Page Objects In Fluent StyleKatalon Introduction
Recently find free (not open source) automation tool Katalon and love it, personally believe it will be more popular in near future, recommend everyone interested in automation no matter beginner or expert to take a look! (No pay from Katalon for voluntary :D )
See what Wikipedia says about Katalon:
- "The software is built on top of open-source automation frameworks Selenium, Appium with a specialized IDE interface (implemented on Eclipse) for web, mobile and API testing."
- "The main scripting language is Groovy, Java, and JavaScript and can be executed against all modern browsers, iOS, and Android applications supported by Selenium and Appium."
- "Katalon Studio follows the Page Object Model pattern."
- "The remote execution can be triggered by CI systems via Docker container or command line interface (CLI)."
Object recognition support multiple methods to get job done which make scripts more tolerate to AUT changes, no as good as IBM RFT's but good enough.
For Selenium 1.0 IDE lover, Katalon Automation Recorder Chrome extension is the latest replacement.
Well organised Official Katalon site has all necessary information (docs, forum etc.)
There is a free Udemy course Automate everything with Katalon Studio.
Aug 31, 2010
IBM RFT: how to invoke object methods
IBM RFT: Enable Java Domain for Java AUT
Normally IBM RFT just regards all objects in Win domain without respect that the AUT is a Java application.It's fine normally, but for some special objects (e.g. grid) against which you'd like call the special methods for your test purpose, you could enable RFT to treat objects in Java domain:
- In RFT, 'Configure->Enable Environments for Testing...';
- In tab 'Eclipse Platforms', 'Add...' browse to where your Java application home_dir (with private jre);
- 'Enable' it, then 'Apply', 'Finish' to close 'Enable Environments' window.
Aug 21, 2007
JMeter HTTPClient HTTP sample might be the reason of jmeter crash
2007/08/20 16:05:35 INFO - org.apache.commons.httpclient.HttpMethodDirector: Redirect requested but followRedirects is disabled
In this case have to change back to simple HTTP sample:(
Jul 26, 2007
MS-DOS Batch Script Tips
rem setlocal not to impact environment variables
setlocal
rem get current disk driver and directory in case needed resources in specific location
set workDriver=%~d0
set workPath=%~p0
%workDriver%
cd %workPath%
rem call some command/executed program here
rem then check ERRORLEVEL for error handling
if ERRORLEVEL 1 (set errorCode=1
rem change color for warning and fun :)
color 4e
echo Error encounter!
pause
color 0f
goto error)
Useful link:
Batch File Command Reference for Windows 2000
MS-DOS/MSDOS Batch Files: Batch File Tutorial and Reference
Jul 12, 2007
Jmeter: Summary Report vs Aggregate Report
Warning in Aggregate Report help: "Calculation of the Median and 90% Line values requires a lot of memory as details of every Sample have to be saved. See the Summary Report for a similar Listener that does not need so much memory."
So use Aggregate Report only when you care Median and 90% Line values. But what Median, do you really care it, good explanation is here.
Jul 11, 2007
JMeter: BeanShell Example
Script:
String amountString = vars.get("_tmpAmount");
int amount = Integer.valueOf(amountString).intValue();
int minAmount = 10;
if (amount > minAmount) {
vars.put("_finalAmount", Integer.toString(minAmount));
} else {
Failure = true;
ErrorMessage = "Must has at least " + minAmount + " in original account!";
SampleResult.setDataType(SampleResult.TEXT);
SampleResult.setResponseMessage(ErrorMessage);
SampleResult.setResponseData(ErrorMessage.getBytes());
SampleResult.setStopThread(true);
}
Jul 10, 2007
JMeter: Dynamic SOAP XML data of WebService(SOAP) Request
a). "Soap/XML-RPC Data" text field directly or
b). reading from file or
c). randomly picking up from directory when data files stored.
When want to send dynamic soap xml data, say different user name, method a) doesn't work since it's hard coded; method c) works here as long as we generate the data files in advance.
How about method b)? It looks more neat, at least for me:) But how?
The way I could think about is using a Java Request before each WebService(SOAP) Request. In the Java Request, just use simple BufferedWriter writer = new BufferedWriter(new FileWriter("soapDataFile")) to generate the soap xml file which includes the dynamic user name variable read from user file. Since the xml is not complex, just regard it as text file, no bother to use the more professional heavy XML processing method like How to Read and Write RSS Feeds (Linked here just in case)
JMeter: BeanShell Usage
Normally "Regular Expression Extractor" could get expected value from sampler response;
Otherwise we might need to use __BeanShell(), __javaScript() (examples) with the help of JMeter's "Options | Function Help Dialog";
For more complex cases, we might have to use "BeanShell Assertion"; How to debug it? Refer to BeanShell Manual and try it in BeanShell Desktop window before copy it to your "BeanShell Assertion". Where to start BeanShell Desktop? Download it from http://www.beanshell.org/download.html and save it to your jmeter_home/lib, just double-click bsh*.jar and play. Enjoy it:)
JMeter: HTTP Proxy Server doesn't support HTTPSampler2
But unfortunately, we will got "ERROR - jmeter.protocol.http.proxy.Proxy: java.lang.NullPointerException" in jmeter.log and gateway timeout in browser. By checking http://article.gmane.org/gmane.comp.jakarta.jmeter.user/14834/match=jmeter+protocol+http+proxy+java+lang+nullpointerexception+httpsampler2 we could know, it's still not supported:(
Jul 9, 2007
OCR in TestComplete
OCR which might be the last choice for identify object/text in screen in TestComplete.
Open OCR sample at,
C:\Program Files\Automated QA\TestComplete 5\Samples\Scripts\OCR\OCR.pjs
Just run it.
There are 2 parts of code, but below is the key point,
TextToFind = "E-mail";
Pic = wOfficeListBox.Picture(0, 0, wOfficeListBox.Width, wOfficeListBox.Height, false);
OCRObjList = OCR.CreateObject(Pic);
FindRes = OCRObjList.FindRectByText(TextToFind);
The only part you need to replace is Pic = wOfficeListBox.Picture(0, 0, wOfficeListBox.Width, wOfficeListBox.Height, false);
You could use the way in this sample if you could get the object, otherwise you could always use Sys.Desktop.ActiveWindow() to get the window snapshot and specify the coordinates of the area you care, and do the OCR, the recognition speed is good.
And check TestComplete Help for topic “Recognition Tips” to know which area is better for your case.
TestComplete Notes: big page handling & memory saving tips
1. Method to identify big page
var childNum = doc.ChildCount;
if (childNum > 13000 || childNum == 0) {
Log.Warning("Can not handle too large page!");
return null;
}
2. Release no use memory for browser and TestComplete
Based on information at,
minimize then maximize window could force process to release the no used memory, code for IE is as below:
// to release memory of browser
var browerWindow = windowPage.Parent;
while (! BuiltIn.IsSupported(browerWindow, "Minimize")) {
browerWindow = browerWindow.Parent;
}
browerWindow.Minimize();
browerWindow.Restore();
// to release memory of TC
Sys.Process("TestComplete").Window("TfrmTCMainForm", "*", 1).Activate();
Sys.Process("TestComplete").Window("TfrmTCMainForm", "*", 1).Minimize();
Sys.Process("TestComplete").Window("TfrmTCMainForm", "*", 1).Maximize();
you could do above at your threshold by checking Sys.Process("TestComplete").MemUsage
But for FireFox, to release the memory, we are not lucky. We have to use task manager to minimize and maximize it. In addition, we also have to turn on MSAA and hard code like below,
function _releaseFFMemory()
{
var p1;
var w1;
var w2;
p1 = Sys.Process("Explorer");
p1.Window("Shell_TrayWnd").Window("Button", "start").btn_Start.Click();
p1.Window("DV2ControlHost", "Start Menu").Window("DesktopSFTBarHost", "", 2).Window("SysListView32").mi_Run.Click();
p1.Window("#32770", "Run").Window("ComboBox").Window("Edit").Keys("taskmgr[Enter]");
p1 = Sys.Process("taskmgr");
w1 = p1.Window("#32770", "Windows Task Manager");
w2 = w1.Window("SysTabControl32", "Tab1");
w2.ClickTab("Applications");
w2 = w1.Window("#32770", "", 1).Window("SysListView32", "Tasks").list_item_Bookie_Home_Page_Mozilla_Firefox;
w2.ClickR();
p1.Window("#32768").mi_Minimize.Click();
w2.ClickR();
p1.Window("#32768").mi_Maximize.Click();
w1.title_bar.btn_Close.Click();
}
3. Use as little memory as possible
Tips from the same message board and my experiences,
1) Log message only when needed;
2) Turn on auto-save log: Options | Engines | Log and set the auto-save interval. This will flush the log to disk every x number of minutes. (better performance, not sure if help reduce memory usage)
3) Store screenshots in PNG format instead of BMP format.
4) Call the Log.LockEvents method to prevent TestComplete from posting event messages; And call Log.UnlockEvents when you need it. If you never care event messages even when you got warning or error, use Log.LockEvents(0).
Jul 4, 2007
JMeter: Recommend using HTTP Request HTTPClient
By checking HTTPSampler2 source code which is for HTTPClient HttpSampler in package org.apache.jmeter.protocol.http.sampler, we could find more checks/controls than normal HttpSampler in same package. Even we don’t care the detailsJ
Conclusion: use HTTPClient HttpSampler but not normal one. They have the exact same GUI.
Coding/Debuging Hessian Service with JMeter in Eclipse
* Eclipse SDK
* JMeter
* Java Hessian library
* The testing service's interface(s) and related classes (such as exceptions, DTOs etc)
2. Write the Java Request Sample driver code
* Create a class that extends org.apache.jmeter.protocol.java.sampler.AbstractJavaSamplerClient
There are 4 methods that need to override:
1) getDefaultParameters()
This method is where you set parameters that you would like to get in JMeter Java Reqeust GUI. You do this by instantiating a org.apache.jmeter.config.Arguments and calling its addArgument method for each parameter. For example:
public Arguments getDefaultParameters() {
Arguments args = new Arguments();
args.addArgument("Service URL", "");
args.addArgument("User Name", "");
args.addArgument("Password", "");
return args;
}
2) setupTest()
Where you do any initialization (only once per thread) such as reading in the parameters, creating the Hessian proxy, etc. For example:
public void setupTest(JavaSamplerContext context) {
url = context.getParameter("Service URL");
try {
mySvc = (mySvcClassName)
proxyFactory.create(mySvcClassName.class, url);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException("Error getting hessian proxy", e);
}
}
3) runTest()
JMeter will invoke it for each iteration of the test. This is where you would put code that invokes the service's methods and evaluate its result. The method returns a "org.apache.jmeter.samplers.SampleResult" object which is what you use to communicate the results to JMeter. For example:
public SampleResult runTest(JavaSamplerContext context) {
SampleResult result = new SampleResult();
result.setSampleLabel("Call Hessian SVC");
result.setDataType(SampleResult.TEXT);
result.sampleStart();
try {
String custId = mySvc.authenticate(context.getParameter("User Name"), context.getParameter("Password"));
if (custId == null) {
result.setSuccessful(false);
result.setResponseMessage("user authentication failed");
result.setResponseData("Returned customer id is null".getBytes());
}
else {
result.setSuccessful(true);
result.setResponseCodeOK();
result.setResponseMessage("Authentication successful");
result.setResponseData(("customer id: " + custId).getBytes());
}
}
catch (Throwable e) {
StringWriter sw = new StringWriter();
e.printStackTrace(new PrintWriter(sw));
result.setSuccessful(false);
result.setResponseMessage("Unexpected exception");
result.setResponseData(sw.toString().getBytes());
}
result.sampleEnd();
return result;
}
In the example above, the method setSuccessful indicates if the test passed or failed. The setting of the responseData is used for displaying whatever information you want in the response data tab of JMeter's "View Results Tree" listener.
4) teardownTest()
Where you do cleanup. Typically there is nothing you need to do here.
3. Create/Copy needed jars to JMeter lib
Create ("Export | Java | Jar file") a jar file that includes the class you wrote above and together with jar(s) of the interface (and related classes) of the service you want to test. Put these jars file in your JMeter's lib/ext folder.
And copy the Hessian jar file and any other helper jar files (eg. database jars) you need under JMeter's lib or lib/ext folder.
4. Debugging Java Request Sample driver code in Eclipse
1) Add external JMeter jar ApacheJMeter_core.jar, ApacheJMeter_java.jar under jmeter_home/lib/ext/ and hessian-3.0.17.jar to current project build path; And add other jars as needed;
2) Browse into ApacheJMeter_core.jar: "org.apache.jmeter | NewDriver.class", right-click and select "Debug As | Debug...", on right panel,
a) In tab "Arguments" change "Working directory" from default to jmeter_home/bin/;
b) In tab "Classpath | User Entries", remove default classpath, and add "Add External JARs..." jmeter_home/bin/ApacheJMeter.jar;
c) "Apply" to save the setting
3) Click Debug/Run on Eclipse toolbar to debug/run JMeter from Eclipse
4) When "Source not found", click "Edit Source Lookup Path... | Add... | Java Project", select the project which contain your code
Additionally: Want to see JMeter code during debug, just download the source from http://apache.osuosl.org/jakarta/jmeter/source/, unzip it and add source of the jmeter source directory and check true of "Search subfolders" :-)
5. Run in JMeter without Eclipse
* Copy the service api and test JARs from above to JMeter's lib or lib/ext
* Launch JMeter
* Add a "Thread Group" under "Test Plan"
* "Add >> Sampler >> Java Request" to the Thread Group
* On the right-hand Java Request panel, select your class in the drop-down java classname list
* In "Send Parameters with Request", supply the appropriate values for the parameters you've specified in the "getDefaultParameters" method of your class
* "Add >> Listener >> Summary Report" to the Thread Group
* "Add >> Listener >> View Results Tree" to the Thread Group
* "Run >> Start".
* Select the Summary Report node in the test plan to see statistics on the run.
* Select the View Results Tree node to see the result of each execution. This is where you'll see any data you've set in the responseData within your runTest method. In above example: customer id.
Jun 12, 2007
All About JMeter
Useful Links:
Apache JMeter Wiki
JMeter Tutorial@InformIT:
# Recording Web Requests Using the HTTP Proxy Server
# ExpandAdvanced Proxy Server Configuration
# ExpandTesting with Dynamic Data
# ExpandDistributed Load Testing
Nabble - JMeter Forum
JMeter User Group
Tips:
在 JMeter 压力测试工具中使用变量
在 JMeter 压力测试工具中使用函数(Function)
使用 JMeter 完成常用的压力测试
改善Jmeter脚本质量