Popular API/UI automation tools/frameworks (2023)

(copied some content from chat.openai.com :D)

API automation:

• Postman(JS)/Newman: Postman is a widely used API development and testing tool that provides a user-friendly interface for designing, executing, and automating API tests. It offers features like test scripting, environment management, test data management, and integrations with other tools.
• RestAssured(Java): RestAssured is a Java library for testing RESTful APIs. RestAssured supports all REST methods. It's widely used to test JSON and XML-based web applications. It provides a domain-specific language (DSL) for writing powerful, maintainable tests for RESTful APIs. But consider Karate vs REST-assured for a new project.
• SoapUI vs ReadyAPI: SoapUI is an open-source API testing tool specifically designed for testing SOAP and RESTful web services. It offers a comprehensive set of features for creating and executing functional, security, and performance tests.
• JMeter(Java): Apache JMeter is primarily known for performance testing but also supports functional API testing. It can simulate high loads and stress test APIs, making it suitable for both functional and performance testing scenarios.

UI automation:

• Selenium(Java, Python, C#, Ruby, JS, Kotlin): Selenium is an open-source automation framework widely used for automating web browsers and web application testing. The Selenium WebDriver is the core component of Selenium that provides a programming interface to interact with web browsers. It allows you to locate and manipulate web elements, handle browser navigation, and perform various actions required for testing.
• Appium(JS, Python, Java, Ruby): Appium is a test automation tool for Android and iOS apps. Appium leverages the WebDriver protocol and shares a similar architecture with Selenium. It allows for code reuse and knowledge transfer between web and mobile automation.

API & UI: 

• Playwright(Node.js, Python, Java, .NET): Playwright is an open-source JavaScript-based automation framework that enables developers and testers to automate web browser interactions across multiple browsers. It provides a high-level, user-friendly API that simplifies the process of browser automation. Playwright provides a JavaScript API for making HTTP requests, enabling you to send API requests, handle responses, and perform assertions. Playwright’s multi-browser support and cross-platform compatibility make it a versatile choice for API testing in a web context.
• Cypress(JS): Cypress is a modern front-end testing tool built using JavaScript and focuses on making testing easier for developers and QA engineers. It uses a unique DOM manipulation technique and runs directly in the browser, providing a more developer-friendly approach to testing applications. It can also be used for API testing using its cy.request() command.

Macro Automation Framework: 

• Karate(Java): Karate is an open-source API testing tool built on top of Cucumber and Gatling, a unified test automation platform combining API testing, API performance testing, API mocks & UI testing.
• Robot Framework (Python): Robot Framework is an open-source test automation framework that is widely used for acceptance testing, robotic process automation (RPA), and acceptance test-driven development (ATDD). It provides a simple, readable syntax that enables easy test case creation and maintenance.
• CodeceptJS(JS): CodeceptJS is an open-source JavaScript-based end-to-end testing framework that simplifies the process of writing and executing functional tests for web applications. It provides a high-level, human-readable API that abstracts away the complexities of underlying test automation tools like Selenium WebDriver or Puppeteer. CodeceptJS uses Helper modules to provide actions to I object. Currently, CodeceptJS has helpers for web testing(Playwright, WebDriver, Puppeteer, Protractor, TestCafe, Nightmare), mobile testing(Appium, Detox), API testing(REST, GraphQL) and AI(OpenAI).

Use puppeteer with waitUntil.networkidle0 to get fully loaded page

To solve 'Is there a way to let cURL wait until the page's dynamic updates are done?', here is the code to solve it after googlings:

const puppeteer = require('/usr/local/lib/node_modules/puppeteer');

const fs = require('fs');

function sleep(ms) {

  return new Promise(resolve => setTimeout(resolve, ms));

}

async function getContent(url, outputFile) {

  const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});

  const page = await browser.newPage();

  retries = 0

  while (retries < 3) {

    try {

      const response = await page.goto(url, {waitUntil: 'networkidle0'});

      const status = response._status

    

      if (status == '200') {

        const html = await page.content();

        fs.writeFile(outputFile, html, function (err) {

          if (err) throw err;

        });

        console.info('Success');

        break;

      } else {

        console.error('Fail to load [' + url + '] w/ status code: ' + status);

      }

    } catch (error) {

      console.error('Exception: ' + error)

    }

    console.info('Retrying ' + ++retries);

    await sleep(3000)

  }

  await browser.close();

};

const myArgs = process.argv.slice(2);

const url = myArgs[0]

const outputFile = myArgs[1]

getContent(url, outputFile);

Supercharged End 2 End Testing with CodeceptJS

IMO best End 2 End testing tool till now: CodeceptJS

Resource Links:

codeceptjs | CodeceptJS@github | CodeceptJS Doc | codeceptjs@npmjs

CodeceptJS Community | CodeceptJS@slack | codeceptjs@stackoverflow


Presentation/Tutorials:

Michael Bodnarchuk: EFFECTIVE END TO END TESTING WITH CODECEPTJS | video

Intro to BDD style E2E testing with CodeceptJS, Docker Compose and Semaphore CI

Visual Testing Powered by CodeceptJS & Resemble.js

End-To-End Testing With CodeceptJS (Registration test w/ codeceptjs-tempmail)

codeceptjs jenkins integration

Running End to End tests as Google Cloud Functions

Daniel's CodeceptJS notes

Proof of Concept: Using CodeceptJS Instead of Python

Pros:

• Support web and mobile;
• Better locator
• Parallel Execution
• Better report (allure), logging (–steps --verbose --debug)
• Interactive debugging (shell, but no VS Code step-by-step debug)
• Active development

Daily cmds:

• npm install codeceptjs --save; npx codeceptjs init; npx codeceptjs def .; npx codeceptjs [gt|gpo|go|gh]
• npm install -g selenium-standalone; selenium-standalone install; selenium-standalone start or use codeceptjs-selenium
• npx codeceptjs run --steps
• npx codeceptjs run -o '{"tests":"./account_test/example_test.js"}' # run test in non-config path
• allure serve output
• For signup: codeceptjs-maildev-helper

JSON Tools - jq

Comparing with Python json.tool, seems jq is much light weight and powerful :)

Sample time diff:

With jq:
real 0m1.666s
user 0m0.262s
sys 0m0.290s 

With Python json.tool:
real 0m2.217s
user 0m0.492s
sys 0m0.539s


Useful links:

• Comparing Command-Line JSON Pretty Printers
• jq is sed for JSON
• jq play

HTML parsing with Python

Scraping Data with Python and XPath:

Sample code from reference link which tell the whole story :)

import requests
from lxml import html

pageContent=requests.get('https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo')
tree = html.fromstring(pageContent.content)

goldWinners=tree.xpath('//*[@id="mw-content-text"]/table/tr/td[2]/a[1]/text()')
silverWinners=tree.xpath('//*[@id="mw-content-text"]/table/tr/td[3]/a[1]/text()')
#bronzeWinner we need rows where there's no rowspan - note XPath
bronzeWinners=tree.xpath('//*[@id="mw-content-text"]/table/tr/td[not(@rowspan=2)]/a[1]/text()')
medalWinners=goldWinners+silverWinners+bronzeWinners

medalTotals={}
for name in medalWinners:
    if medalTotals.has_key(name):
        medalTotals[name]=medalTotals[name]+1
    else:
        medalTotals[name]=1

for result in sorted(
        medalTotals.items(), key=lambda x:x[1],reverse=True):
        print '%s:%s' % result

BeautifulSoup is another option but different style from xpath.

HTML page parsing with xmllint xpath in BASH

Per HTML_parsers, there is no better HTML page parsing options for BASH. Inspired by Retrieve web using xpath, here comes the summary of using xmllint xpath:


xpath='' # sample: '//div[@class = "tides"]'

get_element_by_xpath():
    echo $HTML_PAGE | xmllint --html --xpath $xpath - 2>/dev/null

get_element_text_by_xpath():
    xpath+='/text()'
    echo $HTML_PAGE | xmllint --html --xpath $xpath - 2>/dev/null

get_elements_count_by_xpath():
    xpath="count($xpath)"
    echo $HTML_PAGE | xmllint --html --xpath $xpath - 2>/dev/null

Setup Bash Debugger in VS Code

Recent projects need some bash scripting, besides bashdb in terminal, GUI bash debugging make life much easier:

1. Follow all steps in Upgrading Bash on macOS to update bash (need bash > 4.0)
2. Install bashdb: brew install bashdb
3. In VS Code:
   1. Install extension Bash Debug
   2. Configure bash debugger by https://marketplace.visualstudio.com/items?itemName=rogalmic.bash-debug
   3. Additional configure: add "terminalKind": "debugConsole" to launch.json to each configuration

API Automation Test

As per Top 10 API Testing Tools, API Test becomes a buzz word now because many companies nowadays require strong API performance and shifts towards APIs architecture, hence this summary post of API test.

Use Katalon

As I said in earlier post, I'm a fan of Katalon Studio which combines UI and API test in one Eclipse based user friendly IDE with valuable Recorder(record/play) feature! Only missing part is no direct support of performance test.

As shown in Create your first API test with Katalon Studio (manual style without scripting) and Katalon Web Service Test (video with a little bit scripting), Katalon supports API test in powerful user friendly way as other popular API test tools e.g. Postman, Restlet Client; By Parameterize a Web Service object and scripting, we could have all kind of APIs calls combinations.

The important point here is Katalon uses Groovy as scripting language, besides enjoy the beauty of Groovy, we have full control of the scripts flow! We could introduce any other external Java libraries into Katalon Studio to leverage all powerful features on Data-driven, assertion etc. integrating with UI/API testing. Even make up some performance test indirectly with coding around 3rd party libraries e.g. REST Assured.

Use long live JMeter

And still love JMeter if load test API after feature test is a must. JMeter is born for load test!

• JMeter Performance and Load Testing @ DZone (the detailed basics of JMeter)
• Parameterize sample tutorial 1 | tutorial 2
• Advanced data driven: JMeter – REST API Testing – A Complete Data-Driven Approach

Pure Coding style using REST Assured

With REST Assured along come out home brew BDD style API test! Recommend tutorial.