Use puppeteer with waitUntil.networkidle0 to get fully loaded page

To solve 'Is there a way to let cURL wait until the page's dynamic updates are done?', here is the code to solve it after googlings:

const puppeteer = require('/usr/local/lib/node_modules/puppeteer');

const fs = require('fs');

function sleep(ms) {

  return new Promise(resolve => setTimeout(resolve, ms));

}

async function getContent(url, outputFile) {

  const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});

  const page = await browser.newPage();

  retries = 0

  while (retries < 3) {

    try {

      const response = await page.goto(url, {waitUntil: 'networkidle0'});

      const status = response._status

    

      if (status == '200') {

        const html = await page.content();

        fs.writeFile(outputFile, html, function (err) {

          if (err) throw err;

        });

        console.info('Success');

        break;

      } else {

        console.error('Fail to load [' + url + '] w/ status code: ' + status);

      }

    } catch (error) {

      console.error('Exception: ' + error)

    }

    console.info('Retrying ' + ++retries);

    await sleep(3000)

  }

  await browser.close();

};

const myArgs = process.argv.slice(2);

const url = myArgs[0]

const outputFile = myArgs[1]

getContent(url, outputFile);

Supercharged End 2 End Testing with CodeceptJS

IMO best End 2 End testing tool till now: CodeceptJS

Resource Links:

codeceptjs | CodeceptJS@github | CodeceptJS Doc | codeceptjs@npmjs

CodeceptJS Community | CodeceptJS@slack | codeceptjs@stackoverflow


Presentation/Tutorials:

Michael Bodnarchuk: EFFECTIVE END TO END TESTING WITH CODECEPTJS | video

Intro to BDD style E2E testing with CodeceptJS, Docker Compose and Semaphore CI

Visual Testing Powered by CodeceptJS & Resemble.js

End-To-End Testing With CodeceptJS (Registration test w/ codeceptjs-tempmail)

codeceptjs jenkins integration

Running End to End tests as Google Cloud Functions

Daniel's CodeceptJS notes

Proof of Concept: Using CodeceptJS Instead of Python

Pros:

• Support web and mobile;
• Better locator
• Parallel Execution
• Better report (allure), logging (–steps --verbose --debug)
• Interactive debugging (shell, but no VS Code step-by-step debug)
• Active development

Daily cmds:

• npm install codeceptjs --save; npx codeceptjs init; npx codeceptjs def .; npx codeceptjs [gt|gpo|go|gh]
• npm install -g selenium-standalone; selenium-standalone install; selenium-standalone start or use codeceptjs-selenium
• npx codeceptjs run --steps
• npx codeceptjs run -o '{"tests":"./account_test/example_test.js"}' # run test in non-config path
• allure serve output
• For signup: codeceptjs-maildev-helper

JSON Tools - jq

Comparing with Python json.tool, seems jq is much light weight and powerful :)

Sample time diff:

With jq:
real 0m1.666s
user 0m0.262s
sys 0m0.290s 

With Python json.tool:
real 0m2.217s
user 0m0.492s
sys 0m0.539s


Useful links:

• Comparing Command-Line JSON Pretty Printers
• jq is sed for JSON
• jq play

HTML parsing with Python

Scraping Data with Python and XPath:

Sample code from reference link which tell the whole story :)

import requests
from lxml import html

pageContent=requests.get('https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo')
tree = html.fromstring(pageContent.content)

goldWinners=tree.xpath('//*[@id="mw-content-text"]/table/tr/td[2]/a[1]/text()')
silverWinners=tree.xpath('//*[@id="mw-content-text"]/table/tr/td[3]/a[1]/text()')
#bronzeWinner we need rows where there's no rowspan - note XPath
bronzeWinners=tree.xpath('//*[@id="mw-content-text"]/table/tr/td[not(@rowspan=2)]/a[1]/text()')
medalWinners=goldWinners+silverWinners+bronzeWinners

medalTotals={}
for name in medalWinners:
    if medalTotals.has_key(name):
        medalTotals[name]=medalTotals[name]+1
    else:
        medalTotals[name]=1

for result in sorted(
        medalTotals.items(), key=lambda x:x[1],reverse=True):
        print '%s:%s' % result

BeautifulSoup is another option but different style from xpath.

HTML page parsing with xmllint xpath in BASH

Per HTML_parsers, there is no better HTML page parsing options for BASH. Inspired by Retrieve web using xpath, here comes the summary of using xmllint xpath:


xpath='' # sample: '//div[@class = "tides"]'

get_element_by_xpath():
    echo $HTML_PAGE | xmllint --html --xpath $xpath - 2>/dev/null

get_element_text_by_xpath():
    xpath+='/text()'
    echo $HTML_PAGE | xmllint --html --xpath $xpath - 2>/dev/null

get_elements_count_by_xpath():
    xpath="count($xpath)"
    echo $HTML_PAGE | xmllint --html --xpath $xpath - 2>/dev/null

Setup Bash Debugger in VS Code

Recent projects need some bash scripting, besides bashdb in terminal, GUI bash debugging make life much easier:

1. Follow all steps in Upgrading Bash on macOS to update bash (need bash > 4.0)
2. Install bashdb: brew install bashdb
3. In VS Code:
   1. Install extension Bash Debug
   2. Configure bash debugger by https://marketplace.visualstudio.com/items?itemName=rogalmic.bash-debug
   3. Additional configure: add "terminalKind": "debugConsole" to launch.json to each configuration