Automated Testing Elements on a Lazy Load Page with Selenium WebDriver

How to verify elements that get dynamically loaded on a page using Selenium WebDriver

---

Zhimin’s notes: 

Three years ago, I consulted on a Microsoft Dynamics 365 CRM project. I developed one automated (UI) test for a core end-to-end workflow. The project manager did not expect that, after seeing the test execution, he said: “I have never seen automating Dynamics 365 like this (driving UI) and you did that on the first day. What testing tool did you use?”
 
My answer: “Raw Selenium WebDriver”.
The PM said: “I was told by automated testers, more than once, that Selenium or others, unable to drive Dynamics 365, because it is lazy loading”.
I replied: “Oh well, you are seeing it now”. 
 
In this article, Courtney shows two approaches to automated-testing lazy-load web pages. 
 
Advice on automating Microsoft web apps. The lazy loading in Dynamics 365 is not obvious as the example used in this article. Futhermore, combined with multiple frames used in Microsoft web pages, it can be overwhelming. However, if you understand automating frames and lazy-loading well, using a good tool such as TestWise, it is not that hard. A feature like TestWise’s “Attach test execution to the existing browser” will be very helpful. 

Some modern websites now use lazy loading (also known as progressive loading). Instead of loading everything on the page at once, the items load as you scroll. This brings challenges to test automation, because the element we want to control may not be displayed yet. An example site is Substack, if you scroll down on this site, you will notice the bottom of this page reloads more articles.

This article will show you how to use Selenium WebDriver to force lazy-loaded pages to load your desired element, and drive it.

Test Design

1. Look for the element on the page

2. If the element is not there, scroll down the page 

3. Repeat steps 1–2

1. Look for the element on the page

First, try to locate the element. In the best case, the element is already there on the initial page load. This means there is nothing to do, and we can proceed normally.

driver.get("https://agileway.substack.com/archive")
driver.find_element(:link_text, "12 Payment Test (AJAX)").click
try_for(4) { expect(page_text).to include("Selenium Explicit Waits") }

For this article, I will be trying to click on the article “Fake End-to-End Test Automation Clarified” from the Substack archive list. This was published a while back, so if we try to do this straight away, there will be a NoSuchElement error.

driver.get("https://agileway.substack.com/archive")
driver.find_element(:link_text, "Fake End-to-End Test Automation Clarified").click

Below is a failed test execution in TestWise. 

2. Scroll down if the element is not there

Because the element isn’t there yet, keep scrolling down the page until it is. The easiest way to do this is to send the “Page Down” key to the browser.

driver.find_element(:tag_name, "body").send_keys(:page_down)

We don’t know how many times we have to scroll down, so for now, I will use an arbitrarily large number, 100. But set a condition to exit if the desired element is found. 

100.times do |x|
    puts("Scroll: #{x + 1}")
    driver.find_element(:tag_name, "body").send_keys(:page_down)
    sleep 1
    begin
        driver.find_element(:link_text, "Fake End-to-End Test Automation Clarified").click
        # quit the loop if found
        break 
    rescue => e
        # not found, continue
    end
end

sleep 3

The above code block:

• Start a loop that stops at 100

• Send the “Page Down” key to the browser

• Try to click a specific article.

• If it can find the article, quit the loop (break).

• If it can’t find the article, continue the loop.

After the loop completes, assert that we are now on the article’s page.

expect(page_text).to include("After knowing what is not real, motivated professionals can learn and achieve the real.")

Below is a successful test execution in TestWise, scrolled 14 times.

However, the “Page Down” key doesn’t take us to the bottom of the page and triggers the lazy load. To be more efficient, we should go directly to the bottom of the page.

Alternative Approach