Draw on a Canvas using Selenium WebDriver

Use Selenium WebDriver Advanced User Interactions API to draw and manoeuvre objects in a Canvas on a web page

Nov 14, 2022

This article is included in my “How to in Selenium WebDriver” series.

We can use Selenium WebDriver’s Advanced User Interactions API to drive mouse events to ‘draw’ on a canvas on a web page. In this article, I will show how to draw markers on a body chart (background image) in the WhenWise app, with the following operations:

Drag a circle to the back neck.
Draw the shape of “X” and move it to the right chest
Change brush colour to blue
Draw an arrow, rotate it and point to the left elbow
Add text on the canvas

Here it is.

Animated GIF showing the operations driven by Selenium WebDriver

1. Move the circle to the back neck.

The script:

driver.find_element(:id, "drawing-circle").click
elem = driver.find_element(:id, "service_chart")
driver.action.move_to(elem, 40, 40).click.click_and_hold.move_by(450, 60).release.perform

I will explain the above step by step.

Draw a circle
Click a button (which calls JS to draw a shape on the canvas), the standard Selenium way.

driver.find_element(:id, "drawing-circle").click

Set the starting position (by the control)
As we know, drawing is all about coordinates. Therefore, we need to set a starting point, i.e. 0,0 . Typically, we use the position of the Canvas control. The statement below stores the Canvas control to elem . Later on, we will pass this elem as the starting position.

elem = driver.find_element(:id, "service_chart")

Move the circle
Perform a chain of the following actions.

a) Move the mouse to the object
b) Click and hold the object
c) Move to the target position
d) Release the mouse

driver.action.move_to(elem, 40, 40).click_and_hold.move_by(450, 60).release.perform

The syntax starts driver.action , followed by a list of actions, then ends with perform .

Question: How about identifying the X and Y coordinates?

With X,Y coordinates involved, the most challenging task would be to find the X,Y of the object. The correct answer is to calculate the object's movement.

My method is “Trial and error”. With TestWise’s “attaching executions of the selected test steps to the last browser window” feature, it is quite easy to do.

I switched to the ‘drawing mode’, and ran the ‘moving’ step again and again until I found the correct X, Y coordinates.

the debug script run in TestWise to draw the moving line.

In this TestWise ‘Debug’ mode, there is no opening/closing of a new browser. Just execute the specific test steps against the current browser, quick and easy!

2. Draw the shape of “X” and move it to the right chest

Now we switch to the ‘free hand’ drawing mode.

driver.find_element(:id, "drawing-mode").click

Draw a shape of “X”, i.e. two crossing lines.

driver.action.move_to(elem, 50, 50).click_and_hold.move_by(50, 50).release.perform
driver.action.move_to(elem, 100, 50).click_and_hold.move_by(-50, +50).release.perform

Drag the two lines to a new position, one by one. To do that, we need to exit the drawing mode first.

 driver.find_element(:id, "drawing-mode").click
driver.action.move_to(elem, 80, 80).click_and_hold.move_by(70, 80).release.perform
driver.action.move_to(elem, 80, 80).click_and_hold.move_by(70, 80).release.perform

3. Change brush colour to blue

The brush colour is set by the standard HTML5 input control

<input type='color'>

Please note, we cannot (not necessarily anyway) use Selenium to drive the mouse to change the colour.

The simple and reliable way to use JavaScript.

color_elem = driver.find_element(:id, "drawing-color")
driver.execute_script('arguments[0].value = arguments[1]',  
                      color_elem, "#0000FF")

4. Draw an arrow, rotate it and point to the left elbow

Add an arrow shape to the canvas.

driver.find_element(:id, "drawing-arrow").click

The handle of the above image object was not shown. Therefore, I dragged it down.

driver.action.move_to(elem, 240, 42).click.click_and_hold.move_by(0, 50).release.perform

Select the handle and rotate the shape by moving the handle to a different position.

driver.action.move_to(elem, 242, 30).click.perform # select it
driver.action.move_to(elem, 242, 30).click_and_hold.move_by(-20, 100).release.perform

With the right angle, move it to the ‘elbow’.

driver.action.move_to(elem, 240, 80).click_and_hold.move_by(155, 93).release.perform

5. Add text on the canvas

Click the ‘add text’ button.

driver.find_element(:id, "drawing-text").click
sleep 0.5

In the popup, we enter the text.

driver.find_element(:xpath, "//div[@class='modal open']//input[@type='text' and contains(@id, 'modal-input-')]").send_keys("Selenium WebDriver is the best!")

The above XPath might seem complex (instead of driver.find_element(:id, ‘modal-input-0’), I chose this one as it is more reliable. The ID of the text field was dynamically generated by a JS library as below:

<input type="text" id="modal-input-0">

The value of ID might change if there are other prompt dialogs. By adding @class='modal open', I can ensure that I have located the correct one.

Click the “Confirm” button.

driver.find_element(:xpath, "//div[@class='modal open']/div[@class='modal-footer']/a[@data-name='confirm']").click

Again, I chose the most reliable locator (in my opinion). For more on this topic, please read Working Automated Test ≠ Good Reliable Test.

The final Canvas.

Full Script

Raw Selenium WebDriver (Ruby) in RSpec syntax.

load File.dirname(__FILE__) + "/../test_helper.rb"
describe "Draw canvas" do
  include TestHelper
  before(:all) do
    @driver = Selenium::WebDriver.for(browser_type, browser_options)
    @driver.navigate.to(site_url)
    driver.manage().window().move_to(200, 0)
    driver.manage().window().resize_to(1080, 720)   
    sign_in("physio@biz.com")
  end

  after(:all) do
    @driver.quit unless debugging?
  end

  it "Draw canvas" do
    visit "/work_orders/1"
    work_order_page = WorkOrderPage.new(driver)    driver.find_element(:id, "work-charts").click
    sleep 1    # set the archor location
    elem = driver.find_element(:id, "service_chart")    # add the a circle shape
    driver.find_element(:id, "drawing-circle").click
    sleep 1
    
    # move it to the back neck position
    driver.action.move_to(elem, 40, 40).click_and_hold.move_by(450, 60).release.perform    # draw X
    driver.find_element(:id, "drawing-mode").click
    sleep 0.5
    driver.action.move_to(elem, 50, 50).click_and_hold.move_by(50, 50).release.perform
    driver.action.move_to(elem, 100, 50).click_and_hold.move_by(-50, +50).release.perform
    sleep 0.5    # exiting the drawing mode and move to two lines to target
    driver.find_element(:id, "drawing-mode").click
    driver.action.move_to(elem, 80, 80).click_and_hold.move_by(70, 80).release.perform
    driver.action.move_to(elem, 80, 80).click_and_hold.move_by(70, 80).release.perform   # Change color
    color_elem = driver.find_element(:id, "drawing-color")
    driver.execute_script("arguments[0].value = arguments[1]", color_elem, "#0000FF")    #-  Rotate an arrow and drag it to elbow
    driver.find_element(:id, "drawing-arrow").click
    # move down
    driver.action.move_to(elem, 240, 42).click.click_and_hold.move_by(0, 50).release.perform    # select it
    driver.action.move_to(elem, 242, 30).click.perform    # rotate
    driver.action.move_to(elem, 242, 30).click_and_hold.move_by(-20, 100).release.perform # rotate    # move to elbow
    driver.action.move_to(elem, 240, 80).click_and_hold.move_by(155, 93).release.perform
    
    #- add text
    driver.find_element(:id, "drawing-text").click
    sleep 0.5
    
    elem = driver.find_element(:xpath, "//div[@class='modal open']//input[@type='text' and contains(@id, 'modal-input-')]")
    elem.send_keys("Selenium WebDriver is the best!")
    driver.find_element(:xpath, "//div[@class='modal open']/div[@class='modal-footer']/a[@data-name='confirm']").click
  end
end

I used the raw Selenium WebDriver here to showcase its Advanced User Interactions API. The form is hard to maintain though it may be good for a demo. And it certainly won’t be possible with serious UI Test Automation (Level 2+ of AgileWay Continuous Testing Grading). We need to refine it via Functional Test Refacotorings to follow the Maintainable Test Design, as below by using the Page Object Model.

service_chart_modal_page = ServiceChartModalPage.new(driver)    service_chart_modal_page.click_draw_circle
service_chart_modal_page.move_object([40,40], [450, 60])

The full video version:

AgileWay’s Test Automation & Continuous Testing Blog

Discussion about this post