Skip to content
Get Free Trial

Visual Assist Automation - Getting Started

Find and act on elements by how they look on screen, using image recognition and OCR with the Visual Assist Engine.

Some elements can’t be found from the application’s code: a button that’s only an image, a chart drawn on a canvas, or text inside a picture. Sahi Pro’s usual accessors read the code, so they can’t see these. Visual Assist finds them by how they look on screen instead, either by a picture you capture (image recognition) or by their text (optical character recognition, OCR).

Visual Assist works with web applications and, with the matching engine, with desktop and mobile applications.

Install the Visual Assist Engine. See Install Engines.

  1. In the No-Code Test Editor, select the keyword and click Automate.
  2. In the Automate with Controller dialog, choose the Controller Mode for your application, such as Browser, Desktop, Java, SAP, Android or iOS, and fill in where it starts.
  3. Click Launch Controller.

For more ways to open it, see Open the Controller.

  1. In the Controller, open the Advanced Editor tab.
  2. Click Visual Assist, the first of the four icons at the top right. The Visual Assist panel appears.
  3. Choose Image Recognition to find an element by its picture, or Optical Character Recognition to find it by its text.
The Advanced Editor with Visual Assist turned on and its list open, showing None, Image Recognition and Optical Character Recognition, with the Visual Assist button and the list highlighted

To use an element that has no code behind it, such as a logo or an icon in a desktop application, capture a picture of it first. Sahi Pro then looks for that picture on the screen.

  1. Choose Image Recognition.

  2. If the element only shows after you do something, such as opening a menu, set a delay in seconds in the box next to the list. It gives you time to bring the right window to the front.

  3. Click Capture an image to identify, and select the area of the screen with the element.

    The Visual Assist panel with Image Recognition chosen, the capture delay 0, and the Capture an image to identify button highlighted
  4. Name the image. To keep it in a folder, choose one under Image Directory, or type a new folder in front of the name, such as Logos/CompanyLogo.

  5. Element Accessor shows the accessor for the image, which uses _visualImage.

To see the images you’ve captured, click Open the images directory, next to the capture button.

Text inside an image or a canvas isn’t in the code either, so Sahi Pro can’t read it the usual way. OCR reads it from the screen.

  1. Choose Optical Character Recognition.
  2. Set a delay if you need one, as for an image.
  3. Click Capture an image to identify, and select the area with the text.
  4. Element Accessor shows the accessor for the text, which uses _visualText.
The browser above the Controller. In the browser, the area around the book title Python Cookbook is selected. In the Controller, Visual Assist is set to Optical Character Recognition and Element Accessor shows _visualText("Python Cookbook"), highlighted.

Once an image is captured, or for text on the screen, you can identify it the usual way:

  1. Choose Image Recognition or Optical Character Recognition.
  2. Hold Ctrl (⌘ on Mac) and hover over the image or the text. Element Accessor shows its accessor.
The browser above the Controller. In the browser, the pointer is over the Add button, highlighted. In the Controller, Visual Assist is set to Image Recognition and Element Accessor shows _visualImage("training/add_button.png"), highlighted.

Add a step for the element as for any other: choose a Quick Action, try it with Evaluate Step, then click Add as Step and Save. See Identify an Element and Add a Step.

Sahi Pro acts on a Visual Assist element the way a user would, with real mouse and keyboard events. See Native Events.

To try a Visual Assist accessor on its own, use the Evaluator.

A Visual Assist step names its image only by a file path, such as training/add_button.png. When a test has several such steps, the path alone doesn’t tell you which button or icon each one clicks. Open the image from the step to see it:

  1. In the No-Code Test Editor, open the scenario file and select the keyword. Its steps show on the right, and a Visual Assist step reads like Click on training/add_button.png visualImage.

    The No-Code Test Editor with the Buy Books Using Button Images test case open and the Add Books To Cart keyword selected; its steps list on the right, with the step Click on training/add_button.png visualImage highlighted
  2. Double-click the step to edit it, then click View Image (the eye icon) in the Element box.

    The Edit Step panel for the step, with Action Click and Element _visualImage("training/add_button.png"), and the View Image eye icon in the Element box highlighted
  3. The Visual Assist Image Viewer shows the image, its path and its width and height. Open Parent Folder opens the directory it’s saved in, the snips directory of your project.

    The Visual Assist Image Viewer showing the captured Add button, with Image Path training/add_button.png, Width 47px, Height 29px and an Open Parent Folder link

Close the viewer and click Cancel to leave the step unchanged.

Run it from the Controller, as described in Run Tests via Controller, or from the Editor, as described in Run Tests from the Editor.

For every Visual Assist function, see Visual Assist Accessor Functions and Visual Assist Functions.