Load Testing with Sahi Pro

abstract Sahi Pro lets you run load tests on your application. There are 2 Sahi scripts involved in this. One is used to generate the load. This script is called the noise and is run on a headless browser. The steps in the noise file are generally in an infinite loop so that they never terminate. The other script is called the subject. The subject is the script whose steps you would like to measure periodically at different loads.

Setting up

  1. Get the headless browser, Google Chrome headless. Read this article, Sahi headless execution with Google Chrome.
  2. Start Sahi Pro. Google Chrome headless should be visible on the Dashboard as one of the browsers.
  3. On the dashboard, click on the 'Bin' link. This will open up a command prompt at userdata/bin.
  4. On the command prompt, run
    dload.bat demo/load/noise.sah demo/load/subject.sah http://sahitest.com/demo/training/ firefox
This will do the following:
  1. Run 1 (min) thread(s) of noise.sah on Google Chrome headless (This will not be visible)
  2. Run subject.sah on firefox browser and store the time of steps. Subject will be run 3 (subjectRepeatCount) times for averaging.
  3. Wait for 5 seconds (interval).
  4. Run 2 (incrementBy) more threads of noise.sah on Google Chrome headless.
  5. Run subject.sah 3 times on firefox and store results.
  6. Keep incrementing the noise threads at periodic intervals, run the subject 3 times, and store results, till it reaches max noise threads. Once done, click on 'Logs' link on the Dashboard to view results.

Load Testing Using a Different Master

info From Sahi Pro v9.0.0 onwards, it is possible to do load testing using a different master.
This should be used when the Master is different from the Initiator. A typical case where you would use this or the ANT equivalent, is when you have a continuous integration build server that is not capable of launching any UI. The build server will contain all the scripts but since it cannot launch any UI, it cannot run Sahi tests. In such a case, Sahi can be run on other machines and dload_different_master can be used on the integration build server to trigger the Sahi tests on these machines.

dload_different_master does the following. Let us look at dload_different_master in more detail.
  1. Sahi (running on the Master) distributes the scripts to different machines based on their capacity and availability of browsers. Note that Sahi should be running on each of the nodes. Also, on each machine, the browser paths must be correctly specified.
    info Sahi does not need to be installed on the Initiator machine.
  2. The following directory structure is assumed on the Initiator machine. The requisite files like ant-sahi.jar should be copied over from a Sahi Pro installation. The directory structure of the files is the same in the SahiPro installation.
    <TOP_LEVEL_FOLDER>
    	lib
    		ant-sahi.jar
    	logs
    	userdata
    		bin
    			dload_different_master.bat
    			setjava.bat
    		config
    			email.properties
    		scripts
    			<ALL YOUR SCRIPT FOLDERS AND SCRIPTS>
  3. Open <sahi_home>/userdata/bin/dload_different_master.bat (dload_different_master.sh on linux) using any Editor.
  4. MASTER_HOST should point to the Master machine. For Windows:
    SET MASTER_HOST=machine1
    For Linux:
    export MASTER_HOST=machine1
    danger Make sure you change "machine1" to the actual machine name or IP
  5. INITIATOR_ORIGIN_FOLDER is the path to the base folder containing the various scripts. With the above assumed directory structure, it would point to <TOP_LEVEL_FOLDER>/userdata/scripts.
    info The default value need not be changed.
  6. MASTER_STAGING_PATH refers to the Staging folder on the Master machine to which the contents of INITIATOR_ORIGIN_FOLDER will be first synced to. Distribution of scripts will happen from MASTER_STAGING_PATH.
    info The default value need not be changed.
  7. Sahi can send a mail summarizing the playback. The variable SEND_EMAIL_REPORT can be set as true or false. If true, Sahi sends the mail.

    For Windows:
    SET SEND_EMAIL_REPORT=true
    For Linux:
    export SEND_EMAIL_REPORT=true
    Email specific parameters like username, password etc. are defined in sahi/userdata/bin/email.properties file.
  8. Sahi can send a mail based on success or failure or in both the cases. This condition can be set using the EMAIL_TRIGGER variable.

    For Windows:
    SET EMAIL_TRIGGER=success,failure,user_aborted
    For Linux:
    export EMAIL_TRIGGER=success,failure,user_aborted
  9. For a long running suite, Sahi can send periodic emails indicating progress. To enable periodic emails, set the following property.

    For Windows:
    SET SEND_EMAIL_REPORT_PERIODICALLY=true
    For Linux:
    export SEND_EMAIL_REPORT_PERIODICALLY=true


    To set time interval (in minutes), use SET SEND_EMAIL_REPORT_PERIODICALLY_TIME variable.

    For Windows:
    SET SEND_EMAIL_REPORT_PERIODICALLY_TIME=30
    For Linux:
    export SEND_EMAIL_REPORT_PERIODICALLY_TIME=30
  10. The relative path to the properties file containing information about the mail such as sender, recipient, host, port etc. can be set using the EMAIL_PROPERTIES variable.
  11. Passwords can be hidden or shown from the console and logs by toggling the EMAIL_PASSWORD_HIDDEN variable. It can be set as true or false. If true, Sahi hides the password from Sahi console and logs.
  12. Sahi can set offline logs to be generated in xml, html, junit, tm6 and excel types. The default type is html. These logs will be generated on the Master and pulled back to the Initiator, since the user would want to see the logs on the Initiator. The following set of commands illustrate how HTML and XML logs can be generated.

    For Windows:
    SET UNIQUE_ID=%DATE%__%TIME%
    SET UNIQUE_ID=%UNIQUE_ID: =_%
    SET UNIQUE_ID=%UNIQUE_ID::=_%
    SET UNIQUE_ID=%UNIQUE_ID:/=_%
    
    SET MASTER_HTMLLOGS_DIR=logs/temp/html/%UNIQUE_ID%
    SET MASTER_XMLLOGS_DIR=logs/temp/xml/%UNIQUE_ID%
    
    SET LOGS_INFO=html:%MASTER_HTMLLOGS_DIR%,xml::%MASTER_XMLLOGS_DIR%
    
    SET INITIATOR_OUTPUT_HTMLLOGS_DIR=%TOP_LEVEL_FOLDER%/logs/html/%UNIQUE_ID%
    SET INITIATOR_OUTPUT_XMLLOGS_DIR=%TOP_LEVEL_FOLDER%/logs/xml/%UNIQUE_ID%
    For Linux:
    export UNIQUE_ID=$(date +"%m_%d_%Y")__$(date +"%I_%M_%S")
    
    export MASTER_HTMLLOGS_DIR=logs/temp/html/$UNIQUE_ID
    export MASTER_XMLLOGS_DIR=logs/temp/xml/$UNIQUE_ID
    
    export LOGS_INFO=html:$MASTER_HTMLLOGS_DIR,xml:$MASTER_XMLLOGS_DIR
    
    export INITIATOR_OUTPUT_HTMLLOGS_DIR=$TOP_LEVEL_FOLDER/logs/html/$UNIQUE_ID
    export INITIATOR_OUTPUT_XMLLOGS_DIR=$TOP_LEVEL_FOLDER/logs/xml/$UNIQUE_ID
    info The default values can be left as is.
  13. Configure the NODES variable to contain all the machines on which the tests should run. The command to set the nodes in Windows and Linux respectively are: For Windows:
    SET NODES=machine2:9999,machine3:9999,machine4:9999
    For Linux:
    export NODES=machine2:9999,machine3:9999,machine4:9999
    info The nodes may or may not include the Master machine. If the Master machine is not included, scripts will not be run on the Master.
    danger Do not include localhost (the Initiator) here since Sahi may not be installed on the Initiator. If you want localhost as one of the nodes, run dload instead.
    warning "machine2", "machine3" and "machine4" are placeholders to illustrate usage. Replace them with actual machine names. The nodes can contain 1 or more machines.
  14. SUBJECT_HOST should point to the machine, where subject browser has to run. For Windows:
    SET SUBJECT_HOST=machine1
    For Linux:
    export SUBJECT_HOST=machine1
  15. To execute the scripts using dload_different_master.bat, open a command window and go to the <sahi_home>/userdata/bin directory. This can also be done easily by clicking the bin link on the dashboard.
  16. Run dload_different_master.bat to see the syntax.

    D:\sahi\userdata\bin>dload_different_master.bat -- Usage: dload_different_master.bat <noise sah file> <subject sah file> <startURL> <browserType> File path is relative to userdata/scripts tags are used only if the input suite is a csv file -- Example: dload_different_master.bat demo/load/noise.sah demo/load/subject.sah http://sahitest.com/demo/training/ firefox -- D:\dev\sahi\sahi_pro_g\userdata\bin>


    danger NOTE:  Refer to the notes below for actual details.
    In the usage above,
    • The first argument is the noise script path. This path can be relative to the path set in INITIATOR_ORIGIN_FOLDER, or it can be an absolute path contained in INITIATOR_ORIGIN_FOLDER.
    • The second argument is the subject script path. This path can be relative to the path set in INITIATOR_ORIGIN_FOLDER, or it can be an absolute path contained in INITIATOR_ORIGIN_FOLDER.
    • The third argument is the start URL Type.
    • The forth argument is the Subject Browser Type.

Reporting

Reports show the most expensive steps and URLs in the load test. Clicking on the circles in the graphs takes you to the step/URL which was expensive.

Most Expensive Steps Across Different Loads: Most Expensive URLs Across Different Loads: Steps:
info Note: The table contents are truncated in this image. You should see entries for Load: 9 as well, in the actual log.
To test your application, go to userdata/scripts/demo/load folder.
  1. Open noise.sah and add your sequence of steps. These steps should be representative of a normal user interaction. Note that your steps should be in a while(true){} loop so that the threads don't die out. (Refer to existing noise.sah)
  2. Open subject.sah and specify the sequence of steps whose reaction times you would like to measure at different loads.
  3. The parameters min, max, incrementby, interval, subjectRepeatCount etc. can be edited in dload.bat file.
  4. It is also possible to involve multiple machines in creating the load by modifying the NODES variable (Look in dload.bat)
  5. To get HAR logs and URL timing information, use _startHarLogging() in your script to start logging HAR logs. To stop the logs, use _stopHarLogging().

Running load tests through ANT

Sahi supports load testing through an ANT target.

Load Testing

Copy the following content to an empty file and save the file as an xml, say dload.xml in your <SAHI_INSTALLATION_FOLDER>. You can now run the default ANT target in the xml file as
ant -f dload.xml


<?xml version="1.0" encoding="ISO-8859-1"?>
<!--
/**
 * Copyright Tyto Software Pvt. Ltd.
 */
-->
<project name="demo" default="dload">
	<taskdef name="sahid" classname="in.co.sahi.ant.DAntLoadRunner">
		<classpath>
			<pathelement location="lib/ant-sahi.jar"/>
		</classpath>
	</taskdef>

	<target name="set_timestamp">
		<tstamp>
			<format property="ts" pattern="yyyy_MM_dd_HH_mm_ss" locale="en, IN"/>
		</tstamp>
	</target>

	<target name="set_properties_common" depends="set_timestamp">
		<property name="scriptDir" value="scripts/"/>
		<property name="host" value="localhost"/>
		<property name="port" value="9999"/>
	</target>

	<target name="set_properties_firefox" depends="set_properties_common">
		<property name="subject" value="demo/load/subject.sah"/>
		<property name="noise" value="demo/load/noise.sah"/>
		<property name="browser" value="firefox"/>
		<property name="baseURL" value="http://sahitest.com/demo/training/"/>
		<property name="noiseBrowserType" value="chromeHL"/>
		<property name="minThread" value="1"/>
		<property name="maxThread" value="9"/>
		<property name="increment" value="2"/>
		<property name="interval" value="5"/>
		<property name="repearCount" value="3"/>
		<property name="sendemail" value="false"/>
		<property name="emailtrigger" value="success,failure,user_aborted"/>
		<property name="emailproperties" value="userdata/config/email.properties"/>
		<property name="sendemailperiodically" value="false"/>
		<property name="sendemailperiodicallytime" value=""/>
		<property name="emailPasswordHidden" value="true"/>
	</target>

	<target name="dload" depends="set_properties_firefox, set_timestamp">
		<sahid 	noise="${noise}"
				noiseBrowserType="${noiseBrowserType}"
				min="${minThread}"
				max="${maxThread}"
				incrementBy="${increment}"
				interval="${interval}"
				subject="${subject}"
				subjectRepeatCount="${repearCount}"
				sahihost="${host}"
				sahiport="${port}"
				baseURL="${baseURL}"
				browsertype="${browser}"
				scriptsPathMaster="${scriptDir}"
				ignorePattern=".*(svn|copied).*"
				sendemail="${sendemail}"
				emailtrigger="${emailtrigger}"
				emailproperties="${emailproperties}"
				sendEmailPeriodically="${sendemailperiodically}"
				sendEmailPeriodicallyTime="${sendemailperiodicallytime}"
				emailPasswordHidden="${emailPasswordHidden}">
			<!-- Scripts will be distributed across all the nodes. The nodes may or may not include the Master machine -->
			<node host="localhost" port="9999"/>
			<!-- Offline reports -->
			<report type="html" logdir="logs/playback/html"/>
		</sahid>

		<antcall target="failsahifirefox"/>
	</target>

	<target name="failsahifirefox" depends="set_properties_firefox" if="sahi.failed.${browser}">
		<fail message="Sahi tests failed on browser ${browser}!"/>
	</target>
</project>


Load Testing with Different Master

Copy the following content to an empty file and save the file as an xml, say dload_different_master.xml in your <SAHI_INSTALLATION_FOLDER>. You can now run the default ANT target in the xml file as
ant -f dload_different_master.xml


<?xml version="1.0" encoding="ISO-8859-1"?>
<!--
/**
 * Copyright Tyto Software Pvt. Ltd.
 */
-->
<project name="demo" default="dloaddiffmaster">
	<taskdef name="sahiload" classname="in.co.sahi.ant.DAntLoadRunner" classpath="lib/ant-sahi.jar" />
	<taskdef name="sahisync" classname="in.co.sahi.ant.DAntSync" classpath="lib/ant-sahi.jar" />
	<taskdef name="sahidelete" classname="in.co.sahi.ant.DAntDelete" classpath="lib/ant-sahi.jar" />
	<taskdef name="sahipull" classname="in.co.sahi.ant.DAntPull" classpath="lib/ant-sahi.jar" />

	<target name="set_timestamp">
		<tstamp>
			<format property="ts" pattern="yyyy_MM_dd__HH_mm_ss" locale="en, IN" />
		</tstamp>
	</target>

	<target name="set_properties_common" depends="set_timestamp">
		<property name="scriptDir" value="userdata/scripts/" />
		<property name="masterhost" value="localhost" />
		<property name="masterport" value="9999" />
		<property name="subjectNode" value="localhost:9999" />
	</target>


	<target name="set_properties_browser" depends="set_properties_common">
		<property name="subject" value="demo/load/subject.sah" />
		<property name="noise" value="demo/load/noise.sah" />
		<property name="baseURL" value="http://sahitest.com/demo/training/" />
		<property name="browser" value="firefox" />
		<property name="noiseBrowserType" value="chromeHL" />
		<property name="minThread" value="1" />
		<property name="maxThread" value="9" />
		<property name="increment" value="3" />
		<property name="interval" value="5" />
		<property name="repeatCount" value="3" />
		<property name="sendemail" value="false"/>
		<property name="emailtrigger" value="success,failure,user_aborted"/>
		<property name="emailproperties" value="userdata/config/email.properties"/>
		<property name="sendemailperiodically" value="false"/>
		<property name="sendemailperiodicallytime" value=""/>
		<property name="emailPasswordHidden" value="true"/>
	</target>

	<target name="dloaddiffmaster" depends="set_properties_browser, set_timestamp">
		<!-- Sync suite to Master staging folder -->
		<property name="masterStagingPath" value="temp/scripts/staging/${ts}_${browser}" />
		<property name="ignorePattern" value=".*(svn|copied).*" />

		<sahisync	originFolder="${scriptDir}"
					destFolder="${masterStagingPath}"
					ignorePattern="${ignorePattern}"
					noise="${noise}"
					subject="${subject}"
					csvSeparator=","
					scriptExtensions="sah;sahi;js;"
					scenarioExtensions=".s.csv;xls;xlsx"
					javaDir="java"
					exposedJavaClasses="exposed_classes.txt">
			<node host="${masterhost}" port="${masterport}" />
		</sahisync>

		<sahiload	noise="${noise}"
					noiseBrowserType="${noiseBrowserType}"
					min="${minThread}"
					max="${maxThread}"
					incrementBy="${increment}"
					interval="${interval}"
					subject="${subject}"
					subjectRepeatCount="${repeatCount}"
					sahihost="${masterhost}"
					sahiport="${masterport}"
					baseURL="${baseURL}"
					browsertype="${browser}"
					scriptsPathMaster="${masterStagingPath}"
					subjectNode="${subjectNode}"
					scriptsPathInitiator="${scriptDir}"
					isDifferentMaster="true"
					ignorePattern="${ignorePattern}"
					sendemail="${sendemail}"
					emailtrigger="${emailtrigger}"
					emailproperties="${emailproperties}"
					sendEmailPeriodically="${sendemailperiodically}"
					sendEmailPeriodicallyTime="${sendemailperiodicallytime}"
					emailPasswordHidden="${emailPasswordHidden}">
			<!-- Scripts will be distributed across all the nodes. The nodes may or may not include the Master machine -->
			<node host="localhost" port="9999" />
			<!-- Offline Reports-->
			<report type="html" logdir="logs/temp/html/${ts}_${browser}" />
		</sahiload>

		<!-- Delete scripts from Master staging folder -->
		<sahidelete	host="${masterhost}"
					port="${masterport}"
					filePath="${masterStagingPath}">
		</sahidelete>

		<!-- Pull the HTML logs from the Master onto the Initiator machine -->
		<sahipull	sourceHost="${masterhost}"
					sourcePort="${masterport}"
					originFolder="logs/temp/html/${ts}_${browser}"
					destFolder="logs/html"
					ignorePattern="${ignorePattern}">
		</sahipull>

		<antcall target="failsahi" />
	</target>

	<target name="failsahi" depends="set_properties_browser" if="sahi.failed.${browser}">
		<fail message="Sahi tests failed on browser ${browser}!" />
	</target>

</project>


Attributes and their description

info taskdef classpath is relative to where ant is run from.
scriptDirSpecifies the path to the scripts folder. Can be the absolute path or a path relative to the userdata folder
hostHostname of the server where Sahi is running (Can be IP as well). Leave this as localhost
portPort on which Sahi is running
subjectPath to the subject script.
noisePath to the noise script.
browserThe browser on which the subject script is run (can be IE, Firefox etc.)
baseurlSpecifies URL which is the starting URL for the subject and noise scripts
noiseBrowserTypeThe browser on which the noise script is run (default ChromeHL)
minThreadThe minimum number of noise browsers to run initially
maxThreadThe maximum number of noise browsers to run finally
incrementThe number of noise browsers to be run incrementally after the specified interval
intervalThe interval after which to increment the noise browsers by
repearCountNumber of times the subject script should be run at every noise level. It is good to run it more than 1 time to average out any random behaviour.
nodeEach node attribute specifies a machine on which the noise browsers should run. Add as many node entries as there are machines to run. The nodes may or may not include the Master machine (localhost). If the Master machine is not included, the noise browsers will not be run on the Master. There can be 1 or more nodes.
reportThis attribute is used to store offline report. type parameter specify the format in which the offline logs should be downloaded and logdir is the file location of the offline logs.

Command to execute the above ANT target

ant -f dload.xml