Load Testing with Sahi Pro

Setting up

1. Get the headless browser, Google Chrome headless. Read this article, Sahi headless execution with Google Chrome.
2. Start Sahi Pro. Google Chrome headless should be visible on the Dashboard as one of the browsers.
3. On the dashboard, click on the 'Bin' link. This will open up a command prompt at userdata/bin.
4. On the command prompt, run

   dload.bat sahitests/load/noise.sah sahitests/load/subject.sah http://sahitest.com/demo/training/ firefox

This will do the following:

1. Run 1 (min) thread(s) of noise.sah on Google Chrome headless (This will not be visible)
2. Run subject.sah on firefox browser and store the time of steps. Subject will be run 3 (subjectRepeatCount) times for averaging.
3. Wait for 5 seconds (interval).
4. Run 2 (incrementBy) more threads of noise.sah on Google Chrome headless.
5. Run subject.sah 3 times on firefox and store results.
6. Keep incrementing the noise threads at periodic intervals, run the subject 3 times, and store results, till it reaches max noise threads. Once done, click on 'Logs' link on the Dashboard to view results.

Load Testing Using a Different Master

This should be used when the Master is different from the Initiator. A typical case where you would use this or the ANT equivalent, is when you have a continuous integration build server that is not capable of launching any UI. The build server will contain all the scripts but since it cannot launch any UI, it cannot run Sahi tests. In such a case, Sahi can be run on other machines and dload_different_master can be used on the integration build server to trigger the Sahi tests on these machines.

dload_different_master does the following.

• It first syncs the contents of scripts folder from the Initiator to a staging folder on the Master.
• It then distributes the scripts from the staging folder of the Master to various nodes and runs the tests.
• If offline logs are requested, the logs are copied over from the Master to the Initiator after the run is over.

Let us look at dload_different_master in more detail.

1. Sahi (running on the Master) distributes the scripts to different machines based on their capacity and availability of browsers. Note that Sahi should be running on each of the nodes. Also, on each machine, the browser paths must be correctly specified.
   info Sahi does not need to be installed on the Initiator machine.
2. The following directory structure is assumed on the Initiator machine. The requisite files like ant-sahi.jar should be copied over from a Sahi Pro installation. The directory structure of the files is the same in the SahiPro installation.

   <TOP_LEVEL_FOLDER>
   	lib
   		ant-sahi.jar
   	logs
   	userdata
   		bin
   			dload_different_master.bat
   			setjava.bat
   		config
   			email.properties
   		scripts
   			<ALL YOUR SCRIPT FOLDERS AND SCRIPTS>

3. Open <sahi_home>/userdata/bin/dload_different_master.bat (dload_different_master.sh on linux) using any Editor.
4. MASTER_HOST should point to the Master machine. For Windows:

   SET MASTER_HOST=machine1

   For Linux:

   export MASTER_HOST=machine1

   danger Make sure you change "machine1" to the actual machine name or IP
5. INITIATOR_ORIGIN_FOLDER is the path to the base folder containing the various scripts. With the above assumed directory structure, it would point to <TOP_LEVEL_FOLDER>/userdata/scripts.
   info The default value need not be changed.
6. MASTER_STAGING_PATH refers to the Staging folder on the Master machine to which the contents of INITIATOR_ORIGIN_FOLDER will be first synced to. Distribution of scripts will happen from MASTER_STAGING_PATH.
   info The default value need not be changed.
7. Sahi can send a mail summarizing the playback. The variable SEND_EMAIL_REPORT can be set as true or false. If true, Sahi sends the mail.
   
   For Windows:

   SET SEND_EMAIL_REPORT=true

   For Linux:

   export SEND_EMAIL_REPORT=true

   Email specific parameters like username, password etc. are defined in sahi/userdata/bin/email.properties file.
8. Sahi can send a mail based on success or failure or in both the cases. This condition can be set using the EMAIL_TRIGGER variable.
   
   For Windows:

   SET EMAIL_TRIGGER=success,failure,user_aborted

   For Linux:

   export EMAIL_TRIGGER=success,failure,user_aborted

9. For a long running suite, Sahi can send periodic emails indicating progress. To enable periodic emails, set the following property.
   
   For Windows:

   SET SEND_EMAIL_REPORT_PERIODICALLY=true

   For Linux:

   export SEND_EMAIL_REPORT_PERIODICALLY=true

   
   
   To set time interval (in minutes), use SET SEND_EMAIL_REPORT_PERIODICALLY_TIME variable.
   
   For Windows:

   SET SEND_EMAIL_REPORT_PERIODICALLY_TIME=30

   For Linux:

   export SEND_EMAIL_REPORT_PERIODICALLY_TIME=30

10. The relative path to the properties file containing information about the mail such as sender, recipient, host, port etc. can be set using the EMAIL_PROPERTIES variable.
11. Passwords can be hidden or shown from the console and logs by toggling the EMAIL_PASSWORD_HIDDEN variable. It can be set as true or false. If true, Sahi hides the password from Sahi console and logs.
12. Sahi can set offline logs to be generated in xml, html, junit, tm6 and excel types. The default type is html. These logs will be generated on the Master and pulled back to the Initiator, since the user would want to see the logs on the Initiator. The following set of commands illustrate how HTML and XML logs can be generated.
    
    For Windows:

    SET UNIQUE_ID=%DATE%__%TIME%
    SET UNIQUE_ID=%UNIQUE_ID: =_%
    SET UNIQUE_ID=%UNIQUE_ID::=_%
    SET UNIQUE_ID=%UNIQUE_ID:/=_%
    
    SET MASTER_HTMLLOGS_DIR=logs/temp/html/%UNIQUE_ID%
    SET MASTER_XMLLOGS_DIR=logs/temp/xml/%UNIQUE_ID%
    
    SET LOGS_INFO=html:%MASTER_HTMLLOGS_DIR%,xml::%MASTER_XMLLOGS_DIR%
    
    SET INITIATOR_OUTPUT_HTMLLOGS_DIR=%TOP_LEVEL_FOLDER%/logs/html/%UNIQUE_ID%
    SET INITIATOR_OUTPUT_XMLLOGS_DIR=%TOP_LEVEL_FOLDER%/logs/xml/%UNIQUE_ID%

    For Linux:

    export UNIQUE_ID=$(date +"%m_%d_%Y")__$(date +"%I_%M_%S")
    
    export MASTER_HTMLLOGS_DIR=logs/temp/html/$UNIQUE_ID
    export MASTER_XMLLOGS_DIR=logs/temp/xml/$UNIQUE_ID
    
    export LOGS_INFO=html:$MASTER_HTMLLOGS_DIR,xml:$MASTER_XMLLOGS_DIR
    
    export INITIATOR_OUTPUT_HTMLLOGS_DIR=$TOP_LEVEL_FOLDER/logs/html/$UNIQUE_ID
    export INITIATOR_OUTPUT_XMLLOGS_DIR=$TOP_LEVEL_FOLDER/logs/xml/$UNIQUE_ID

    info The default values can be left as is.
13. Configure the NODES variable to contain all the machines on which the tests should run. The command to set the nodes in Windows and Linux respectively are: For Windows:

    SET NODES=machine2:9999,machine3:9999,machine4:9999

    For Linux:

    export NODES=machine2:9999,machine3:9999,machine4:9999

    info The nodes may or may not include the Master machine. If the Master machine is not included, scripts will not be run on the Master.
    danger Do not include localhost (the Initiator) here since Sahi may not be installed on the Initiator. If you want localhost as one of the nodes, run dload instead.
    warning "machine2", "machine3" and "machine4" are placeholders to illustrate usage. Replace them with actual machine names. The nodes can contain 1 or more machines.
14. SUBJECT_HOST should point to the machine, where subject browser has to run. For Windows:

    SET SUBJECT_HOST=machine1

    For Linux:

    export SUBJECT_HOST=machine1

15. To execute the scripts using dload_different_master.bat, open a command window and go to the <sahi_home>/userdata/bin directory. This can also be done easily by clicking the bin link on the dashboard.
16. Run dload_different_master.bat to see the syntax.
    
    
    D:\sahi\userdata\bin>dload_different_master.bat -- Usage: dload_different_master.bat <noise sah file> <subject sah file> <startURL> <browserType> File path is relative to userdata/scripts tags are used only if the input suite is a csv file -- Example: dload_different_master.bat sahitests/load/noise.sah sahitests/load/subject.sah http://sahitest.com/demo/training/ firefox -- D:\dev\sahi\sahi_pro_g\userdata\bin>
    
    
    
    danger NOTE:  Refer to the notes below for actual details.
    In the usage above,
    • The first argument is the noise script path. This path can be relative to the path set in INITIATOR_ORIGIN_FOLDER, or it can be an absolute path contained in INITIATOR_ORIGIN_FOLDER.
    • The second argument is the subject script path. This path can be relative to the path set in INITIATOR_ORIGIN_FOLDER, or it can be an absolute path contained in INITIATOR_ORIGIN_FOLDER.
    • The third argument is the start URL Type.
    • The forth argument is the Subject Browser Type.

Reporting

Reports show the most expensive steps and URLs in the load test. Clicking on the circles in the graphs takes you to the step/URL which was expensive.

Most Expensive Steps Across Different Loads: Most Expensive URLs Across Different Loads: Steps: To test your application, go to userdata/scripts/sahitests/load folder.

1. Open noise.sah and add your sequence of steps. These steps should be representative of a normal user interaction. Note that your steps should be in a while(true){} loop so that the threads don't die out. (Refer to existing noise.sah)
2. Open subject.sah and specify the sequence of steps whose reaction times you would like to measure at different loads.
3. The parameters min, max, incrementby, interval, subjectRepeatCount etc. can be edited in dload.bat file.
4. It is also possible to involve multiple machines in creating the load by modifying the NODES variable (Look in dload.bat)
5. To get HAR logs and URL timing information, use _startHarLogging() in your script to start logging HAR logs. To stop the logs, use _stopHarLogging().

Running load tests through ANT

Sahi supports load testing through an ANT target.

Load Testing

Copy the following content to an empty file and save the file as an xml, say dload.xml in your <SAHI_INSTALLATION_FOLDER>. You can now run the default ANT target in the xml file as

ant -f dload.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<!--
/**
 * Copyright Tyto Software Pvt. Ltd.
 */
-->
<project name="demo" default="dload">
	<taskdef name="sahid" classname="in.co.sahi.ant.DAntLoadRunner">
		<classpath>
			<pathelement location="lib/ant-sahi.jar"/>
		</classpath>
	</taskdef>

	<target name="set_timestamp">
		<tstamp>
			<format property="ts" pattern="yyyy_MM_dd_HH_mm_ss" locale="en, IN"/>
		</tstamp>
	</target>

	<target name="set_properties_common" depends="set_timestamp">
		<property name="scriptDir" value="scripts/"/>
		<property name="host" value="localhost"/>
		<property name="port" value="9999"/>
	</target>

	<target name="set_properties_firefox" depends="set_properties_common">
		<property name="subject" value="sahitests/load/subject.sah"/>
		<property name="noise" value="sahitests/load/noise.sah"/>
		<property name="browser" value="firefox"/>
		<property name="baseURL" value="http://sahitest.com/demo/training/"/>
		<property name="noiseBrowserType" value="chromeHL"/>
		<property name="minThread" value="1"/>
		<property name="maxThread" value="9"/>
		<property name="increment" value="2"/>
		<property name="interval" value="5"/>
		<property name="repearCount" value="3"/>
		<property name="sendemail" value="false"/>
		<property name="emailtrigger" value="success,failure,user_aborted"/>
		<property name="emailproperties" value="userdata/config/email.properties"/>
		<property name="sendemailperiodically" value="false"/>
		<property name="sendemailperiodicallytime" value=""/>
		<property name="emailPasswordHidden" value="true"/>
	</target>

	<target name="dload" depends="set_properties_firefox, set_timestamp">
		<sahid 	noise="${noise}"
				noiseBrowserType="${noiseBrowserType}"
				min="${minThread}"
				max="${maxThread}"
				incrementBy="${increment}"
				interval="${interval}"
				subject="${subject}"
				subjectRepeatCount="${repearCount}"
				sahihost="${host}"
				sahiport="${port}"
				baseURL="${baseURL}"
				browsertype="${browser}"
				scriptsPathMaster="${scriptDir}"
				ignorePattern=".*(svn|copied).*"
				sendemail="${sendemail}"
				emailtrigger="${emailtrigger}"
				emailproperties="${emailproperties}"
				sendEmailPeriodically="${sendemailperiodically}"
				sendEmailPeriodicallyTime="${sendemailperiodicallytime}"
				emailPasswordHidden="${emailPasswordHidden}">
			<!-- Scripts will be distributed across all the nodes. The nodes may or may not include the Master machine -->
			<node host="localhost" port="9999"/>
			<!-- Offline reports -->
			<report type="html" logdir="logs/playback/html"/>
		</sahid>

		<antcall target="failsahifirefox"/>
	</target>

	<target name="failsahifirefox" depends="set_properties_firefox" if="sahi.failed.${browser}">
		<fail message="Sahi tests failed on browser ${browser}!"/>
	</target>
</project>

Load Testing with Different Master

Copy the following content to an empty file and save the file as an xml, say dload_different_master.xml in your <SAHI_INSTALLATION_FOLDER>. You can now run the default ANT target in the xml file as

ant -f dload_different_master.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<!--
/**
 * Copyright Tyto Software Pvt. Ltd.
 */
-->
<project name="demo" default="dloaddiffmaster">
	<taskdef name="sahiload" classname="in.co.sahi.ant.DAntLoadRunner" classpath="lib/ant-sahi.jar" />
	<taskdef name="sahisync" classname="in.co.sahi.ant.DAntSync" classpath="lib/ant-sahi.jar" />
	<taskdef name="sahidelete" classname="in.co.sahi.ant.DAntDelete" classpath="lib/ant-sahi.jar" />
	<taskdef name="sahipull" classname="in.co.sahi.ant.DAntPull" classpath="lib/ant-sahi.jar" />

	<target name="set_timestamp">
		<tstamp>
			<format property="ts" pattern="yyyy_MM_dd__HH_mm_ss" locale="en, IN" />
		</tstamp>
	</target>

	<target name="set_properties_common" depends="set_timestamp">
		<property name="scriptDir" value="userdata/scripts/" />
		<property name="masterhost" value="localhost" />
		<property name="masterport" value="9999" />
		<property name="subjectNode" value="localhost:9999" />
	</target>


	<target name="set_properties_browser" depends="set_properties_common">
		<property name="subject" value="sahitests/load/subject.sah" />
		<property name="noise" value="sahitests/load/noise.sah" />
		<property name="baseURL" value="http://sahitest.com/demo/training/" />
		<property name="browser" value="firefox" />
		<property name="noiseBrowserType" value="chromeHL" />
		<property name="minThread" value="1" />
		<property name="maxThread" value="9" />
		<property name="increment" value="3" />
		<property name="interval" value="5" />
		<property name="repeatCount" value="3" />
		<property name="sendemail" value="false"/>
		<property name="emailtrigger" value="success,failure,user_aborted"/>
		<property name="emailproperties" value="userdata/config/email.properties"/>
		<property name="sendemailperiodically" value="false"/>
		<property name="sendemailperiodicallytime" value=""/>
		<property name="emailPasswordHidden" value="true"/>
	</target>

	<target name="dloaddiffmaster" depends="set_properties_browser, set_timestamp">
		<!-- Sync suite to Master staging folder -->
		<property name="masterStagingPath" value="temp/scripts/staging/${ts}_${browser}" />
		<property name="ignorePattern" value=".*(svn|copied).*" />

		<sahisync	originFolder="${scriptDir}"
					destFolder="${masterStagingPath}"
					ignorePattern="${ignorePattern}"
					noise="${noise}"
					subject="${subject}"
					csvSeparator=","
					scriptExtensions="sah;sahi;js;"
					scenarioExtensions=".s.csv;xls;xlsx"
					javaDir="java"
					exposedJavaClasses="exposed_classes.txt">
			<node host="${masterhost}" port="${masterport}" />
		</sahisync>

		<sahiload	noise="${noise}"
					noiseBrowserType="${noiseBrowserType}"
					min="${minThread}"
					max="${maxThread}"
					incrementBy="${increment}"
					interval="${interval}"
					subject="${subject}"
					subjectRepeatCount="${repeatCount}"
					sahihost="${masterhost}"
					sahiport="${masterport}"
					baseURL="${baseURL}"
					browsertype="${browser}"
					scriptsPathMaster="${masterStagingPath}"
					subjectNode="${subjectNode}"
					scriptsPathInitiator="${scriptDir}"
					isDifferentMaster="true"
					ignorePattern="${ignorePattern}"
					sendemail="${sendemail}"
					emailtrigger="${emailtrigger}"
					emailproperties="${emailproperties}"
					sendEmailPeriodically="${sendemailperiodically}"
					sendEmailPeriodicallyTime="${sendemailperiodicallytime}"
					emailPasswordHidden="${emailPasswordHidden}">
			<!-- Scripts will be distributed across all the nodes. The nodes may or may not include the Master machine -->
			<node host="localhost" port="9999" />
			<!-- Offline Reports-->
			<report type="html" logdir="logs/temp/html/${ts}_${browser}" />
		</sahiload>

		<!-- Delete scripts from Master staging folder -->
		<sahidelete	host="${masterhost}"
					port="${masterport}"
					filePath="${masterStagingPath}">
		</sahidelete>

		<!-- Pull the HTML logs from the Master onto the Initiator machine -->
		<sahipull	sourceHost="${masterhost}"
					sourcePort="${masterport}"
					originFolder="logs/temp/html/${ts}_${browser}"
					destFolder="logs/html"
					ignorePattern="${ignorePattern}">
		</sahipull>

		<antcall target="failsahi" />
	</target>

	<target name="failsahi" depends="set_properties_browser" if="sahi.failed.${browser}">
		<fail message="Sahi tests failed on browser ${browser}!" />
	</target>

</project>

Attributes and their description

scriptDir	Specifies the path to the scripts folder. Can be the absolute path or a path relative to the userdata folder
host	Hostname of the server where Sahi is running (Can be IP as well). Leave this as localhost
port	Port on which Sahi is running
subject	Path to the subject script.
noise	Path to the noise script.
browser	The browser on which the subject script is run (can be IE, Firefox etc.)
baseurl	Specifies URL which is the starting URL for the subject and noise scripts
noiseBrowserType	The browser on which the noise script is run (default ChromeHL)
minThread	The minimum number of noise browsers to run initially
maxThread	The maximum number of noise browsers to run finally
increment	The number of noise browsers to be run incrementally after the specified interval
interval	The interval after which to increment the noise browsers by
repearCount	Number of times the subject script should be run at every noise level. It is good to run it more than 1 time to average out any random behaviour.
node	Each node attribute specifies a machine on which the noise browsers should run. Add as many node entries as there are machines to run. The nodes may or may not include the Master machine (localhost). If the Master machine is not included, the noise browsers will not be run on the Master. There can be 1 or more nodes.
report	This attribute is used to store offline report. type parameter specify the format in which the offline logs should be downloaded and logdir is the file location of the offline logs.

Command to execute the above ANT target

ant -f dload.xml