1.2. Your first performance test

We have actually executed a performance test while validating our installation. Let's take a look on it in more detail. Then we will walk through a quickstart example taken from our web pages.

1.2.1. First out-of-the box demo with httpbin.org

Performance test execution in PerfCake is driven by a so called scenario. Scenarios are by default placed in $PERFCAKE_HOME/resources/scenarios. You can place them in any location you want, or they can be even online. But just this location is searched automatically. You even do not need to specify the file extension when running a scenario. There are couple of supported formats (see Section 2.1 for a complete list) that are recognized automatically.

The scenario simply specifies how the load should be generated, where the load/requests/messages should be sent to, what the request should look like and what do you want to measure/report. You can also ask PerfCake to validate your messages or use sequences to make each request uniqe. These advanced concepts are described later.

The scenario specification is pretty self-explaining. You just need to know what are the possibilities (see Chapter 4, Reference Guide for hints on this).

  1 <?xml version="1.0" encoding="utf-8"?>
  2 <scenario xmlns="urn:perfcake:scenario:7.0">
  3   <run type="${perfcake.run.type:time}"
  4        value="${perfcake.run.duration:30000}"/>
  5   <generator class="DefaultMessageGenerator"
  6        threads="${perfcake.thread.count:100}"/>
  7   <sender class="HttpSender">
  8      <target>http://${server.host}/post</target>
  9      <property name="method" value="POST"/>
 10   </sender>
 11   <reporting>
 12      <reporter class="IterationsPerSecondReporter">
 13         <destination class="ConsoleDestination">
 14            <period type="time" value="1000" />
 15         </destination>
 16      </reporter>
 17   </reporting>
 18   <messages>
 19      <message uri="plain.txt"/>
 20   </messages>
 21 </scenario>

As you can see, the simplest scenario runs for 30000ms = 30 seconds. It generates messages/requests using 100 threads and sends them via HTTP to the server specified in a property (see below for explanation) using the POST method. Performance test results are reported to the console every 1000ms or 1 second. The content of the messages that are being send is specified in the plain.txt file.

As you can see, there are some strange construts at a few places in the scenario. These are ${property:default}. These are replaced by real values specified at the command line. If they are not specified, they are replaced by the default values (configured following the colon). If there is no default value and you do not pass the value at the command line, an empty string is used instead. These properties can be used to dynamically change the behavior of the scenario without actually changing the file.

To run the scenario, we can simply invoke PerfCake via the shell/bat script as described earlier. The only mandatory command line argument is -s <scenario name>. To provide property values we use -Dproperty=value. That's it. Try it once more.

./bin/perfcake.sh -s http -Dserver.host=httpbin.org

1.2.2. Your own quickstart

Preparing PerfCake

After getting (see Section 1.1.1, “Downloading distribution” ) and unpacking PerfCake you will have your directory with the following structure:

$PERFCAKE_HOME
├── bin/
│ ├── perfcake.bat
│ └── perfcake.sh
├── docs/
│ └── perfcake-7.x-javadoc.jar
├── lib/
│ ├──ext/
│ ├──plugins/
│ └── *.jar
├── resources/
│ ├── keystores/
│ ├── messages/
│ ├── scenarios/
│ ├── schemas/
│ └── xslt/
├── LICENSE.txt
└── log4j2.xml 

You may try to run PerfCake, you should receive output like this:

$PERFCAKE_HOME/bin/perfcake.sh
=== Welcome to PerfCake 7.5 ===
usage: ScenarioExecution -s <SCENARIO> [options] [-D<property=value>]*
 -d,--debug                                start debug JMX agent for
                                           external monitoring
 -D <property=value>                       system properties
 -dn,--debug-name <AGENT_NAME>             debug agent name in the JMX
                                           tree
 -h,--help                                 prints help/usage
 -log,--log-level <LOG_LEVEL>              logging level
 -md,--messages-dir <MESSAGES_DIR>         directory for messages
 -pd,--plugins-dir <PLUGINS_DIR>           directory for plugins
 -pf,--properties-file <PROPERTIES_FILE>   custom system properties file
 -r,--replay <RAW_FILE>                    raw file to be replayed
 -s,--scenario <SCENARIO>                  scenario to be executed
 -sd,--scenarios-dir <SCENARIOS_DIR>       directory for scenarios
 -skip,--skip-timer-benchmark              skip system timer benchmark          

The script assumes you have JDK installed and available on the system path, minimal version 1.8 is required. Please note that the system being tested is not required to run on Java 8. It might not run on Java at all!

In the bin directory you can find executable scripts for running PerfCake on Linux, Windows and Mac.

The lib directory contains application libraries. You do not have to take any care of these.

What is more interesting is the resources directory. In its subdirectories you can find sample scenarios, messages and all versions of XSD schemas for scenario files. The keystores directory is used for specific message sender, but we will not deal with it in this quickstart.

If you feel like going wild, you can download [1] the source distribution and compile it by

mvn clean package assembly:assembly 

Then you can find the binary distributions in the target directory and continue with this quickstart guide. You will also see the output of tests so you can be sure the project works fine on your system.

Configure and run

In these days, your only possibility to prepare your first scenario is an XML file. You can use your favourite editor to create this file. The structure is defined by an XSD schema that can be found under resources/schemas directory. Some of the editors are able to use the schema file to suggest you valid tags. Our future plans include providing GUI editor for Eclipse and InteliJ Idea that would allow you to create and edit scenarios, stay tuned! If you wanted to contribute, we are happy to welcome you in our community [2].

At minimum, simple scenario has to contain definitions for:

Let’s assume you need to stress your web application that has some function exposed on the following URL: http://your-app.com/cool-app , and you need to test how fast the function is. You want to generate maximum load for 10 seconds (10000 miliseconds) with 10 simultaneous clients (working threads).

If you do not have any such application at hand, you can consider using http://httpbin.org/get but be polite and do not overload their server. It is provided for free.

Now the important part comes. Create a file called http-echo.dsl ( http-echo.xml resp.) in the $PERFCAKE_HOME/resources/scenarios directory. Now insert the following DSL (XML resp.) snippet in it:

http-echo.dsl:

  1  scenario "http-echo"
  2     run 10.s with ${thread.count:10}.threads
  3     generator "DefaultMessageGenerator"
  4     sender "HttpSender" target "http://your-app.com/cool-app" method "GET"
  5  end

http-echo.xml:

  1 <?xml version="1.0" encoding="utf-8"?>
  2 <scenario xmlns="urn:perfcake:scenario:7.0">
  3    <run type="time" value="10000"/>
  4    <generator class="DefaultMessageGenerator" 
  5               threads="${thread.count}"/>
  6    <sender class="HttpSender">
  7       <target>http://your-app.com/cool-app</target>
  8    </sender>
  9 </scenario> 

You can see "${thread.count:10}" in the generator’s definition. That is a system property "thread.count" that you may set and the actual value of the property will be used. If the property is not set, the default value (10) will be used.

Now, all you need to do is to execute your new test scenario by running the following command:

$PERFCAKE_HOME/bin/perfcake.sh -s http-echo 

Please note you do not need to specify the DSL (XML) extension. Only if you used both DSL and XML variants.

Now you are running your fisrt stress test. Even if you cannot see what is going on, PerfCake sends requests to your application in many threads. The test should run approximately for 10 seconds. If you want to see some numbers (e.g. how fast your system is), you have to add one more element to your scenario to evaluate the results - the reporting.

Evaluate results

For reporting some results of your measurement, you have to configure a Reporter - an object that is capable of computing results in some way and outputing them wherever you can imagine.

Copy your http-echo.dsl ( http-echo.xml resp.) file to http-reporting.dsl ( http-reporting.xml resp.) and have it look like the listing below.

http-reporting.dsl:

  1 scenario "http-reporting"
  2    run 10.s with ${thread.count:10}.threads
  3    generator "DefaultMessageGenerator"
  4    sender "HttpSender" target "http://your-app.com/cool-app" method "GET"
  5    reporter "ResponseTimeStatsReporter" minimumEnabled "false" maximumEnabled "false"
  6       destination "ChartDestination" every 1.s name "Response Time" group "rt" yAxis "Response Time [ms]" attributes "Result,Average"
  7       destination "ConsoleDestination" every 1.s
  8 end 

http-reporting.xml:

  1 <?xml version="1.0" encoding="utf-8"?>
  2 <scenario xmlns="urn:perfcake:scenario:7.0">
  3    <run type="time" value="10000"/>
  4    <generator class="DefaultMessageGenerator" 
  5               threads="${thread.count:10}"/>
  6    <sender class="HttpSender">
  7       <target>http://your-app.com/cool-app</target>
  8       <property name="method" value="GET"/>
  9    </sender>
 10    <reporting>
 11       <reporter class="ResponseTimeStatsReporter">
 12          <property name="minimumEnabled" value="false"/>
 13          <property name="maximumEnabled" value="false"/>
 14          <destination class="ChartDestination">
 15             <period type="time" value="1000"/>
 16             <property name="name" value="Response Time"/>
 17             <property name="group" value="rt"/>
 18             <property name="yAxis" value="Response Time [ms]"/>
 19             <property name="attributes" value="Result,Average"/>
 20          </destination>
 21          <destination class="ConsoleDestination">
 22             <period type="time" value="1000"/>
 23          </destination>
 24       </reporter>
 25    </reporting>
 26 </scenario>

Adding the reporting section you let your scenario to log results to some destination - in our case to the PerfCake’s console. Output will be provided every 2 seconds (2000 miliseconds).

Try to run the scenario again by the following command:

2016-06-14 22:01:56,155 INFO  {org.perfcake.ScenarioExecution} === Welcome to PerfCake 7.5 ===
2016-06-14 22:01:56,386 INFO  {org.perfcake.scenario.ScenarioBuilder} Scenario configuration: file:/home/perfcake/perfcake-7.5/resources/scenarios/http-reporting.xml
2016-06-14 22:01:56,455 INFO  {org.perfcake.util.TimerBenchmark} Benchmarking system timer resolution...
2016-06-14 22:01:56,456 INFO  {org.perfcake.util.TimerBenchmark} This system is able to differentiate up to 334ns. A single thread is now able to measure maximum of 2994011 iterations/second.
2016-06-14 22:01:56,468 INFO  {org.perfcake.message.generator.DefaultMessageGenerator} Starting to generate...
[0:00:01][84 iterations][17%] [141.868302 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 136.702416 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 170.70281994047613 ms]
[0:00:02][155 iterations][27%] [140.435482 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 135.436574 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 157.30522732903225 ms]
[0:00:03][225 iterations][37%] [136.248079 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.629032 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 151.3000900355556 ms]
[0:00:04][292 iterations][47%] [146.018769 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.629032 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 151.79159557191784 ms]
[0:00:05][360 iterations][57%] [140.239096 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.629032 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 150.78039243888892 ms]
[0:00:06][430 iterations][67%] [140.608011 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.077543 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 149.0632376767442 ms]
[0:00:07][500 iterations][77%] [150.971978 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.077543 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 148.20699049399997 ms]
[0:00:08][571 iterations][87%] [136.759408 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.077543 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 147.44373894921188 ms]
[0:00:09][641 iterations][97%] [137.870229 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.077543 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 146.7737520499219 ms]
2016-06-14 22:02:06,508 INFO  {org.perfcake.message.generator.DefaultMessageGenerator} Reached test end. All messages were prepared to be sent.
2016-06-14 22:02:06,509 INFO  {org.perfcake.reporting.ReportManager} Checking whether there are more results to be reported...
[0:00:10][662 iterations][100%] [142.601692 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [Minimum => 134.077543 ms] [failures => 0] [RequestSize => 0.00 B] [Average => 146.55668455286997 ms]
2016-06-14 22:02:06,620 INFO  {org.perfcake.message.generator.DefaultMessageGenerator} Shutting down execution...
2016-06-14 22:02:06,620 INFO  {org.perfcake.ScenarioExecution} === Goodbye! ===         

The warmUp attribute you can see in the results determines the mode of the test. In our example we do not wait for the server to warm up so the attribute is set to false all the time.

ChartDestination specified in the scenario produces a chart report with the chart similar to the following that can be found under $PERFCAKE_HOME/perfcake-charts directory.

Your first performance test chart report

Figure 1.1. Your first performance test chart report