Apache JMeter
JMETER TERMINOLOGY
Before we go further it would probably be helpful to discuss some basic terminology.
Thread: For all intents and purposes, a thread can be thought of as a user.
Thread Group: Think of a thread group as the scenario or use case being tested. A test script that has a user log in to webtop, execute a search, and log out might be one thread group. During a test there might be one or several hundred users executing a thread group. The thread group might contain conditional logic that loops through parts of the web application. I use the words "script" and "thread group" interchangeably. Script seems more intuitive.
Test plan: A test plan is one or more thread groups. In a previous example we discussed a thread group that performs a search in webtop. Suppose we also had a thread group that performed a checkout on a randomly-selected object in the user's default cabinet and another that simply browsed through all of the folders in a given cabinet. All three thread groups could be included in a test plan to simulate the various activities different types of users might perform over a period of time.
Sampler: A sampler is something that sends a request to a server. JMeter comes with many different types of samplers.
Listener: A listener listens to the responses generated by samplers. Listeners can be used to compile the results of the test run and to check that a sampler generates the expected response.
Assertion: An assertion is used to specify what a response should or should not contain. There are several different types of assertions depending on what kind of response you are expecting and what sort of check you'd like to perform. If you want to specify that a response should be of a given size, you would use the Size Assertion. If you want to specify that a response should return in a given amount of time you would use the Duration Assertion.
Logic Controller: Logic controllers provide a mechanism for controlling the flow of the thread group. Adding a logic controller to your thread group is like adding if-then or do- while logic to a program. Your test script can make run-time decisions about how to execute. Suppose you want to open every object in a list but you do not know how many objects will be in the list ahead of time. A "Loop Controller" could be used to execute an action (opening an object) a certain number of times without requiring you to hard-code the requests into the script.
Workbench: The Workbench is like a working area. When recording scripts, Jmeter places the automatically-generated samplers under the workbench. They will not get executed as part of a thread group until you move them from the workbench to a thread group.
WHAT IS JMETER?
As the name indicates Jmeter is one of the Java tools which is used to load testing client/server applications. Earlier it was used for testing Web Applications only however now-a-days its being used for other test functions. It is typically used to measure performance and to load test functional behavior of client-server applications. It can also handle FTP and JDBC requests. It is absolutely free and can be modified at ease as its an open source.
The important functionalities of Jmeter is that a heavy load on a server can be stimulated by using it, not only on a server but also a heavy load on a network or object to test its strength under different load types. A graphical analysis of performance can also be done by using Jmeter or the behaviour of your server/script/object can also be tested under heavy concurrent load.
The JMeter's main window is displayed below which is a Swing application.
As shown in the figure above, there are two panes of the user interface. The elements used in our testing is shown by the left pane. Here we are only concerned with the Test Plans as there were two sub-elements earlier that are Test Plan and WorkBench. Furthermore an element can be added to a node on right-clicking and can be removed also by selecting the element and then choosing the remove option by right-clicking on it. Whereas the details of each of the elements is shown by the right pane.
JMeter supports creation of various types of test plans which are:
1.Web Test Plan
2.Database Test Plan
3.FTP Test Plan
4.JMS Test Plan
5.LDAP Test Plan
6.Webservice Test Plan
7.Monitor Test Plan
Load Testing
Jmeter
JMETER PROXY STEP-BY-STEP
This short tutorial attempts to explain the exact steps for JMeter's proxy. For those new to Jmeter, one easy way to create a test plan is to use the Proxy. What the proxy does is it records the requests sent to the server.
Basic Proxy Instructions
1.start jmeter with jmeter.bat on windows and jmeter.sh on unix
2.select “test plan” on the tree
3.right click on the “test plan” and add a new thread group: add -> thread group
4.select the thread group
5.right click “add -> config element -> Http Request Defaults”
6.Protocol – enter “HTTP”
7.Server name – enter “jakarta.apache.org”
8.Path – leave blank
9.Port number – enter “80”
10.select workbench
11.right click on workbench and add the Http proxy: add -> non-test elements -> Http Proxy Server
12.Port field - Enter “9090”
13.Target Controller – click on the drop down and select “test plan > thread group”
14.Click the “add” button in “Patterns to include”. This will create a blank entry.
15.Enter “.*\.html”
16.Click the “add” button in “Patterns to exclude”. This will create a blank entry.
17.Enter “.*\.gif”
18.Click the “start” button at the bottom
19.Start Internet Explorer, but do not close Jmeter.
Note: make sure the include and exclude patterns are correct. Below are some common patterns for images and types of pages.
.* - all
.*\.png – png images
.*\.gif – gif images
.*\.jpg – jpeg images
.*\.php
.*\.jsp
.*\.html
.*\.htm
.*\.js
As a general tip, it is a good idea to set the homepage for your browser to a blank page. This way, it reduces the number of unwanted pages Jmeter records during the session. It is a good idea to try the proxy with different sites and get comfortable with different filtering patterns.
20.From the tool bar, click “tools -> internet options”. This should bring up the options.
21.Select the “connection” tab
22.Click “lan settings” button near the bottom.
23.On the connections tab, check “Use a proxy server for your LAN”. The address and port fields should be enabled now.
24.Address – enter “Localhost” or the IP address of your system
25.Port – enter “9090”.
26.Click “ok” button
27.Click “ok” button again. This should return you to the browser
28.In the “Address” bar at the top, enter “http://jakarta.apache.org/jmeter/index.html” and hit the “enter” key.
29.Click on a few links on Jmeter's page.
30.Close internet explorer and bring up the Jmeter window.
Reviewing the Test Plan
Expand the thread group and there should be several samplers. At this point, the test plan can be saved as is. If you forget to add default http request settings, you will have to manually delete the servername, and port. In this sample, there aren't any default request parameters. If a particular request parameter is required by all pages, the request defaults is where one would add the entries.
1.Select “thread group”
2.Right click “add -> listener -> aggregate report” to add an aggregate listener. The aggregate listener will show some basic statistics.
3.Select “thread group”
4.Number of threads – enter “5”
5.Ramp up Period – do not change
6.Loop count – enter 100
Running the Test
At this point, we are ready to run our test plan and see what happens. Save the test plan. When you're ready to run the test, there's two ways:
1.Run -> Start
2.CTRL – R
Before you start the test, select “aggregate report”. As the test runs, the statistics will change until the test is done. At the end of the test, the aggregate report should look like this.
While the test is running, in the upper right-hand corner, there should be a green square. When the test is done, the box should be grey.
SAMPLERS
Samplers perform the actual work of JMeter.
FTP Request
This controller lets you send an FTP "retrieve file" request to an FTP server. If you are going to send multiple requests to the same FTP server, consider using a FTP Request Defaults Configuration Element so you do not have to enter the same information for each FTP Request Generative Controller.
Control Panel
Parameters
Attribute
Description
Required
Name
Descriptive name for this controller that is shown in the tree.
No
Server Name or IP
Domain name or IP address of the FTP server. JMeter assumes the FTP server is listening on the default port.
Yes
File to Retrieve From Server
Path and name of the file to retrieve.
Yes
Username
FTP account username.
Usually
Password
FTP account password.
Usually
FTP Request Defaults
Building an FTP Test Plan
In this section, you will learn how to create a basic Test Plan to test an FTP site. You will create four users that send requests for two files on the O'Reilly FTP site. Also, you will tell the users to run their tests twice. So, the total number of requests is (4 users) x (2 requests) x (repeat 2 times) = 16 FTP requests. To construct the Test Plan, you will use the following elements: Thread Group , FTP Request , FTP Request Defaults , and Spline Visualizer .
This example uses the O'Reilly FTP site, www.oro.com. Please be considerate when running this example, and (if possible) consider running against another FTP site.
Adding Users
The first step you want to do with every JMeter Test Plan is to add a Thread Group element. The Thread Group tells JMeter the number of users you want to simulate, how often the users should send requests, and the how many requests they should send.
Go ahead and add the ThreadGroup element by first selecting the Test Plan, clicking your right mouse button to get the Add menu, and then select Add --> ThreadGroup.
You should now see the Thread Group element under Test Plan. If you do not see the element, then "expand" the Test Plan tree by clicking on the Test Plan element.
Next, you need to modify the default properties. Select the Thread Group element in the tree, if you have not already selected it. You should now see the Thread Group Control Panel in the right section of the JMeter window
Start by providing a more descriptive name for our Thread Group. In the name field, enter O'Reilly Users.
Next, increase the number of users to 4.
In the next field, the Ramp-Up Period, leave the the default value of 0 seconds. This property tells JMeter how long to delay between starting each user. For example, if you enter a Ramp-Up Period of 5 seconds, JMeter will finish starting all of your users by the end of the 5 seconds. So, if we have 5 users and a 5 second Ramp-Up Period, then the delay between starting users would be 1 second (5 users / 5 seconds = 1 user per second). If you set the value to 0, then JMeter will immediately start all of your users.
Finally, enter a value of 2 in the Loop Count field. This property tells JMeter how many times to repeat your test. To have JMeter repeatedly run your Test Plan, select the Forever checkbox.
In most applications, you have to manually accept changes you make in a Control Panel. However, in JMeter, the Control Panel automatically accepts your changes as you make them. If you change the name of an element, the tree will be updated with the new text after you leave the Control Panel (for example, when selecting another tree element).
Adding Default FTP Request Properties
Now that we have defined our users, it is time define the tasks that they will be performing. In this section, you will specify the default settings for your FTP requests. And then, you will add FTP Request elements which use some of the default settings you specified here.
Begin by selecting the O'Reilly Users element. Click your right mouse button to get the Add menu, and then select Add --> Config Element --> FTP Request Defaults. Then, select this new element to view its Control Panel.
Like most JMeter elements, the FTP Request Defaults Control Panel has a name field that you can modify. In this example, leave this field with the default value.
Skip to the next field, which is the FTP Server's Server Name/IP. For the Test Plan that you are building, all FTP requests will be sent to the same FTP server, ftp.oro.com. Enter this domain name into the field. This is the only field that we will specify a default, so leave the remaining fields with their default values.
The FTP Request Defaults element does not tell JMeter to send an FTP request. It simply defines the default values that the FTP Request elements use.
Adding FTP Requests
In our Test Plan, we need to make two FTP requests. The first one is for the O'Reilly mSQL Java README file (ftp://ftp.oro.com/pub/msql/java/README), and the second is for the tutorial file (ftp://ftp.oro.com/pub/msql/java/tutorial.txt).
JMeter sends requests in the order that they appear in the tree.
Start by adding the first FTP Request to the O'Reilly Users element (Add --> Sampler --> FTP Request). Then, select the FTP Request element in the tree and edit the following properties
1.Change the Name to "README".
2.Change the File to Retrieve From Server field to "pub/msql/java/README".
3.Change the Username field to "anonymous".
4.Change the Password field to "anonymous".
You do not have to set the Server Name field because you already specified this value in the FTP Request Defaults element.
Next, add the second FTP Request and edit the following properties
1.Change the Name to "tutorial".
2.Change the File to Retrieve From Server field to "pub/msql/java/tutorial.txt".
3.Change the Username field to "anonymous".
4.Change the Password field to "anonymous".
Adding a Listener to View/Store the Test Results
The final element you need to add to your Test Plan is a Listener . This element is responsible for storing all of the results of your FTP requests in a file and presenting a visual model of the data.
Select the O'Reilly Users element and add a Spline Visualizer listener (Add --> Listener --> Spline Visualizer).
Regression Testing With JMeter
As a general rule, web applications should be tested thoroughly. Unfortunately, in practice, testing is spotty and often overlooked. As an alternative to manual testing create small PHP scripts to parse pages. Run them regularly to decide if last changes did more harm then good. That's what regression testing is all about, checking to make sure your application hasn't taken steps backward. But it always seemed like this particular problem deserved a more elegant solution. That's where JMeter comes in. JMeter is slated as a "load tester", but it delivers much more than that. Originally developed to test load test Jserv (the forerunner of Apace Tomcat), JMeter has since expanded to load test FTP servers, database servers as well as java servlets and objects. Of particular interest is JMeter's ability to run regression tests on web applications.
To start JMeter, locate and run the jmeter.bat file. When it starts up you'll see two items in the left tree. If you right click on the "Test Plan" item you'll see all the modules available for test. The first element to add to test is the "Thread Group". The Thread Group simulates users. Since we're not doing performance testing right now, we'll set the number of threads and loops to 1. The next element we'll need is a Sampler. The Samplers control the requests made to your web application. Right clicking on the Thread Group icon, we add a "HTTP Request" off of the Sampler menu. Now if you click on the HTTP Request icon we can set it up to make requests to our application.
We don't have a web application handy, so we'll just make a pseudo application like so:
JMeter will test for the existence of the echoed text. In practice, the echoed text should be out of the database, or another choke point in your code. Now, we'll configure a "Response Assertion" element to ensure the text "This is data" is present on the page (Note that you can use regular expressions in the assertion elements for more advanced validation). If the data doesn't show up on our page, the assertion element will mark the test as a failure.
Then, using the "Not" checkbox we'll add another Response Assertion to make sure that other text is not present, such as the common error strings "warning" and "Parse error".
Now, we could start the test, but we'd have no way of knowing what the results of the test were. So we add another element to the HTTP Request: a "View Results in Table" Listener. The Listener group gives feedback on your requests. This particular Listener doesn't need any additional configuration.
Finally we're ready to run our test plan. Choose "Start" from the "Run" menu to begin the test and highlight the "View Results in Table" element to see your results. You should get something similar to the following:
Of course, your run times will vary. The "Success" box should be checked if everything went as planned. If not, add an "Assertion Results" element to help debug your problem. Or, you can see the page exactly as JMeter sees it by adding a "View Results Tree" from the Listener menu. The View Results Tree shows the full request made to the page and the full response, which is very useful when debugging your test plan.
Pay special care to how you nest the elements, It does make a difference. For example, if you have a Listener element on the same level in the tree as 2 sampler items, the Listener will get results for both of the samplers. While if the listener was added as a child item of the Sampler, the Listener would only get results from its parent element. The same can be said about assertion elements. So our generic assertion that looks for common error strings on any page can be added on the same level as the samplers. While the data specific assertions should be a child of the page they check.
You're already starting to see how the complexity of these tests can grow exponentially on large applications. When you have one sampler for each module in your web application, the tree fills up fast! One element that can help keep your test plan manageable is the "HTTP Request Defaults" element. Instead of filling in the server and port options on each Sampler, you can leave them blank and the sampler will automatically use values from a HTTP Request Defaults element on the same or higher level in the tree. This can save gobs of time not only when setting the plan up, but also if the application ever moves to a new server.
Why use Jmeter?
Automated testing saves time and money
JMeter is flexible:
many interface options
suitable for Unit, Functional, Regression and Load testing
JMeter files are simple XML
JMeter is open source:
Mature
Strong community
Constant updates and improvements
Free!
Types of testing
Unit: specific test against documentation to a particular piece of code, such as a single class
Functional: ensuring an application or feature works as expected and fulfils the specifications
Regression: ensuring an application still operates as expected after changes to the code
Load: measuring the performance of an application when under heavy usage
Usability: measuring how easy an application is to interact with
Defining your tests
Use your documentation
Functional requirements
Release notes
Sales literature
Always keep your tests small and simple
Always make sure your tests can be repeated without user intervention
Configuring a new functional test
Open JMeter (a new test is the default behaviour)
Select Test Plan and make sure the “Functional Test Mode” option is checked
The “Run each Thread Group separately” option enforces the running of thread groups in order, where this is important
Add a HTTP Request Defaults element:
Enter the details for your web application:
Select Test Plan again and add a HTTP Cookie Manager element
This is essential if your application relies on sessions and/or cookies
Select Test Plan and add a Listener called Assertion Results:
This listener will show you the result, pass or fail, of each Response Assertion you create
Add a Listener called View Results Tree:
This listener will show you all the results returned by the server for every request, regardless of assertions, and will highlight any obvious failures
Add a Thread Group
Rename the Thread Group to “Recorded Actions”
Select WorkBench and add a Non-Test Element called HTTP Proxy Server:
This special element will allow you to record actions directly from a configured web browser
Make sure the Target Controller is set to your Recorded Actions Thread Group:
Save your test plan (do this regularly!!)
We recommend naming this plan something similar to: “0_starting_point.jmx”
You will use this a lot when creating other tests and the zero in the filename ensures it is always at the top of the working directory
The building blocks of Jmeter
Samplers:
These represent the actions you may perform
They are only available within the context of a Thread Group
Adding a sampler
Create a new Thread Group
Add a new HTTP Request sampler
Set the “Path” field to “/” (the root)
Because of your HTTP Request Defaults you only need to supply a path
Everything else is inherited from the Config Element of the test plan
You may override it here if you wish:
Running a test
Now we have a very simple test, we can use the Run menu to control it:
Note “Start” has a shortcut of Ctrl+R, “Stop” is Ctrl+period and “Clear All” for results is Ctrl+E
Press Ctrl+R to start your test
Test progress can be seen in the top right corner of the GUI:
Left = total number of threads to execute in this test
Right = number of threads executed so far
Viewing your test results
Once your test has run, select the View Results Tree listener:
The green “tick” icon tells us that no known errors occurred
Select the “tick” icon to see the statistical response data returned by the server:
You can also see the actual output and render it in a variety of formats, including HTML:
What if something goes bad?
Change the path of your sampler to “/foo”
Clear the results (Ctrl+E) and run the test (Ctrl+R)
Look at the View Results Tree again
Oh no, a 404!
Instead of a green “tick” there's a red “exclamation”
Adding a Response Assertion
Select Test Plan and add an Assertion of type Response Assertion
Add a pattern to test for by clicking Add
Double-click the pattern to apply text
Now clear your test results (Ctrl+E) and run your test again (Ctrl+R)
Now select Assertion Results:
As you can see, format is sampler name, followed by reason for failure
If there are no failures, the Assertion Results listener simply lists the sampler names
Using the interface, drag your Response Assertion under HTTP Request and select “Add as Child”
The assertion is now nested under that sampler
if you run the test again, the result is the same, but now the assertion only applies to that specific sampler
Change your sampler by removing the override pointing it to www.google.com
Now you if you view results/assertion results you can see no error is flagged, because the page meets the assertion criteria
