Friday 17 May 2013

Running SOAPui Tests from Jenkins/Hudson/Ant


The challenge

A while ago I came across a problem which I suspect has been tackled by a lot of people over the years: running one/many SOAPUI unit test packs from Jenkins. So far all the solutions I've seen from the web basically involve putting an exec call into your ant script, and pointing it at the testrunner cmd/sh script. This is fine and does work, but below is a slightly more complete solution which I hope is easy to just drop in and configure - plus I included some report generation stuff which is useful from a QA audit perspective if anyone wants to check what testing you've been doing.


Essentially this blog covers a simple ant build script which includes:
  • A generic way of calling soapui from ant (using a re-usable ant macrodef)
  • HTML report generation - generating a junit style html report on the result of your tests
  • A command line so you can invoke the build script and enter the test to run (although the only reason you'd not do this from the soapUI GUI is if you wanted the html report for a single test suite).
  • An “all” target (far more useful) which can be configured to run all the tests in the test suites, and can be invoked from Jenkins (which is sort of the point).

Getting Started

Before we get going I assume you have (or need to get) the following:
  1. SoapUI: I originally wrote the script using soapUI 3.6.1 so I know it's good to go with that version and presumably any version since.
  2. Apache Ant: If you wish to generate test reports you need version 1.5 or higher.
  3. AntContrib: v0.3 or higher to be added to your ant lib folder.
To start with there's a simple zip of stuff to be working with. It includes:
  1. build.xml - A sample ant build file – this is the crux of everything we'll look at.
  2. VerySimpleProject – a soap UI project with 2 test suites
  3. VerySimpleProject2 – another soap UI project with a single test suite
  4. junit-noframes.xsl – a modified verison of the xsl which comes with ant but has been altered to make it work better with the SOAPUI

Making build.xml your own

Configuuring Properties

The xml file has a number of properties which need to be configured specific to your environment and/or project. They are:
  • jar.soapui: The name of the soapui JAR file, which is sadly version specific
  • dir.soapui: Self explanatory really, this is the directory soapui is in on the machine running ant
  • dir.temp: a directory to store temporary files created during the execution. This is created upon build and deleted once the run has completed, so make sure it's specific to this purpose and not c:\. Generally ./temp is good enough so shouldn't need to be changed
  • dir.reports: a directory where the report outputs are stored (this needs to exist before the script is run)
  • dir.xsl: the directory the custom xsl for generating reports is stored in, this is by default ./xsl which shouldn't need to be altered
  • java.executable: the full path to java on this machine
  • test.report.title: The title placed at the top of your output report (passed as a parameter to the report generating xsl)

Configuring build all

Once these are set, you're nearly ready to go. All you need to do is say which tests to run. The best way of doing this is to put all the tests in the "all" target and then ant will invoke them all each time. Due to the naming convention I was using at the time this assumes a default test suite called "Full Test" and as such if you don't specify a suite name it will default to that. If you want to call other test suites you just need to name them. This can be seen in the all target example below:

So this calls:
  1. clean - a general tidy up target and creation of the temporary directory (dir.temp)
  2. runTest macro - as no suiteName property is passed to defaults to "Full Test".
  3. calls runTest with a different suite from the same sample project
  4. calls runTest with a suite from a second soapui project
  5. calls the report target. The report target:
    • generates a single html report covering all the tests which have been run (since the last clean - step 1 in this case) 
    • zips up the test runner output log/error files - in case a test fails and you want to look at these detailed logs
    • Sets the names of both the zip and the html to include the date and time the ant script started running - this is just so subsequent runs don't overwrite it
  6. calls tidyup to delete the log directory and a couple of the soapui files left over from the job.

Default target

The default target of the build.xml is "test" which will expect keyboard inputs for the name of the project file (without the .xml on the end) and the test suite. This can occasionally be useful if you want to test and report on a single report, however you might find it more useful to set the project default from test to all.

Really that's all you need to know to get this up and running... but just for completeness I'll do a quick tour of the runTest macro and the reports target if you're interested in understanding the nuts and bolts. If not then you're ready to go.

What's under the covers?

Run Test

The run test macro essentially invokes soapui, but rather than calling the cmd/sh script to do it it does it directly (hence the need to know where java was in the properties steps above). Most of this is quite obvious from looking at the code and you can play with the output, and the max memory should you need to. 

The macro does pass a number of arguments to soapUI, which are explained on their website. Of these we only use:
  • a to ensure verboseness in the output of errors
  • j to tell soapui to generate the xml which gets turned into the html report
  • s to tell it the name of the test suite to run
  • f to say where to put the outputs
Optionally, if you have any properties which are used in your tests (in our case we had a jdbc connection string as a property) you can get ant to pass these in, that way you can externalise these for running on different environments without the need to change your test scripts. These can be set for the developers environment within the soapUI project, and then ant will merely overwrite them on the continuous integration environment - cool huh?

Finally there's some tidy up in renaming the outputs to ensure that two project files with the same test suite name (e.g. Full Test) won't overwrite each other.

Reports 

The reports target is very simple, which is largely because it uses the standard junitreport functionality, and because most of the work I did here was hacking the xslt to be more soapUI friendly (describing that is too big a job for today). 

Essentially it grabs the xml files for all the test suites run since last clean (each renamed to be unique by the runTest macro above) and runs the junit-noframes.xsl on them. Unlike the standard ant one, the xsl now has three parameters:
  • runDateTime (because unlike java SoapUI doesn't include this in the output xml)
  • resultsFile - which means the html can have some text to say which zip contains the full logs
  • title - the title of the html report
Finally, all the logs are zipped into a file in the reports folder, and the html is moved from the temp folder to the reports folder, as it would be a shame for the tidyup job to delete it after we've only just created it.

Licensing

...and finally: 

The junit-noframes.xsl is under the apache 2.0 license - and my updates to it fall under the same.
The custom build.xml is open source under the GPLv3 license - hope it's useful.

6 comments:

  1. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. The problem is at the line "Java Result: 1". It appears that the Java run failed (perhaps a missing Java library or some such?) and thus when it tried to rename the file it couldn't find it. Try running the Java execution which is in the ant script on the command line and see what the error is.

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. I have a Jenkins licen to run soap ui but not sure where I have to give the licen's number.Can someone help me out where I have to give this number?

    ReplyDelete
  4. Hello. I have an issue by creating reports.
    Builds are succesful. SoupUI is working fine.
    I have configured Jenkins. But the graphs are not showing anything after i ran the task several times. What could be the issue? I stucking already several days on it.

    ReplyDelete
  5. Hi,

    While running the above steps I encountered below error. Can any one post the resolution for the same. Thanks in advance.

    java.io.IOException: Cannot run program "C:\Program Files\Java\jdk1.8.0_121\bin": CreateProcess error=5, Access is denied

    ReplyDelete