Test log lines

Some times the only thing we are interested in testing is weather a log line correctly gets logged. This is not the most optimal test but sometimes its required... in cases where log lines are getting monitored by a monitoring agent and we want to make sure correct format gets logged etc.

The trick is to mock the Log Appender and then capture the logs on that appender in a safe way. Once these log lines are captured you can verify them to your hearts content....

A fully working example of this is given at: https://github.com/njaiswal/logLineTester

Following snippet shows how to achieve this:

Run background server during integration tests

Most of the times during integration testing we have to run the built java code in server mode and run client junit/testng tests against the server. Mostly this has to be also done on the CI Jenkins server as well. Following post will go through some of the techniques to achieve the same.

Assuming the project is built in maven, run the integration test server (your application) in the pre-integration-test phase using maven-ant-run plugin.



Above maven plugin will run your main class in a forked jvm process in background and then you can run your junit/testNg integration tests against this server.



One of the issues that you will face is to stop the server once integration tests are complete.
For this you can create a method call to exit the test server if possible or exit the test server after a time interval. However this needs to be implemented server side. If you only run integration tests on Jenkins, Jenkins will make sure to kill all the pids that were created during a integration test run and keep integration test environment clean.

Happy testing...

Skip TestNG test based on a condition

Sometimes you might want to skip tests based on a condition. Following snippet shows a simple way to do it by throwing a testngSkipException. This integrates seamlessly with test reports and skipped tests are counted displayed correctly in it.

Custom logger for TestNG

Another small post on TestNG custom logger. I have a TestNG test suite that runs for hours doing through integration tests. Now at the end of the test run I can see testNG reports and see what failed and what passed, however I get no response during the test run as to if all tests are failing due to some configuration issue or its a normal test run. This is happening due to a fact that its basically only 1 test with DataProvider providing different test parameters. If you have separate tests then yes you will get messages for each test.

Anyhow I wanted a custom TestNG logger which logs a message that I understand at the end of each test and here is a way to do so.


Extend TestListenerAdapter class and override a few methods.


Add a custom Listener to your testng xml config:

and wola I get the much needed indication on the std out:

Increase JBehave story timeout

Small post on how to increase the default story timeout in JBehave. This might be required if you have a long running story as I have with many Steps/Examples etc and you see error message similar to following:

Story my.story duration of 301 seconds has exceeded timeout of 300 seconds

or

STORY CANCELLED (DURATION 301 s)

The solution is the increase the story timeout in your maven config (default is 300s). Note this will only take effect when you run integration-tests via mvn command and not when you run the stories via Eclipse/IntelliJ which will bypass your pom.xml and other maven config.

Full pom.xml with timeout setting can be found here: https://github.com/jbehave/jbehave-core/blob/master/examples/threads/pom.xml

Relevant portion below:

restIT: Testing Framework for REST based web services

I would like to call this testing framework restIT (REST Integration Test).

This test framework will give a simple way to verify that REST webservice is responding with right JSON responses. All you need to provide is a couple of yml config files declaring the REST urls and JSON schema files for the expected JSON responses.

Adding new tests should not take more than 10 mins of your time. Framework is flexible enough for you to add more advanced features such as Authentication headers and complex JSON response validation which cannot be done via JSON schema files.

Project is hosted on github at: https://github.com/njaiswal/restITframework

Steps to make use of this framework.

• Git clone the project.

• Import the project in eclipse or your fav ID as existing maven project.
• Replace src/test/resources/testcases.yml file with your own.
• Make sure you change baseurl to map to base url of your rest service.
• Add services which you want to test with/out prameters, this depends on your rest service. See examples in the file.

• For each of the test case you have added in the above step, you need to add a schema map. Replace file src/main/resources/schemaMap.yml with your own.

Each schema map in this yml file has 3 fields:

• id: This is just a name.Appears in logs/reports.
• file: This is the JSON schema file path
• regex: If the URL matches this regex, response of this URL will be validated againts above JSON schema file
• In src/test/resoruces/schemas directory add all JSON schema files defined in schemaMap.yml file.
• See examples in the git project.
• Find more details on how to write schema files at: http://json-schema.org/
• You can add as much or as little JSON response validation as you want depending on what json-schema provides out of the box. Think of this like xml schema for xml responses.
• If you want to do more complicated response validation, extended the response validation code, or if you have specific requests let me know and I might accommodate it in my free time.

Steps to run the test framework and view reports.

Easy: 

View reports at: target/surefire-reports/index.html

Make sure to check the 'Reporter Output' tab for any JSON schema validation error reports. It shows all json schema vaildation errors pretty printed.

Enjoy. Let me know if you have any questions/suggestions.

Small but useful utilities

Many times working inside corporate env you cannot use online utilities like url decoders, base64 decoders etc where you copy paste the encoded string into a webpage and wola its gets decoded on the browser. Though these websites say all decoding is done on the client side, which is true most of the times, its always smart not to copy paste company data on to random websites.

I have a set of small utilities which i have shared here, might prove useful to you.

• Decode encoded URL string. Useful for me to read encoded urls and make sense of it at times.
  
  
• Base64 decoder. Useful when you want a quick decoding requirements.
  
  .
• json to yml: Useful tool to quickly convert json to yml
  

These are  very simple but useful utilities, obviously will need your perl modules and setup ready to run them.

Making Unit and Integration Tests coexist - Maven/TestNG

Recently I had to make unit and integration tests coexist in the same maven project (pom), up-till then integration tests resided in a separate project (nice and easy).  However ideally these should co-exists in same project to make versioning/branching possible.

This small post I will share the steps taken to make this possible.

Technologies used here are:

- Maven
- TestNG

Goals: 

- mvn test should only run unit tests
- mvn integration-test should only run integration tests so that they do not cause issues building the project and are not included in the maven lifecycle for mvn install

Steps:

• Make 2 different testng.xml files called:
• ITtestng.xml (for integration tests)
  
• UTtestng.xml (for unit tests)
  
• Include maven-failsafe-plugin in your pom.xml to run just the integration test cases. Please not maven-failsafe-plugin is designed to run integration tests while the Surefire plugins is desinged to run unit test more info on this can be found here.
  
• That is it! 
• Now your mvn test command will run only the unit tests. Also mvn install will not run integration tests and cause the build to fail
• mvn integration-test will run both unit and integration tests.

wsdl2java

This is a very short post on installation and command line interface for wsdl2java. I wanted to explore the command line interface and understand the tool more since calling wsdl2java from maven is somewhat magic to me.

Installation

• Download apache-cxf from http://cxf.apache.org/download.html
• Extract .zip file and put location of extracted bin directory in your PATH
  E.g. I did the following
  export PATH=$PATH:/Library/Apache-CXF/apache-cxf-2.7.7/bin
• Try out wsdl2java on command prompt. This should return the usage.

WSDL

• Get hold of the wsdl, in this blog we are using airport-information which is a slightly non-trivial wsdl from http://free-web-services.com/

Generate fully annotated Java code from WSDL

• Run following to generate Java code
  
  

More on wsdl2java help can be found online.

In future blogs I plan to use this wsdl and make a proper application out of it to get airport information from around the world, if I get free time ;-)

Jenkins CI - Intro

I find CI tools like Jenkins extremely helpful in any project which is serious about quality. It enforces the whole team into a mindset were everyone works towards a common goal of keeping Jenkins report clean!

This blog will touch upon getting Jenkins up and running with couple of jobs.

Installing Jenkins

I would be running Jenkins under Tomcat container. I tested that my tomcat container is up and running by going to http://localhost:8080. If its not running for you, please get it running first.

Once tomcat is up, place jenkins.war file (which you can find on Jenkins website) in the webapps directory in my CATALINA_HOME and that's done.

Jenkins should be up and running at http://localhost:8080/jenkins

Configuring Jenkins

There are a few recommended configuration items that I quickly setup. All this configuration can be done by clicking 'Manage Jenkins'.

Set up JDK

Add your JAVA_HOME path.

 

Setup up Maven

Add your MAVEN_HOME path.

There are other configuration items like Ant, Email etc however above 2 should get us started straight away.

Plugins

Since my test projects are on git hub and Jenkins does not come by default with Git, I need to install the Git Plugin, installing which is super easy via the  'Manage Jenkins' -> 'Manage Plugins' tab. Just search for 'Git Plugin' and select install.

 

Setting up first job

Setting up a job is pretty easy as well. Select 'New Job' and click 'Build a free-style software project' which will allow me to choose the git repository for my projects.

• Give the Git Repository URL.

• Select a Build Trigger which I want to be random minute every hour.

• Invoke top-level Maven target. 
  

•  Add a post build action of publishing Junit test reports.
  

Save changes and in some time your first job would be run buy Jenkins and a fantastic dashboard view of all my projects is available to me with a setup of barely 15 mins. This to me is great power of Jenkins.

This blog does not contain usage of any advanced features of Jenkins, but show that even with few basic setup steps Jenkins can be used to make big leaps in get CI with your existing automated test cases.

Cucumber-JVM BDD for RESTful API

In this post I will like to work out a example in Cucumber which takes on a very familiar setup these days in any kind of software organisation: Acceptance testing of a RESTful web service.

For anyone who has been following my posts, I tend to make use of online services in my testing examples. This time is no different and the goal is to:

Write set of acceptance tests for http://imdbapi.org's RESTful API

And I am going to use Cucumber-JVM achieve the objective. As the name suggest imdbapi.org exposes IMDB data via a RESTful api and we are going to search for movies! Try out following to understand the scope of this testing:

Setup Java Project with Maven

Put all dependencies in pom.xml 

Since I have already coded up the test case my directory structure is looking like following:


All or most of the fun in Cucumber is in writing the .feature file which is in Gherkin which is the language which many BDD tools like Cucumber understands.

Lets start looking my .feature file. It contains 2 Scenarios. First one is a simple scenario which represents 1 test case, while the second one is actually a set of test cases with different input and expected output. There are a lot of other ways which one can represent their set of acceptance test cases in Gherkin but for sake of example I have used the following and this file should be self explanatory which is what Gherkin is good at.



Now create a simple RunImdbTest.java file which will be used to run test cases in .features file via JUnit.



At this point you can run 'mvn test' and let Cucumber print out the snippet of code that you need to include in your Step Definition file where basically the conversion between plain english test statements happen to actual java code behind. In our case this file is ImdbSteps.java

ImdbStep.java

This Step Definition file contains a set of annotations with capture regular expressions that drive the whole automation between plain english statements and java code behind to do the job.

JsonReader.java

This helper class has to logic to http get the movie titles and convert the response in JSON objects for comparison in Step Definition class.

Finally our test cases are ready and we can run them using 'mvn test' command. This run will also create the green cucumber test report which all product managers like to see.

All project files are uploaded on github.

Run Selenium tests in parallel on multiple browsers

This blog is follow up from my previous one where I worked out an example to take screenshots on test failure. In this blog I am going to attempt to run the same tests however on multiple browsers and in parallel.


To achieve multi-browser parallel run of the test cases:

• I have updated the testng.xml and added config to run the tests in parallel with browser_type parameters each with its own thread, one for firefox and one for chrome.
• Updated pom.xml to start using testng.xml file to kickoff the test cases.
• Updated WebDriverManager class to be thread aware and return correct WedDriver (Firefox or Chrome) instance so that in case of test failure screenshot of correct browser can be taken.
• Minor tweak in the test case so that its aware of which browser type to run with.

The entire project is uploaded on github for reference: https://github.com/njaiswal/council-tax-band

Right, lets get to the nuts and bolts now.

testng.xml

Two test cases in the suite, TestNG will run these 2 test case with a threadpool size of 2, which means that both browsers will fire-up at almost the same time and go through the test cases in parallel which saves us time!

pom.xml (snippet)

Above pom.xml snippet tells maven to start using testng.xml to kick off the test cases. Please refer full pom.xml on github

WebDriverManager

WebDriverManager is a bit more meaty than previous blog. Instead of keeping the WebDriver instance as static variable it has a static HashMap in which it will store thread Id as the key and the WebDriver instance as value. Since our tests are now going to run on more than 1 thread we need this arrangement so that WebDriverManager knows which instance to return to onTestFailure() method based on the Thread Id.

CTaxBandTest (snippet)

As you can see above I have used the @Parameters annotation along with @BeforeClass. This basically fetches parameter browser_type from the testng.xml file and passes it as method parameter to onTimeSetUp() method. This is a key step where we call WebDriverManager.startDriver() and get correct browser instance created. As mentioned above, WebDriverManager will store the instance reference in a HashMap with thread Id as key so that any future calls to WebDriverManager.getDriverInstance() can identify which browser instance to return.

Reports

Run the tests using 'mvn test' command and you should see 2 browser instances (Firefox and Chrome) getting started at the same time and the tests being run in parallel. 

Two reports get generated (one for each browser type) with total of 4 test cases being run and more importantly both contain correct screenshots of test failures.

Again the entire project is on git hub for reference, happy testing!

Selenium: Take screenshot on test failure

Going ahead on the theme of testing publicly available websites using selenium, in this blog I want to work out an example which touches on following points:

    •    Run Selenium WebDriver tests under TestNG
    •    Use the following best practices:
    ◦    Page Object Model
    ◦    UI Mapping
    ◦    Data Driven Testing
    ◦    And most importantly take screenshot on an event of test failure
Now to do all of the above I need a publicly available website that sounds interesting enough and devise testing around it. So here we go...

Test the DirectGov Council Tax Band website, given postcode, first line of address and an expected band.
The site that we are using is  http://www.voa.gov.uk/cti/InitS.asp
 

Setup Java project with Maven 

Now import the maven project in your favorite IDE. I am using Eclipse.

Since we are going to use selenium with testng instead of junit my pom.xml looks like following.

Test and Helper Class

Now that we have got a Java project set up, lets get cracking. Since I have already done my code before writing this blog, my final directory tree looks like following.

Let me add a bit of context around what is the role of each file in the directory tree above is. Later I will add more details about each of the relevant files.

• CTaxBandTest.java - This is the testng file that contains all the test cases just by the virtue of keyword 'test' in the file name.
• CTaxBandSearch.java - Page Object Model Class. The Page Object Design Pattern provides the followingadvantages.
• There is clean separation between test code and page specific code suchas locators (or their use if you’re using a UI map) and layout
• There is single repository for the services or operations offered by thepage rather than having these services scattered through out the tests.
• WebDriverManager.java - This class will manage the singleton WebDriver instance that will be used in our tests. The importance of this manager class will become relevant when we go through the details of how to get screenshot on test failure.
• ScreenShotOnFailure.java - Contains code to capture and store screenshot on event of test failure at correct location so that these screenshots is available via test reports.
• ui_mapping.properties - UI Mapping file. Contains all the UI locators in one file for ease of modification.

 

CTaxBandTest.java

In the test case above we have used various testng annotations. The important one is @Listeners which basically defines one or more of users classes, which implement some interfaces that TestNG provides, so that when TestNG raises certain events it will look for all “classes” which are basically looking for such events and call the respective event handlers in them. In this case we are using it to define handler for onTestFailure so that we can capture the screenshot and include it in testng report.
We are also using the @DataProvider annotation to make our test case data driven which basically means that we have more than 1 data set points to run through same test case.

WebDriverManager.java

This is a static helper class which manages the only Selenium WebDriver instance we are using for all test cases. This makes access to this webdriver instance very easy later on when we want to take a screenshot on event of test failure. 

CTaxBandSearch.java

This is the Page Object Model class which knows how to navigate the website page which we want to test. As you can see I have also made use of UI Mapping best practice here and get all locators from a properties file.

ScreenShotOnFailure.java

And lastly the listener class which will override the onTestFailure handler of testng and help us get the crucial screenshot when a test case fails.

We can get hold of the webdriver instance very easily since its now managed by a WebDriverManager class. Above class also demonstrates how to include the screenshot link in the test reports.

Run the test

Now that we have all the bits and pieces in place lets run our test case. You might have not noticed but I have deliberately included a council tax band incorrect in our @DataProvider which means that 1 test case will pass and 1 fails with a screenshot!

The reports are generated in target/surefire-reports/ directory and looking at the reports... hurray screenshots at last!!!

This has been kind of a long post, but I did want to touch on a few points in one go and glad that its out there now.

Selenium WebDriver - Maven

This post I am going to share the basics of setting up and running a simple but working Selenium WebDriver example. The hypothetical requirement this example is trying to accomplish is following:

Print out all addresses on rightmove.co.uk for 3 bedroom properties for sale in Reading under £175K

Requirements looks simple. Lets get started.

Setup Java project using Maven

Assuming you have maven installed on your system, run the following command to setup a generic java project.

Update pom.xml to 

• Include selenium-firefox-driver and selenium-support dependencies.
• Generate a executable jar file. This example pom.xml assumes that your main class would be named PropertyAddressFinder.

Selenium

Import this maven project in your favourite IDE and add PropertyAddressFinder class. Java code below  should be self explanatory. It uses Selenium FirefoxDriver to open the website and search for properties .  

Run the Project

Now that we have the bits and pieces in place, we can let maven do the magic and make us a executable jar file, refer pom.xml's build section above to see how we achieve this. Executing the jar file via command line will open a firefox browser and print out whatever properties it finds.

Conclusion

Above blog demonstrates a very basic usage of selenium webdriver to automate web based tasks that cannot be otherwise achieved via well known api's.  

a good unit test...

This is my first blog in years and I would like to share my experience on unit testing.

Lets start with the obvious, so what is a unit test?
According to wikipedia a unit test is a method by which individual units of software are tested to determine if they are fit for use. In my humble opinion some other characteristics of a good unit test are as follows:

• Repeatable and Clean
• Easy to debug and maintain
• Fast

Repeatable and Clean
Make sure your unit test case can be run multiple times without failing. It should not assume any special setup nor should it leave behind any state and always cleanup at the end.

Obviously if your unit test case is a part of ordered test suite where next in the order test case depends on the state created by previous tests you are allowed to leave state, however the test suite as whole should be repeatable and do cleanup towards the end.

Easy to debug and maintain 
Ideally a short stdout message should indicate why the test failed however sometimes its recommended to have appropriate amount of logging built into your unit test. Maintainability of test is of huge importance. It should be easy to identify what part of code/feature a unit test case is testing so that if it fails, its easy for a developer to verify if the test is outdated or the code is wrong.

Fast
I hate slow tests. Slow tests usually indicate a badly written test which is depending on external data sources. Obviously if the operation your unit test is testing is slow, your test itself will be slow, but more often its not the reason.