Jenkins CI - Intro

I find CI tools like Jenkins extremely helpful in any project which is serious about quality. It enforces the whole team into a mindset were everyone works towards a common goal of keeping Jenkins report clean!

This blog will touch upon getting Jenkins up and running with couple of jobs.

Installing Jenkins

I would be running Jenkins under Tomcat container. I tested that my tomcat container is up and running by going to http://localhost:8080. If its not running for you, please get it running first.

Once tomcat is up, place jenkins.war file (which you can find on Jenkins website) in the webapps directory in my CATALINA_HOME and that's done.

Jenkins should be up and running at http://localhost:8080/jenkins

Configuring Jenkins

There are a few recommended configuration items that I quickly setup. All this configuration can be done by clicking 'Manage Jenkins'.

Set up JDK

Add your JAVA_HOME path.

 

Setup up Maven

Add your MAVEN_HOME path.

There are other configuration items like Ant, Email etc however above 2 should get us started straight away.

Plugins

Since my test projects are on git hub and Jenkins does not come by default with Git, I need to install the Git Plugin, installing which is super easy via the  'Manage Jenkins' -> 'Manage Plugins' tab. Just search for 'Git Plugin' and select install.

 

Setting up first job

Setting up a job is pretty easy as well. Select 'New Job' and click 'Build a free-style software project' which will allow me to choose the git repository for my projects.

• Give the Git Repository URL.

• Select a Build Trigger which I want to be random minute every hour.

• Invoke top-level Maven target. 
  

•  Add a post build action of publishing Junit test reports.
  

Save changes and in some time your first job would be run buy Jenkins and a fantastic dashboard view of all my projects is available to me with a setup of barely 15 mins. This to me is great power of Jenkins.

This blog does not contain usage of any advanced features of Jenkins, but show that even with few basic setup steps Jenkins can be used to make big leaps in get CI with your existing automated test cases.

Cucumber-JVM BDD for RESTful API

In this post I will like to work out a example in Cucumber which takes on a very familiar setup these days in any kind of software organisation: Acceptance testing of a RESTful web service.

For anyone who has been following my posts, I tend to make use of online services in my testing examples. This time is no different and the goal is to:

Write set of acceptance tests for http://imdbapi.org's RESTful API

And I am going to use Cucumber-JVM achieve the objective. As the name suggest imdbapi.org exposes IMDB data via a RESTful api and we are going to search for movies! Try out following to understand the scope of this testing:

curl http://imdbapi.org/?title=Finding+Nemo

view raw curl cmd hosted with ❤ by GitHub

Setup Java Project with Maven

mvn archetype:generate -DgroupId=com.clearqa.test -DartifactId=imdb_rest

view raw gistfile1.txt hosted with ❤ by GitHub

Put all dependencies in pom.xml 

Since I have already coded up the test case my directory structure is looking like following:

	├── pom.xml
	└── src
	├── main
	│   └── java
	└── test
	├── java
	│   └── com
	│   └── clearqa
	│   └── test
	│   ├── ImdbSteps.java
	│   ├── JsonReader.java
	│   └── RunImdbTest.java
	└── resources
	└── com
	└── clearqa
	└── test
	└── imdb_rest.feature

view raw imdb_rest-tree.out hosted with ❤ by GitHub

All or most of the fun in Cucumber is in writing the .feature file which is in Gherkin which is the language which many BDD tools like Cucumber understands.

Lets start looking my .feature file. It contains 2 Scenarios. First one is a simple scenario which represents 1 test case, while the second one is actually a set of test cases with different input and expected output. There are a lot of other ways which one can represent their set of acceptance test cases in Gherkin but for sake of example I have used the following and this file should be self explanatory which is what Gherkin is good at.

	Feature: IMDB rest api gets
	Scenario: Get movie by title
	Given I query movie by "Finding Nemo"
	When I make the rest call
	Then response should contain:
	"""
	{
	"title":"Finding Nemo",
	"genres":["Animation","Adventure","Comedy","Family"],
	"release_date":20030530
	}
	"""
	Scenario Outline: Get many movies by title
	Given I query movie by "<title>"
	When I make the rest call
	Then response should contain "<genres>"
	Examples: No special characters in movie titles
	| title | genres |
	| Finding Nemo | {"genres":["Animation","Adventure","Comedy","Family"]} |
	| Inception | {"genres": ["Action", "Adventure", "Mystery", "Sci-Fi", "Thriller"]} |
	Examples: Special characters in movie titles
	| title | genres |
	| WALL·E | {"genres": ["Animation", "Adventure", "Family", "Romance", "Sci-Fi"]} |
	| 8½ | {"genres": ["Mystery", "Sci-Fi", "Thriller"] } |

view raw imdb_rest.feature hosted with ❤ by GitHub

Now create a simple RunImdbTest.java file which will be used to run test cases in .features file via JUnit.

	package com.clearqa.test;
	import cucumber.api.junit.Cucumber;
	import org.junit.runner.RunWith;
	@RunWith(Cucumber.class)
	@Cucumber.Options(format = {"pretty", "html:target/cucumber-html-report"})
	public class RunImdbTest {
	}

view raw RunImdbTest.java hosted with ❤ by GitHub

At this point you can run 'mvn test' and let Cucumber print out the snippet of code that you need to include in your Step Definition file where basically the conversion between plain english test statements happen to actual java code behind. In our case this file is ImdbSteps.java

ImdbStep.java

	package com.clearqa.test;
	import java.io.IOException;
	import java.io.UnsupportedEncodingException;
	import java.net.URLEncoder;
	import org.json.JSONException;
	import org.json.JSONObject;
	import org.skyscreamer.jsonassert.JSONAssert;
	import org.skyscreamer.jsonassert.JSONCompareMode;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	import cucumber.api.java.en.*;
	public class ImdbSteps {
	final Logger logger = LoggerFactory.getLogger(ImdbSteps.class);
	private String imdb_url;
	private JSONObject json_response;
	@Given("^I query movie by \"(.*)?\"$")
	public void I_query_movie_by_title(String key) throws UnsupportedEncodingException {
	imdb_url = "http://imdbapi.org/?title=" + URLEncoder.encode(key, "UTF-8") + "&type=json";
	logger.info("http query = " + imdb_url);
	}
	@When("^I make the rest call$")
	public void I_make_the_rest_call() throws IOException, JSONException {
	json_response = JsonReader.readJsonFromUrl(imdb_url);
	logger.info("Response = " + json_response.toString());
	}
	@Then("^response should contain:$")
	public void response_should_contain_JSON(String expected_json_str) throws JSONException {
	logger.info("Comparing reponse with" + expected_json_str);
	JSONObject expected_json = new JSONObject(expected_json_str);
	JSONAssert.assertEquals(expected_json, json_response, JSONCompareMode.LENIENT);
	}
	@Then("^response should contain \"(.+)\"")
	public void response_should_contain(String expecgted_json_str) throws JSONException {
	response_should_contain_JSON(expecgted_json_str);
	}
	}

view raw ImdbStep.java hosted with ❤ by GitHub

This Step Definition file contains a set of annotations with capture regular expressions that drive the whole automation between plain english statements and java code behind to do the job.

JsonReader.java

	package com.clearqa.test;
	import java.io.*;
	import java.net.URL;
	import java.net.URLConnection;
	import java.nio.charset.Charset;
	import org.json.JSONArray;
	import org.json.JSONException;
	import org.json.JSONObject;
	public class JsonReader {
	private static String readAll(Reader rd) throws IOException {
	StringBuilder sb = new StringBuilder();
	int cp;
	while ((cp = rd.read()) != -1) {
	sb.append((char) cp);
	}
	return sb.toString();
	}
	public static JSONObject readJsonFromUrl(String imdb_url) throws IOException, JSONException {
	URL url = new URL(imdb_url);
	URLConnection uc = url.openConnection();
	uc.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
	uc.connect();
	InputStream is = uc.getInputStream();
	try {
	BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
	String jsonText = readAll(rd);
	JSONArray json = new JSONArray(jsonText);
	return (JSONObject) json.get(0); //There can be more than 1 movies
	} finally {
	is.close();
	}
	}
	}

view raw JsonReader.java hosted with ❤ by GitHub

This helper class has to logic to http get the movie titles and convert the response in JSON objects for comparison in Step Definition class.

Finally our test cases are ready and we can run them using 'mvn test' command. This run will also create the green cucumber test report which all product managers like to see.

All project files are uploaded on github.

Run Selenium tests in parallel on multiple browsers

This blog is follow up from my previous one where I worked out an example to take screenshots on test failure. In this blog I am going to attempt to run the same tests however on multiple browsers and in parallel.


To achieve multi-browser parallel run of the test cases:

• I have updated the testng.xml and added config to run the tests in parallel with browser_type parameters each with its own thread, one for firefox and one for chrome.
• Updated pom.xml to start using testng.xml file to kickoff the test cases.
• Updated WebDriverManager class to be thread aware and return correct WedDriver (Firefox or Chrome) instance so that in case of test failure screenshot of correct browser can be taken.
• Minor tweak in the test case so that its aware of which browser type to run with.

The entire project is uploaded on github for reference: https://github.com/njaiswal/council-tax-band

Right, lets get to the nuts and bolts now.

testng.xml

	<?xml version="1.0"?>
	<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
	<suite name="Direct Gov Test Suite" thread-count="2" parallel="tests">
	<test name="Council Tax Band Tests - Firefox">
	<parameter name="browser_type" value="firefox"/>
	<classes>
	<class name="com.clearqa.test.CTaxBandTest"/>
	</classes>
	</test>
	<test name="Council Tax Band Tests - Chrome">
	<parameter name="browser_type" value="chrome"/>
	<classes>
	<class name="com.clearqa.test.CTaxBandTest"/>
	</classes>
	</test>
	</suite>

view raw testng.xml hosted with ❤ by GitHub

Two test cases in the suite, TestNG will run these 2 test case with a threadpool size of 2, which means that both browsers will fire-up at almost the same time and go through the test cases in parallel which saves us time!

pom.xml (snippet)

	....
	<build>
	<plugins>
	<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.14</version>
	<configuration>
	<suiteXmlFiles>
	<suiteXmlFile>target/test-classes/testng.xml</suiteXmlFile>
	</suiteXmlFiles>
	</configuration>
	</plugin>
	</plugins>
	</build>
	....

view raw pom.xml hosted with ❤ by GitHub

Above pom.xml snippet tells maven to start using testng.xml to kick off the test cases. Please refer full pom.xml on github

WebDriverManager

	package com.clearqa.test;
	import java.util.HashMap;
	import org.openqa.selenium.WebDriver;
	import org.openqa.selenium.chrome.ChromeDriver;
	import org.openqa.selenium.firefox.FirefoxDriver;
	public class WebDriverManager {
	private static HashMap<Long, WebDriver> map = new HashMap<Long, WebDriver>();
	public static WebDriver getDriverInstance() {
	WebDriver d = map.get(Thread.currentThread().getId());
	return d;
	}
	public static WebDriver startDriver(String type) {
	WebDriver d;
	if(type.equalsIgnoreCase("chrome")){
	d = new ChromeDriver();
	map.put(Thread.currentThread().getId(), d);
	} else if(type.equalsIgnoreCase("firefox")) {
	d = new FirefoxDriver();
	map.put(Thread.currentThread().getId(), d);
	} else {
	throw new IllegalArgumentException("Browser type not supported: " + type);
	}
	return d;
	}
	public static void stopDriver() {
	WebDriver d = map.get(Thread.currentThread().getId());
	d.close();
	}
	}

view raw WebDriverManager.java hosted with ❤ by GitHub

WebDriverManager is a bit more meaty than previous blog. Instead of keeping the WebDriver instance as static variable it has a static HashMap in which it will store thread Id as the key and the WebDriver instance as value. Since our tests are now going to run on more than 1 thread we need this arrangement so that WebDriverManager knows which instance to return to onTestFailure() method based on the Thread Id.

CTaxBandTest (snippet)

	@Listeners({ com.clearqa.test.ScreenShotOnFailure.class })
	public class CTaxBandTest {
	@BeforeClass
	@Parameters ("browser_type")
	public void oneTimeSetUp(String browser_type) {
	WebDriverManager.startDriver(browser_type);
	}
	...

view raw CTaxBandTest.java hosted with ❤ by GitHub

As you can see above I have used the @Parameters annotation along with @BeforeClass. This basically fetches parameter browser_type from the testng.xml file and passes it as method parameter to onTimeSetUp() method. This is a key step where we call WebDriverManager.startDriver() and get correct browser instance created. As mentioned above, WebDriverManager will store the instance reference in a HashMap with thread Id as key so that any future calls to WebDriverManager.getDriverInstance() can identify which browser instance to return.

Reports

Run the tests using 'mvn test' command and you should see 2 browser instances (Firefox and Chrome) getting started at the same time and the tests being run in parallel. 

Two reports get generated (one for each browser type) with total of 4 test cases being run and more importantly both contain correct screenshots of test failures.

Again the entire project is on git hub for reference, happy testing!

Selenium: Take screenshot on test failure

Going ahead on the theme of testing publicly available websites using selenium, in this blog I want to work out an example which touches on following points:

    •    Run Selenium WebDriver tests under TestNG
    •    Use the following best practices:
    ◦    Page Object Model
    ◦    UI Mapping
    ◦    Data Driven Testing
    ◦    And most importantly take screenshot on an event of test failure
Now to do all of the above I need a publicly available website that sounds interesting enough and devise testing around it. So here we go...

Test the DirectGov Council Tax Band website, given postcode, first line of address and an expected band.
The site that we are using is  http://www.voa.gov.uk/cti/InitS.asp
 

Setup Java project with Maven 

mvn archetype:generate -DgroupId=com.clearqa.test -DartifactId=council-tax-band

view raw gistfile1.txt hosted with ❤ by GitHub

Now import the maven project in your favorite IDE. I am using Eclipse.

Since we are going to use selenium with testng instead of junit my pom.xml looks like following.

	<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.clearqa.test</groupId>
	<artifactId>council-tax-band</artifactId>
	<version>1.0-SNAPSHOT</version>
	<packaging>jar</packaging>
	<name>council-tax-band</name>
	<url>http://maven.apache.org</url>
	<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	</properties>
	<dependencies>
	<dependency>
	<groupId>org.testng</groupId>
	<artifactId>testng</artifactId>
	<version>6.8</version>
	</dependency>
	<dependency>
	<groupId>org.seleniumhq.selenium</groupId>
	<artifactId>selenium-firefox-driver</artifactId>
	<version>2.31.0</version>
	</dependency>
	<dependency>
	<groupId>org.seleniumhq.selenium</groupId>
	<artifactId>selenium-support</artifactId>
	<version>2.31.0</version>
	</dependency>
	</dependencies>
	</project>

view raw gistfile1.xml hosted with ❤ by GitHub

Test and Helper Class

Now that we have got a Java project set up, lets get cracking. Since I have already done my code before writing this blog, my final directory tree looks like following.

	$ tree
	|____pom.xml
	|____src
	| |____main
	| | |____java
	| | | |____com
	| | | | |____clearqa
	| | | | | |____test
	| |____test
	| | |____java
	| | | |____com
	| | | | |____clearqa
	| | | | | |____test
	| | | | | | |____CTaxBandSearch.java
	| | | | | | |____CTaxBandTest.java
	| | | | | | |____ScreenShotOnFailure.java
	| | | | | | |____WebDriverManager.java
	| | |____resources
	| | | |____testng.xml
	| | | |____ui_mapping.properties

view raw tree hosted with ❤ by GitHub

Let me add a bit of context around what is the role of each file in the directory tree above is. Later I will add more details about each of the relevant files.

• CTaxBandTest.java - This is the testng file that contains all the test cases just by the virtue of keyword 'test' in the file name.
• CTaxBandSearch.java - Page Object Model Class. The Page Object Design Pattern provides the followingadvantages.
• There is clean separation between test code and page specific code suchas locators (or their use if you’re using a UI map) and layout
• There is single repository for the services or operations offered by thepage rather than having these services scattered through out the tests.
• WebDriverManager.java - This class will manage the singleton WebDriver instance that will be used in our tests. The importance of this manager class will become relevant when we go through the details of how to get screenshot on test failure.
• ScreenShotOnFailure.java - Contains code to capture and store screenshot on event of test failure at correct location so that these screenshots is available via test reports.
• ui_mapping.properties - UI Mapping file. Contains all the UI locators in one file for ease of modification.

 

CTaxBandTest.java

	package com.clearqa.test;
	import junit.framework.Assert;
	import org.openqa.selenium.WebDriver;
	import org.testng.annotations.AfterClass;
	import org.testng.annotations.BeforeClass;
	import org.testng.annotations.DataProvider;
	import org.testng.annotations.Listeners;
	import org.testng.annotations.Test;
	@Listeners({ com.clearqa.test.ScreenShotOnFailure.class })
	public class CTaxBandTest {
	@BeforeClass
	public void oneTimeSetUp() {
	WebDriverManager.startDriver();
	}
	@AfterClass
	public void oneTimeTearDown() {
	WebDriverManager.stopDriver();
	}
	@DataProvider(name = "dataset1")
	public Object [][] create_dataset1() {
	return new Object[][] {
	{"RG5 4NF", "1, Concorde Way", "F"},
	{"RG5 4NF", "2, Concorde Way", "C"}
	};
	}
	@Test(dataProvider = "dataset1")
	public void testCTaxBands(String postcode, String addr_first_line, String expected_band) {
	WebDriver d = WebDriverManager.getDriverInstance();
	d.get("http://www.voa.gov.uk/cti/InitS.asp");
	CTaxBandSearch search_page = new CTaxBandSearch(d);
	String band = search_page.search(postcode, addr_first_line);
	Assert.assertEquals(expected_band, band);
	}
	}

view raw CTaxBandTest.java hosted with ❤ by GitHub

In the test case above we have used various testng annotations. The important one is @Listeners which basically defines one or more of users classes, which implement some interfaces that TestNG provides, so that when TestNG raises certain events it will look for all “classes” which are basically looking for such events and call the respective event handlers in them. In this case we are using it to define handler for onTestFailure so that we can capture the screenshot and include it in testng report.
We are also using the @DataProvider annotation to make our test case data driven which basically means that we have more than 1 data set points to run through same test case.

WebDriverManager.java

	package com.clearqa.test;
	import org.openqa.selenium.WebDriver;
	import org.openqa.selenium.firefox.FirefoxDriver;
	public class WebDriverManager {
	private static WebDriver d;
	public static WebDriver getDriverInstance() {
	return d;
	}
	public static WebDriver startDriver() {
	d = new FirefoxDriver();
	return d;
	}
	public static void stopDriver() {
	d.close();
	}
	}

view raw WebDriverManager.java hosted with ❤ by GitHub

This is a static helper class which manages the only Selenium WebDriver instance we are using for all test cases. This makes access to this webdriver instance very easy later on when we want to take a screenshot on event of test failure. 

CTaxBandSearch.java

	package com.clearqa.test;
	import java.io.IOException;
	import java.util.Properties;
	import org.openqa.selenium.By;
	import org.openqa.selenium.WebDriver;
	import org.openqa.selenium.WebElement;
	public class CTaxBandSearch
	{
	private WebDriver driver;
	private Properties prop = new Properties();
	public CTaxBandSearch(WebDriver d) {
	this.driver = d;
	if(!driver.getTitle().equals("Search for your Council Tax band")) {
	throw new IllegalStateException("This is not Council Tax band search page. It title is: " + driver.getTitle());
	}
	try {
	prop.load(getClass().getClassLoader().getResourceAsStream("ui_mapping.properties"));
	} catch (IOException e) {
	throw new IllegalStateException(e);
	}
	}
	public String search(String postcode, String first_addr_line) {
	driver.findElement(By.id(prop.getProperty("ctband.postcode_search"))).clear();
	driver.findElement(By.id(prop.getProperty("ctband.postcode_search"))).sendKeys(postcode);
	driver.findElement(By.xpath(prop.getProperty("ctband.search_button_xpath"))).click();
	driver.findElement(By.partialLinkText(first_addr_line.toUpperCase())).click();
	WebElement e = driver.findElement(By.xpath(prop.getProperty("ctband.result_table_xpath")));
	return e.getText();
	}
	}

view raw CTaxBandSearch.java hosted with ❤ by GitHub

This is the Page Object Model class which knows how to navigate the website page which we want to test. As you can see I have also made use of UI Mapping best practice here and get all locators from a properties file.

ScreenShotOnFailure.java

	package com.clearqa.test;
	import java.io.File;
	import java.io.IOException;
	import java.text.DateFormat;
	import java.text.SimpleDateFormat;
	import java.util.Date;
	import org.apache.commons.io.FileUtils;
	import org.openqa.selenium.OutputType;
	import org.openqa.selenium.TakesScreenshot;
	import org.openqa.selenium.WebDriver;
	import org.testng.ITestResult;
	import org.testng.Reporter;
	import org.testng.TestListenerAdapter;
	public class ScreenShotOnFailure extends TestListenerAdapter {
	@Override
	public void onTestFailure(ITestResult tr) {
	WebDriver driver = WebDriverManager.getDriverInstance();
	File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
	DateFormat dateFormat = new SimpleDateFormat("dd_MMM_yyyy__hh_mm_ssaa");
	String destDir = "target/surefire-reports/screenshots";
	new File(destDir).mkdirs();
	String destFile = dateFormat.format(new Date()) + ".png";
	try {
	FileUtils.copyFile(scrFile, new File(destDir + "/" + destFile));
	} catch (IOException e) {
	e.printStackTrace();
	}
	Reporter.setEscapeHtml(false);
	Reporter.log("Saved <a href=../screenshots/" + destFile + ">Screenshot</a>");
	}
	}

view raw ScreenShotOnFailure.java hosted with ❤ by GitHub

And lastly the listener class which will override the onTestFailure handler of testng and help us get the crucial screenshot when a test case fails.

We can get hold of the webdriver instance very easily since its now managed by a WebDriverManager class. Above class also demonstrates how to include the screenshot link in the test reports.

Run the test

Now that we have all the bits and pieces in place lets run our test case. You might have not noticed but I have deliberately included a council tax band incorrect in our @DataProvider which means that 1 test case will pass and 1 fails with a screenshot!

	$ mvn test
	[INFO] Scanning for projects...
	...
	[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ council-tax-band ---
	[INFO] Surefire report directory: /Users/nj/workspace/council-tax-band/target/surefire-reports
	-------------------------------------------------------
	T E S T S
	-------------------------------------------------------
	Running com.clearqa.test.CTaxBandTest
	Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.627 sec <<< FAILURE!
	Results :
	Failed tests: testCTaxBands(com.clearqa.test.CTaxBandTest): expected:<[F]> but was:<[E]>
	Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
	...
	[ERROR] Please refer to /Users/nj/workspace/council-tax-band/target/surefire-reports for the individual test results.
	...

view raw testRun.out hosted with ❤ by GitHub

The reports are generated in target/surefire-reports/ directory and looking at the reports... hurray screenshots at last!!!

This has been kind of a long post, but I did want to touch on a few points in one go and glad that its out there now.

Selenium WebDriver - Maven

This post I am going to share the basics of setting up and running a simple but working Selenium WebDriver example. The hypothetical requirement this example is trying to accomplish is following:

Print out all addresses on rightmove.co.uk for 3 bedroom properties for sale in Reading under £175K

Requirements looks simple. Lets get started.

Setup Java project using Maven

Assuming you have maven installed on your system, run the following command to setup a generic java project.

mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=com.clearqa.app -DartifactId=rightmove-scraper

view raw gistfile1.txt hosted with ❤ by GitHub

Update pom.xml to 

• Include selenium-firefox-driver and selenium-support dependencies.
• Generate a executable jar file. This example pom.xml assumes that your main class would be named PropertyAddressFinder.

	<?xml version="1.0"?>
	<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.clearqa.app</groupId>
	<artifactId>rightmove-scraper</artifactId>
	<version>1.0-SNAPSHOT</version>
	<packaging>jar</packaging>
	<name>rightmove-scraper</name>
	<url>http://maven.apache.org</url>
	<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	</properties>
	<build>
	<plugins>
	<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-shade-plugin</artifactId>
	<version>2.0</version>
	<executions>
	<execution>
	<phase>package</phase>
	<goals>
	<goal>shade</goal>
	</goals>
	<configuration>
	<transformers>
	<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
	<manifestEntries>
	<Main-Class>com.clearqa.app.PropertyAddressFinder</Main-Class>
	</manifestEntries>
	</transformer>
	</transformers>
	</configuration>
	</execution>
	</executions>
	</plugin>
	</plugins>
	</build>
	<dependencies>
	<dependency>
	<groupId>org.seleniumhq.selenium</groupId>
	<artifactId>selenium-firefox-driver</artifactId>
	<version>2.31.0</version>
	</dependency>
	<dependency>
	<groupId>org.seleniumhq.selenium</groupId>
	<artifactId>selenium-support</artifactId>
	<version>2.31.0</version>
	</dependency>
	<dependency>
	<groupId>junit</groupId>
	<artifactId>junit</artifactId>
	<version>3.8.1</version>
	<scope>test</scope>
	</dependency>
	</dependencies>
	</project>

view raw gistfile1.xml hosted with ❤ by GitHub

Selenium

Import this maven project in your favourite IDE and add PropertyAddressFinder class. Java code below  should be self explanatory. It uses Selenium FirefoxDriver to open the website and search for properties .  

	package com.clearqa.app;
	import java.util.List;
	import org.openqa.selenium.By;
	import org.openqa.selenium.WebDriver;
	import org.openqa.selenium.WebElement;
	import org.openqa.selenium.firefox.FirefoxDriver;
	import org.openqa.selenium.support.ui.Select;
	public class PropertyAddressFinder
	{
	public static void main( String[] args )
	{
	System.out.println( "RightMove Scraper begins..." );
	WebDriver driver = new FirefoxDriver();
	driver.get("http://www.rightmove.co.uk");
	// Enter search term and click buy
	WebElement search = driver.findElement(By.id("searchLocation"));
	search.clear();
	search.sendKeys("Reading, Berkshire");
	driver.findElement(By.id("buy")).click();
	// Setup additional filters
	new Select(driver.findElement(By.id("maxPrice"))).selectByVisibleText("175,000");
	new Select(driver.findElement(By.id("minBedrooms"))).selectByVisibleText("3");
	driver.findElement(By.id("submit")).click();
	boolean next_page = true;
	while(next_page) {
	// Print out addresses
	List<WebElement> addresses = driver.findElements(By.className("displayaddress"));
	for (WebElement a : addresses) {
	System.out.println(a.getText());
	}
	if(driver.findElements(By.linkText("next")).size() == 0) {
	next_page = false;
	} else {
	// Go to next page
	driver.findElement(By.linkText("next")).click();
	}
	}
	driver.close();
	}
	}

view raw PropertyAddressFinder.java hosted with ❤ by GitHub

Run the Project

Now that we have the bits and pieces in place, we can let maven do the magic and make us a executable jar file, refer pom.xml's build section above to see how we achieve this. Executing the jar file via command line will open a firefox browser and print out whatever properties it finds.

	$ mvn clean compile package
	$ java -jar target/rightmove-scraper-1.0-SNAPSHOT.jar
	RightMove Scraper begins...
	Western Road, Reading, Berkshire
	Shirley Avenue, Reading
	Salisbury Road, Reading, RG30
	Salisbury Road, Reading, Berkshire
	Southcote Parade, Reading, Berkshire
	Loddon Bridge Road, Woodley, Reading
	Coniston Drive, Tilehurst, Reading
	Catherine Street, Reading, Berkshire
	Lincoln Road, Reading, RG2
	Belmont Road, Reading, RG30
	Ambrook Road Reading
	Little Johns Lane, Reading, RG30
	Salcombe Road, Reading
	Tilehurst, Reading, Berkshire
	Westerham Walk, Reading, Berkshire
	Crockhamwell Road, Woodley, Reading, Wokingham, RG5
	Caversham

view raw gistfile1.txt hosted with ❤ by GitHub

Conclusion

Above blog demonstrates a very basic usage of selenium webdriver to automate web based tasks that cannot be otherwise achieved via well known api's.