Saturday, 2 April 2011

How the play framework test runner works

After working for a while with the play framework we had the need to write better integration tests.
By better I mean easier to maintain.
I like selenium but there re-usability and maintenance issues. Anyway without going any further into details I had to figure out how the @tests page works.

The @tests page for one of the sample applications


First thing let me assure you: there is no magic. Actually is quite easy to follow once you understand how the play works. The only complicated part is the selenium bit.

TestRunner is one of the default modules in the play framework together with secure, crud and some others. You can find it under PLAY_HOME/modules. You can also browse the code on github.

Adding the routes - plugin code


As far as I understand the module only contains a plugin a
Under testRunner there are four folders
  • src - the plugin source code
  • public - contains all the static html as in every play project
  • app - controllers and views
  • firephoque - just a library folder
Let's go into the details.
Under src there is a file called play.plugins that tells the framework the plugin implementation is play.modules.testrunner.TestRunnerPlugin.
This class extends PlayPlugin and has 3 methods onLoad(), onRoutesLoaded(), onApplicationReady().
This methods are invoked by the framework itself...guess when? :)
onLoad() simply adds the test folder to the classpath for the application and for all the modules.

onRoutesLoaded() adds the test routes to the application
@Override
 public void onRoutesLoaded() {
    Router.addRoute("GET", "/@tests", "TestRunner.index");
    Router.addRoute("GET", "/@tests.list", "TestRunner.list");
    Router.addRoute("GET", "/@tests/{<.*>test}", "TestRunner.run");
    Router.addRoute("POST", "/@tests/{<.*>test}", "TestRunner.saveResult");
    Router.addRoute("GET", "/@tests/emails", "TestRunner.mockEmail");
 }

and the last one, onApplicationReady() simply prints the test url to the console.

Note: This line is then used in auto-test mode to detect the application is ready. Look at base.py lines 204-215

The other class in the same folder,called FirePhoque.java is used in auto-test mode to simulate the browser.

To summarise, so far we have the application started with some additional routes pointing at TestRunner methods.

Where is all happening - controllers and views


Let's have a look at the TestRunner controller.
You can see there all the methods added as routes in TestRunnerPlugin. Let's look at what some of methods do.

index()
is using TestEngine to figure out which ones are unit test, functional tests and selenium tests. Then it just puts the three lists in the render scope and the view page index.html renders them nicely in what you see when you go to /@tests

list()
This is a list of all the tests rendered as text. As far as i know is only used by auto-test

run()
This is the method that actually runs the java tests and sends back the results.

saveResult()
This method saves a file in the test-results folder for every test that has been executed


    The java tests execution

    Unit and functional test are easy enough to understand. If you want to have more details, look at the run method in TestEngine




    The selenium tests execution


    Now here comes the tricky part

    Files ending in .test.html are selenium tests and they are passed to the TestRunner controller as .test.html.suite.

    The controller simply execute

    test = test.substring(0, test.length() - 6);
    render("TestRunner/selenium-suite.html", test);
    

    Now if you look at selenium-suite.html you'll be scratching your head because there is nothing else that a table with a row and a link to the test.

    To understand you need to look at index.html lines 362-380.
    This code is simply loading the selenium-core runner (pure DHTML) in an iframe, is making it visible and is calling the Testrunner controller to get the result every 2 seconds until it either returns an HTTP 500 (for a test failure) or an HTTP 200 (you guessed, test passed)

    a picture is worth a thousand words


    The frame in the mean time calls the controller (lines 82-100) loads the html test as any other application page, using the TemplateLoader.

    That is why you can use tags in your selenium tests! Isn't that great?

    Now the selenium-core runs the html test and calls the controller saveResult method (lines 120-131) that saves a file either .passed or .failed depending on the test outcome.
    That will allow the flow in index.html to move on to the next test.

    Quite good isn't it?

    Thanks for reading, please leave any feedback / corrections you feel.

    6 comments:

    opensas said...

    great article!!!
    unfortunately, so far, there is not much documentation about the inner side of play framework, it's great to see articles like this
    following play's code is not that hard, but sometimes you just need a guide...
    ps: you should publish this article on dzone...

    Gas said...

    Excellent article! It's amazing how the play core is simple and common sense!

    Adelar S.Q. said...

    Very good article!!!! Thanks.

    opensas said...

    Hey, I've just voted for this article on http://www.dzone.com/links/r/how_the_play_framework_test_runner_works.html

    looking forward for more articles like this, I'm definetely put your blog on my radar...

    mericano1 said...

    thank you all, i'm glad you find this useful

    Felipe Pedrini said...

    Understanding it deeply is nice, but teaching it to others in a simple way is awesome! Congratz and thanks!