Recently my team started working on a new project which is actually a re-developing of a legacy system in a new platform. We decided to move from a windows based application to a new web-based J2EE application. It is a project that has all the odds against failure. We know the domain, we know the new technology, the team is working together many years and there is only one restriction about it. We ought to use the same database schema since it may be used in conjunction with the old windows – client. Good news since we don’t have to redesign the database. Bad news since we have to stick on some bad database smells of the past. Anyway we have the chance to complete a project on time with no serious risks. It is also a very good opportunity to apply in practice the testing strategy we dreamed for all these years. A strategy that is fully automated, repeatable with a single key press and covers all aspects of the system. We decided to use Jenkins as our build system and in the following lines I briefly describe the testing strategy
Unit Testing
Since it is a Java system we had to choose between the two famous unit test frameworks. JUnit and TestNG. For no particular reason (maybe because we are already experienced with it) we have chosen JUnit. All unit tests run after each commit with our first Jenkins job (let’s call it app-trunk). If coverage falls below a predefined threshold (i.e <75%) the the build automatically fails,team members are notified and last commiter(s) strive to fix the build.
Database Unit Testing / Database Integration Testing
You can name the next level of testing as you wish. I prefer to call it Database Integration Testing since we test how well is our system integrated with the database. Remember that we had to keep the database unchanged so we have a double challenge here. Test our code (EJB3 entities) against an existing / productive database with real data and test our code against a database that is automatically created with sample data. If am not mistaken the only available mature framework for database testing is DBUnit. It provides some flexibility about database testing but we had some more requirements so we created a layer over it to automate things like data generation, testing of entity objects manipulation etc. Maybe in another post we will describe this layer in details. For each entity we test functions like create, edit, find, delete and some schema-related issues, such as indexes, foreign keys and primary keys. Due to the fact that there are some tables with many columns we would like to sure that all mappings (columns and relations are correct). All kinds of these tests run through an automated build-job,that is being triggered whenever app-trunk succeeds. There are two matrix jobs in Jenkins that test our code against all supported database platforms with existing and sample data. Obviously, if something goes wrong, again all team members are notified and try to fix the problem.
Integration Testing
Our system runs on an application server so we need somehow to test its behavior and how well is integrated with some services and frameworks like JSF and CDI. Here comes Arquillian to make our life much easier (with JSFUnit extensions). The idea of writing tests just like unit tests is brilliant and since we don’t have to learn a new framework we can adopt it quickly enough, from the day – one of our project. Each class that uses services within the container should be tested at this level. Same rules with unit testing about coverage apply here as well and integration tests run after the success of all database integration test jobs. We have created once again a matrix build job in Jenkins that runs all integration tests against supported platforms and application servers.
Acceptance Testing
Probably the hardest job because this is the point where we have to test our system if it meets the end-user’s requirements. It is also the step that requires the strongest hardware since the software has to be deployed in a real environment and automatically be tested for the most critical scenarios and the most used application flows. There are plenty of tools, however we have chosen to use Selenium, due to the fact that has a quite stable integration with Jenkins CI and has a very powerful add on for Firefox browser. We try to cover in depth not all possible user screens but those that are the most . All acceptance tests run during our nightly build once a day and only if there is an existing commit since last run. Although it is very difficult to test our system in all different environments we try to run the tests in the most commonly used. Obviously acceptance testing does not end with the automated build job. It is QA team’s responsibility to perform a complete acceptance test of all scenarios, but we strongly believe that core functionality should be always tested automatically to catch serious defects before QA team.
Stress Testing
Our application stores a large amount of data and is accessed by many users so we have to ensure that its performance is acceptable by them and does not fall below some predefined thresholds. Without any automation it would be very hard to achieve this, so to conquer the last frontier of testing we have used Apache JMeter. Although JMeter has no tight integration with Jenkins we have created a different project (JAR) that includes all performance and stress tests, triggered also during our nightly build job. Stress testing is not our number one priority, however at the end of each iteration we evaluate the usefulness of our existing tests and we modify them accordingly if some serious requirements change has occurred during the last iteration.
The UDIAS (unit, database, integration, acceptance, stress) testing strategy is probably not a silver bullet, but it covers all the different layers and views of an application. The time needed to set running all the above build jobs is significantly large but the ROI of investment worth the effort. I don’t think that there is something more important in a system under development / maintenance than an automated testing plan.
Thanks for reading this post and feel free to rate it, post your comments or share it with others.
Only Software matters 