Regression Revolution: Testing the pulse with high-speed PDF comparison
We are pleased to present a powerful test automation solution that increases test efficiency and accuracy. First, we explain the concept of Essent and their Inspire platform, then we explain the Robot Framework and finally we give you an insight into the functionality of the framework.
Our primary focus remains on delivering valuable test results for Essent. As a subsidiary of E.ON, Essent plays a pivotal role in the Dutch energy landscape. Within Essent there is a platform called Inspire. Inspire lets us send letters to customers or clients if you will, as well as companies.
How does our toolbox transform testing with Robot Framework?
Harnessing the power of Robot Framework, we've developed an advanced automated regression test suite. We test the content of letters and verify that they have not been changed inadvertently by a software change. These communications inform Essent’s customers about their contracts and payments. Hardly anyone’s favourite subject, so let’s make sure there aren’t any bugs found in them!
Robot Framework is an open-source framework for automating tests and is regularly maintained. It uses a keyword-driven testing approach and features an easily understood syntax for test data, avoiding industry-specific jargon. The testing capabilities of Robot Framework can be expanded by implementing test libraries in either Python or Java. In this case we've chosen Python. You can also create new higher-level keywords using the same programming syntax as for creating test cases.
How does Robot Framework supercharge testing at Essent?
We leverage the Remote Server feature of Robot Framework which allows us to separate the test tooling from the test server. To compare the control PDF with the test PDF, we use a high-speed print stream comparison tool called StreamDiff. Using Remote Server, we can remote control multiple virtual machines that run our test tool, from a central test server. Finally, we use the Robot Framework add-on Pabot. The regression test currently has 95 test cases. Pabot allows us to spread the test cases across 3 test suites and run them in parallel so that each connect to a separate instance of our tool servers. This setup spreads the testing load across multiple virtual machines, spreading out the load and decreasing run time.
As discussed above, we want to compare the contents of our letters against a previous version of the letter, generated with a known good version of the software. In order to cover all the differences in each type of communication, we test with hundreds to thousands of letters per communication type, depending on importance of the communication. In total the regression test verifies the content of 60.000+ pdf files.
The beauty lies in its adaptability; with each run, the system updates the baseline, ensuring that future regression tests remain consistent and easy to (re)use.
Functionality of the regression test
The regression test is kicked off on a StreamDiff server machine for optimal performance and reliability. To use this testing suite, we make use of:
- access to the Essent network
- a setup compatible with CloudPC (Windows).
- Python 3.7 or a newer, as well as Java JDK.
Once the test run is complete, we use a source-code editor to look at the test results.
Now you have a better idea of what our test automation at Essent entails. As said, due to the open-source character of Robot Framework, we can continuously adapt and improve our testing processes to meet evolving needs.
Bob van Schaik & Michel Sluis, Test Automation Engineers