Anatomy of a Test Automation Tool
We’ve looked at the anatomy of a great test automation system and scripting framework, but an important aspect of any automation strategy are the tools that we use to perform our automation scripts. It may not seem like a primary focus for many companies where the focus is on rightly implementing the correct framework. However, with tools often coming at a cost, its important people be empowered with understanding how they work a little better to ensure they make the correct decisions about them. This is especially important when there are currently so many different open-source alternatives to the more established products which may often work better for companies if they are able to harness their capabilities effectively.
So today I will take some time in unpacking the basic anatomy of automation tools and some of the things that differentiate them from each other. As there are many different tools out there the intent is not to evaluate tools against each other or necessarily cover aspects of every tool, but simply just give you a high-level perspective of how automation tools are designed to work. And by work, I’m not referring to how you use the tools and automate them, but the actual architecture in works that make them able to do what they do.
While automation tools differ vastly between Web Based UI, Desktop UI, API or unit testing tools, they all generally consist of the same principles. At its very core, a test automation tool is an application or piece of code that serves as an interface between your test script and the application under test. While for unit tests, this is easy to do because it is essentially a piece of code that executes the underlying code to check its output, tools that test a full application work a little different.
The components generally consist of a parser, which acts as the code interpreter or compiler, an assertion library that provides a variety of functions that allow for the verification of outputs and runners which enable the actual execution of your test cases.
Essentially, a parser is a compiler or interpreter component that breaks data into smaller elements for easy translation into another language. A parser takes input in the form of a sequence of program instructions and usually builds a data structure in the form of a parse tree or an abstract syntax tree.
The overall process of parsing involves three stages:
Lexical Analysis: A lexical analyser is used to produce instructions from a stream of input string characters, which are broken into small components to form meaningful expressions.
Syntactic Analysis: Checks whether the generated instructions form a meaningful expression. This makes use of a context-free grammar that defines algorithmic procedures for components. These work to form an expression and define the order in which tokens must be placed.
Semantic Parsing: The final parsing stage in which the meaning and implications of the validated expression are determined, and necessary actions are taken.
A parser's main purpose is to determine if input data may be derived from the start symbol of the grammar. If yes, then in what ways can this input data be derived? This is achieved as follows:
Top-Down Parsing: Involves searching a parse tree to find the leftmost derivations of an input stream by using a top-down expansion. Examples include LL parsers and recursive-descent parsers.
Bottom-Up Parsing: Involves rewriting the input back to the start symbol. This type of parsing is also known as shift-reduce parsing.
2) Assertion Library
An assertion library (or also called framework) is essentially a function built into the respective automation tool that allows for a variety of checks or conditions against the expected outputs. For API testing, this can be very simple in terms of just checking that certain fields can exist and then expands more with UI testing where it can recognise and verify against the various state of objects.
The whole purpose of assertion libraries is to remove the need to code a variety of functions to check against certain outputs and essentially makes it easier for testers to verify aspects of an object under test without needing to do extensive coding, which is often the reason many companies go with the different tools, to enhance the development of their test scripts.
Assertion libraries vary between tools in how they work an are often mostly customisable at a unit test level, but how they work applies consistently across them all.
3) Test Runners
A test runner is the library or tool that picks up an assembly (or a source code directory) from the parser, that contains unit tests, and a bunch of settings, and then executes them and writes the test results to the console or log files.
Often tools may make use of different runners’ dependent on the code or tool in operation. In Selenium, this would typically be your web.driver for a browser, but there are many runners that tackle things for different APIs and programming languages for unit testing (NUnit, JUnit, NUnit, MS Test, etc).
These differ in their architecture, but should each contain something resembling the below:
A describable is essentially a function that gets specific information about the object it is interacting with to know what to execute and what not to.
A run function is essentially something that executes code. It compiles the test script as code and then executes it in the provided framework. It is designed to execute a given piece of compiled code against a target code and return a relevant result.
- Parent Runner
A Parent Runner is an abstract base class which essentially allows the handling of multiple runs. Any testing suite is likely made of multiple tests that need to be executed against the different aspects of code or executable objects and a parent runner is what handles the execution of these individual run jobs, intercepts their logs and stores them in a log file.
While the parent runner may handle the execution of multiple individual runners, these essentially each pass-through output which needs to be collected in various log forms. A reporter function takes these logs and collates the information in a way that can then be read. While some tools may provide a form of default graphical reporting, most test frameworks will be scripted to handle reporting in their own way and so these reporter functionalities are not always utilised but remain an aspect of the tool that can be useful.
No two tools may be alike in design, but there are similarities that make understanding their design easier and hopefully aid you in your quest to finding the best tools that suit your purpose.