In this article we take a look at "Test Automation" as it’s an area that is getting increasing attention as organisations evolve further down the Agile/DevOps road.
Test Automation is the use of special tools to set up and control the automated execution of, sets of tests which require to be executed on a number of occasions. It displaces significant manual effort and historically was used in regression testing or for repeat cycles of testing.
It is often closely aligned with Performance Testing where the scripts used for functional testing are run in volume to varying patterns to simulate varying workloads in a simulated production environment to establish/prove response times and performance/scalability.
There are different levels of sophistication, from simple record and playback tests, to data table driven, reusable component-based automation environments. The latter are normally developed in organisations where automated testing is a key element of the testing cycle on a continuing basis.
As with any software development programme, the development of a comprehensive automated testing environment requires an initial investment and ongoing sustainment, e.g. updating automated suites with each software release, to yield a continuing benefit to the organisation.
The initial investment covers tool selection, normally through the client’s investigation, evaluation and decision-making process, installation and commissioning with vendor support, training initial staff and then steadily building a library of reusable assets. As part of commissioning the structure of the set of scripts, the driver table structure should be designed and the standards put in place. Ideally it would have a clear place and defined role in an organisation’s overall SQA and Testing strategy.
Historically, and in particular with projects using waterfall type approaches automation was used for regression testing and scripts tended to be developed/updated towards the latter end of the project as tests cases became settled and software releases became more incremental. The cost benefit approach in vogue at the time meant the displacement of manual effort for regression testing at that stage in the project made sense.
It led to a situation where individual projects tended not to invest in automation as a priority, and the sustainment of accurate up to date automated regression suites was compromised due to the phased investment and handoff mindset that accompanies a waterfall project approach.
Individual projects often ran out of “budget” and sacrificed the costs of updating regression suites as part of project closure. If there was not a central team picking this up and carrying automated regression forward the suites were compromised over time. This led to challenges with justifying test automation on an ongoing basis and realising the benefits of it.
This was neatly summarised in the World Quality Report 2019/20 – “In summary, test automation has been around for almost two decades now. A key reason why organizations have not been able to get the desired return of investment from automation initiatives is because most frameworks were designed to automate manual steps, but were not intelligent.”
With the continuing move to Agile approaches and the further evolution towards DevOps the focus on automated and continuous testing has been an increasing focus of software quality assurance. If you need speed with quality and your objective is quality working software at the end of a sprint then automation is a critical component in maintaining agility, and is a priority for the entire team through established practices/disciplines and a focus on continuous improvement. Continuous integration/builds, unit, functional & integration test execution and continuous/automated deployment are common examples of applying automation beyond the scope of traditional automated tests.
As well as bringing automated testing to the fore, the whole focus on quality through the sprint cycle(s) has a focus on testing early and constantly. This has significant implications for Testing roles and skill sets, for team organisation and for the technical tools and environment within which quality software is delivered. Test automation is key to meeting the needs of the agile environment by delivering faster results and resolving a lot of the issues around manual testing. Although test automation helps enterprises in automating critical, repetitive and lengthy test processes, it can turn out to become somewhat chaotic if it is conducted in an unplanned manner, without appropriate understanding and analysis, and without being part of a considered overall SQA and Testing strategy.
With the increased focus on test automation and the increasing reliance on test automation as a key enabler of quality at speed the consequences of failure are more significant and costly than with historic waterfall based automated regression. Across the industry some of the main reasons offered for the failure of test automation are:
· Lack of proper planning before implementing test automation
· Inadequate resources to purchase the right tools
· Wrong selection of tools
· Underestimation of time, costs and efforts required, both for setup and sustainment
· Lack of training to initiate/support test automation
· Lack of skilled testers – skill sets are changing
As we noted previously moving to Agile practices is a culture change. Part of that change involves moving to automated testing so good planning, selecting the right tools, appropriate training and reflecting the role and practice of Test Automation in an SQA and Testing Strategy that is right for the organisation are key success factors.
As well as having the right strategy and process we also need to keep the people factor in sight. Automation is meant to reinforce people, not replace them. While the repeatable part of software testing can be automated, this is really only one element of what is involved in the testing craft. This can be easily overlooked if test automation is seen as a panacea for all quality issues. With automation you can execute poorly designed and scripted tests just as quickly as good ones!!
The third dimension of test automation is technology, i.e. test tooling and test environments. Selecting the right tools and the right number of tools (it is as easy to have too many as to have too few), deploying them and supporting them effectively are key issues. Doing this against a backdrop of a constantly changing tool market in parallel with increasing demand for quality at ever increasing speed makes this even more challenging.
Couple that with the constantly evolving technical development and delivery environments and the implications for test environments, the impact regulatory and other changes are having on test data generation and management and the overall delivery of quality at speed is more difficult than ever.
If we consider testing as moving in step with development even more closely these days with constant development, testing and deployment and with automation as key then we need to think about testing in development terms. Test automation these days requires scripting, modularisation, updating scripts in line with software module updates(refactoring) and to be effective requires a technical architecture and technical environment availability (including test data bases). This has significant implications for developers, tester and infrastructure delivery teams. These days delivering a SQA and Testing capability is significantly about quality of the supporting platform.
At Vantage Resources we use our METISURE framework to guide our delivery of services to client This covers SQA and Test strategy development including automation, support for tool selection etc. and supplying expert staff to support deployment and implementation.