The Role of Continuous Testing In DevOps

In today’s business environment, being agile is crucial to be able to respond faster to the rapidly changing market demands. IT organizations across the globe, thus, are focused on reinventing their development methodologies and are looking to release faster updates and accelerate their time to market so that they can stay ahead of the curve.

Considering that today, change is the only constant in the software development landscape, we have seen new development methodologies such as DevOps emerge help organizations respond faster to rapidly changing market demands.

---

According to Puppet Labs annual 2016 State of DevOps survey, 63% of the respondents have implemented DevOps practices in their organization. It also iterated that organizations leveraging DevOps had 50% fewer failure rates with an ability to recover 12 times faster and could ship code 30 times more frequently.

---

One of the main reasons why DevOps has become popular in the development circuit is because –

• It provides organizations the ability to align the software application development with business demands

• It makes the development process more organic by integrating different stakeholders of software development

• It makes the developers, the operations and the testing teams work as one harmonious singular unit.

DevOps and the Impact on Testing

Considering that software delivery has to be accelerated, it puts a greater focus on testing. For DevOps, it becomes imperative to also adopt a top-down approach in testing that helps in mitigating business risks along with functional and non-functional requirements in a continuous manner. Given that applications demand greater connectivity between systems, processes, and infrastructure, the testing process has to become more efficient and, therefore, needs greater automation. Clearly, testing in DevOps has had to evolve further in the following ways:

• Since the software development is a collaborative process between developers, testers and build engineers, there is a demand for greater integration in terms of both test coverage and ease of use

• Testers have to ensure that the automated suites and systems are in a completely reliable and maintainable state in the shortest possible time frame. Only then they can focus more on monitoring these systems and spend more time on exploratory testing

• Testing professionals have to work across teams and in a DevOps environment, often have to share responsibilities with the development team, build engineers and program managers. They also have to assess code quality and iterations for development tools

• The testing process such as design and development of test automation suit, creation of test environment, test script creation etc. has to start early in the development cycle and testers cannot afford to wait till a feature is complete to start the test process.

• The exit criteria have to be well defined for each run so that based on the results of the tests, a Go or No-Go decision to Production can be taken.

• Testing outcomes have to provide automated feedback at specific checkpoints which is an auto-trigger for the next action in the delivery chain pipeline – If the feedback is green the development team can move forward and if it is negative, then the process can be stopped immediately and corrective action can be taken

• The test automation suite has to be integrated with the build deployment tool to enable centralized execution and reporting and enable faster builds

The role of QA in DevOps is more about preventing defects rather than finding them to implement an environment of continuous improvement and quality

Why is Continuous Testing Essential for Successful DevOps?

In DevOps, the speed of testing is matched with the speed of development. The entire DevOps methodology is based on collaboration with Continuous Integration and Continuous Delivery as the hallmark. Here, individual developers create the code to add new features, fix defects or simply to make the code work faster and then integrate it with unchanged code. Along the way, they have to run unit tests on their individual code and follow it up with another unit test for the integrated code. This consolidated code is then delivered to the common integration area of the team where all the code components of are integrated. This Continuous Integration process ensures that the code in production integrates at all levels without error. Clearly, in DevOps, the Continuous Testing process starts with the developers themselves.

Once the code for the feature is validated, it is then delivered to the testing team starting off the Continuous Delivery stage. The testing team then runs its own set of performance and functional tests in a production-like environment complete with associated test data. While it might seem that Continuous Delivery constitutes sending the code to another team, the aim of Continuous Delivery is to ensure that the code is ready for test so that each application component under production can reach the desired environment in a continuous manner.

DevOps also demands that the operations teams also run their set of smoke tests, system stability tests and acceptance tests before the application is delivered to the production environment. The environment in this stage has to be production-like with a ready availability of required test scripts and test data of the performance and acceptance tests. Post production, the Operations team needs to monitor the application performance and ensure that the application environment is stable. They, of course, can leverage tools that enable end-to-end Continuous Monitoring.

When looked at closely, it becomes quite clear that while faster code development sits high on the priority list, it can only be achieved via Continuous Testing. Continuous Testing ensures that the code is in the right state and also works in the right environment. Continuous Testing becomes the enabler of the Continuous Integration and Continuous Delivery process and constantly validates that application function and performance. At the same time, since the Continuous Testing process has to check each and every aspect of the application and the code changes as well, test automation becomes a critical enabler.

Since DevOps sprints are lightning fast, testing practices in such an environment cannot afford to be sluggish. With Continuous Testing, instead of slowing down the development conveyor belt so that QA can catch up, testing teams have to focus on testing early, testing often and testing faster to maintain the continuity of fast development cycles. Continuous Testing, thus, has to start within the development process and run as a part of the build deployment process itself to support the fast cadence of releases. If there is any fail anywhere in the development process, the deployment process is halted immediately and all invested stakeholders of delivery, i.e. the developers, testers, and operations teams are immediately notified and can take immediate corrective action. This enables the team to fail fast and recover quickly and thereby validate the DevOps methodology.