So, are we just coining terms these days? Apparently, TestOps is not just a term. It is the answer to the increased demand for quality and has been around for a couple of years now.
But simply speaking, if DevOps is operationalising your development lifecycle and aiming towards faster delivery, TestOps is operationalising the various test approaches and making sure the deliveries are faster as well as reliable.
And to avoid over complicating things, I shall abstain from throwing around jargons such as DevTestOps or BizTestOps or TestDataOps!
How does TestOps fit into the bigger picture?
Well, DevOps is a commonly used term these days, and every organisation using Agile methodology defines some pragmatic approach to help establish a Production delivery model that is fuelled by Continuous Integration and Continuous Delivery methods. However, the very backbone of a successful CICD model is dependent on the use of tools and practices that are the vertebrae and are further identified by the needs of software maturity, agility and quality.
An effective DevOps strategy rides on the success of its individual atomic stages. These stages need to be properly defined and calibrated as per the Engineering needs. It should also have the right blend of tools and practices in order to build, test, and release software faster and more reliably.
Testing, in particular, is complex because, besides the functional aspects, it needs to encompass the various aspects of Performance, Security, Resiliency, Soak, and so on. So it’s important to chalk out these needs separately. This is where TestOps comes in. It is the underlying strategy responsible for identifying and utilising the necessary tech stack to execute such test scenarios.
So, what does it take?
Imagine having to go through the different test approaches (as mentioned above) in each release and yet manage to make the results measurable and comprehensible. This calls for an efficient operationalisation strategy. This strategy must be supplemented by an ecosystem consisting of suitable practices, environment, tools, and frameworks to support a quality deliverable. Yet another important thing is to understand ‘What’ should be operationalised and ‘Why’; answers to which could be crucial before investing your efforts. Let’s try answering these queries.
To begin with, the need is ubiquitous for something like Performance Testing where monitoring system-level as well behavioural-level parameters is crucial. System-level parameters comprise utilisation of memory, cores, disk I/O, network bandwidth usage, thread count, open file descriptors, and so on. Behavioural parameters, on the other hand, are indicative of your application’s characteristics. Functions such as variation in data processing throughput, latency and jitter in server response, slowness during concurrency and parallelism, data distribution or caching correctness, and transactional lineage are a few examples that should be measured to observe the behaviour of an application or a service by varying the traffic conditions.
For Resiliency Testing, your Ops must help you assess the impact on your Functional workflow. While monitoring the behavioural parameters is again crucial here, it needs to be assessed repeatedly though, because one might encounter altogether different issues if the services are brought down by targeting specific stages during the workflow.
When it comes to something like Soak Testing, alerting mechanisms are mandatory. Your Ops should have the necessary alerting mechanisms during longish runs to ensure that you get notifications as soon as your configured thresholds are breached. Also, these thresholds could vary based on the time elapsed during the run.
Code Instrumentation tools could be yet another useful aspect. Apart from Online Instrumented areas, that is an obvious thing to monitor in either your dev or test setup, the test setups in particular can also help profile applications through offline code instrumentation techniques by embedding agents. This can help in targeting very different category of issues.
You also need a robust reporting mechanism that should make observing and logging bugs easier. These reports should be well integrated with communication channels for early sharing of results with stakeholders.
A number of such tools are being used in the industry that fulfil the needs mentioned above and could possibly be labelled as TestOps tools but the explanation on their use cases can probably be a discussion topic for some other time.
And you would need Continuous Testing, because…
If you raise a question on Continuous Testing, the most probable response would be that it is essential for frequent and continuous development and execution of your automation suites. However, there is more to it than that. It should be perceived as a continuous enablement for automation rather than just automating things quickly. In other words, it’s equally important to keep scalability and coverage in mind when you are choosing to develop or reuse your Test Automation Framework.
Continuous testing also addresses the much-needed balance of availability, usability and scalability in an existing test environment that enables the Quality personnel to leverage and benefit from it across different projects. So your environment should have the necessary flexibility of automated on-demand integration with the Ops needed for your Test needs. This is how Continuous Testing rides on an effective TestOps strategy.
And the above requirements should be defined through a skeletal workflow template considering the larger scheme of things. Such template should allow for easy on-boarding across all quality stages in case of a new project. It should be simple for teams to adopt and, at the same time, be bespoke enough for them to understand the benefits for their specific needs.
Okay, what else is needed?
We also need to inculcate the best practices for TestOps to prosper and for the outcomes to be useful within the agile culture of an organisation. A few of the more relevant ones are:
- Plan User Stories around non-functional test needs. It needs time and effort.
- Spend time to review the Test Ops needs and provide feedback. Awareness and involvement of the team is a must.
- Prioritise fixing non-functional defects.
- Carefully assess production environment and SLAs and plan a scaled down similar one which should be readily available.
- Invest the bandwidth of your squad in automation of test setup, data generation, monitoring and analysis of results.
- Plan your budget to buy the necessary tools based on your requirements.
- Design smart keeping modularisation in mind. And if needed, take that expensive yet critical decision to redesign if needed.
- If you need specialists, go ahead and hire.
Sounds good, but do tell that it gets better with TestOps mindset?
To start with, upgrading technical skills would come naturally. One must understand that awareness and execution of test scenarios is always going to be project specific, but creating a test environment to portray the necessary System and Behavioural Indicators that are measurable requires in-depth knowledge. It also involves a knack for identifying the problem areas.
In other words, it requires a persona having the technical skills of one or more of the following: a performance test specialist, a programmer, a database administrator, a domain expert, a site reliability engineer and a Network Engineer.
Besides skills, TestOps requires a larger responsibility; the responsibility to comprehend the basic need for quality assurance at every stage without the need for continuous involvement and then planning the right environment for it.
The need for smart and continuous deliveries begets the need for Quality Engineering teams to provide faster feedback. Hence, curiosity and proactiveness in identifying and filling such gaps in the TestOps environment is yet another much-needed skill.
I would like to say that TestOps is a must-have for any organisation that strives for a “Fail Fast but build it the Right way” approach. And, any worker who abides by this motto will be an invaluable asset!
