I recently came across this article, Evolving to Continuous Testing. Have a quick read and come on back.
I found myself initially nodding along with the author’s sentiments, but that didn’t last too long, I’m afraid. It is fair to say that software delivery will be as fast as your slowest component and often that component will be testing if you hold a very narrow view of the term. If you view testing as a phase in a process, then the only bit you see is the part that tends to occur just before release. IT doesn’t take into account the testing that occurs during things like requirements/epic review, story grooming, pairing with programmers during implementation and so on.
The term ‘continuous testing’ seems to be an odd way of saying ‘sufficiently advanced monitoring’. I wonder if I’d have been so critical if the author had written about evolving monitoring. My first real reaction was to the concept of automating risk measurement. There are many different kinds of risk. I wonder how you automate measurement of things like potential harm to brand image for example. That’s not to say machine evaluation of known rules around known risk might not be useful, but that should be used to assist/supplement rather than to replace human evaluation.
The author talks of testing (manual and automated) being focused on passing or failing. It is a very small part of what testing focuses on – the ‘known knowns’. I appreciate that the author is saying that there is more that can be done to point to potentially poor code quality, but at this point I’m on the alert to see what else he doesn’t understand about testing.
The ‘defect documentation to simulated replay’ section is one potential way of solving the issue of bug ping pong. Being able to set up a containerised environment with the required test pre-requisites does sound like a nifty thing to have up your sleeve, but it ignores the fact that bug ping-pong points to a systemic problem of relationships between roles. Testers and programmers should be working closely together. Preferably close enough that a tester can turn around and ask a developer if what they’ve just found is significant enough to fix immediately. Ticket ping-pong is an issue that crops up due to the tyranny of distance/time and to some extent of tribalism.
The author’s point about structured and unstructured data is about as close as he comes to actually understanding testing. Providing data from a host of unstructured sources and providing it to an audience that matters in a way that they can immediately digest.
The last couple of points are around speed of release and my initial reaction was to roll my eyes at those too. The old school tester in me says that you always want some sort of human intervention when rolling code to production. Someone to provide that sanity check to make sure that things really are as we need them to be. There are a lot of interesting things going on in the industry though. Take a look at Shopify for example with multiple developers releasing code to production multiple times per day. In many ways this seems antithetical to good testing, but they’re doing it successfully and have been for quite some time. It feels like there’s an opportunity to dig in and learn there.
Getting back to the article, The author is exhorting us to make use of monitoring to improve code quality and to assist with release decisions. That’s not a bad thing in and of itself. It would have been far more palatable if it hadn’t been hung on an emaciated understanding of testing.