BITE – reinventing the broken wheel

Wednesday, November 16, 2011

I’ve been ranting a bit in various fora lately. I very much want to see fake testing die. What do I mean by fake testing? You might describe it as test theatre, in the same way that the TSA performs security theatre. It’s a well-known pantomime that we’re all forced to go through allegedly to keep us safe. We all know it’s bullshit, but it seems no one really has the power to change it.

Fake testing is the blind worship of process and procedure (over the sapient and conscientious application of skill). There’s been some talk of its demise recently and while I want to believe we’re heading in the right direction, I also want to keep in mind that this is not an easy problem to solve and if we’re not careful we won’t solve the problem, we’ll just shift it elsewhere.

Let me give you an example.

At GTAC, there were some very interesting tools demo’ed. I looked at BITE (Browser Integrated Testing Environment) and thought – yeah, there’s another nail in the fake testing coffin. I particularly like the concept of making it easy for the end-user to report bugs that contain information that programmers can use to solve issues quickly.

As I continued to watch, something nagged at me. A sense that something was not quite right. Not quite deja vu, but certainly the sense that if it were this easy, we’d have done it before.

Shortly thereafter, I was chatting with good friend and colleague Jared Quinert. I had forgotten how good he is at taking a subject, stripping back all the bullshit and laying its foundations bare. If you ever get the chance to work with, or hire him. Do it. Seriously. As usual, Jared locked onto what I’d been feeling uneasy about without me even having to explain myself. Having shared the link to the talk I’ve linked to at the start of this post, our conversation went something like this:

 

Jared: Record playback software is apparently central to the life of a manual tester…

Ben: Yeah I chuckled at that too.

Jared: Recording scripts, converting to English doesn’t solve the abstraction problem. Waiting to hear them elaborate on re-recording being cheaper than maintaining…and they’re still just talking about QTP features, but nobody has piped up with this :) QTP can re-learn missing controls…

Ben: True

Jared: Tool running in the browser probably creates other issues too I would think.

Ben: Likely yes. That and when you have several thousand tests and you need to re-record a couple hundred, the appeal wears thin.

Jared: The abstraction is the problem, not the approach and as always, I’m skeptical that the test can simply be replayed without any issues when there is dynamic content.

Ben: Agreed

Jared: I think I get annoyed by the ignorance they have of other tools, which is why programmers seem to keep reinventing the automation wheel.

Nerds have far too much faith in machines and solving problems through programming. Some good things come out, and some massively misguided efforts come out as well. But ultimately, these guys will decide our future, because they will dangle an appealing, human-free testing future, which will be doggedly pursued by management. If there are two choices for how work is designed, and one gives management more control, they’ll choose the latter.

Effectiveness is secondary. They’re not solving my problems. They’re solving developer problems. And management problems that arise due to poor work design. But they’ll probably win. Those of us who are skilled enough and market well will retain decent jobs, but there may not be many of those roles. Mostly people will forget how to do the skilled work. When they say ‘I wanted to help the manual testers’, I always hear ‘I wanted to help the shit testers (Even though they didn’t ask for any help)’.

I really wanted these tools to be of assistance in helping to put a crapload of fake testers out of a job, but I really think the reality will be different. While the bug reporting side of things might help developers fix problems faster and free up testers to do more valuable testing, the automation side of things will a) be farmed out to low-skilled low-paid workers who know enough to hack a little bit of javascript and b) spawn a bunch of new frameworks that deal with the abstraction problem that this tool has inherited from every other record and playback tool of its ilk. In the case of a) it may eliminate a bunch of drones writing test plans and test cases then attempting to execute them, but it’s shifting the problem into the automation sphere.

The fact that Google has released the client but not the server will be a deterrent for some, but it won’t take long before someone puts together a workable open source server and the crap automation tool + low paid maintenance worker farce will have a new player. It might mean that that you won’t need quite as many fake test drones in your organization, but in order to make a dent in the plague that is fake testing, we need something a little better than crap + 1.

2 Comments

  1. Chris K says:

    Hi Ben,

    Good post, can you expand on the abstraction problem as you see it?

    1. Ben Kelly says:

      Hi Chris,

      Thanks for stopping by.
      The abstraction problem is what you have when a suite of automated checks share similarities, but do not share code (such as those you get by using record/playback tools). Each check repeats the same code for any common action.

      Left as they are, your suite grows such that the effort you expend to maintain them outweighs their usefulness because a change in one page, be it element, DOM order, CSS or otherwise means potentially changing every single script that uses that page.

      Generally people either write or borrow an abstraction layer of some form or other e.g. the page object model, in order to minimize maintenance required when pages change. Tests share common elements and instructions for interacting with them instead of each test being completely self-contained. Rather than having to change the same item in every script that uses that page, you change it in one place.

Leave a Reply