Software testing? Not going away anytime soon.

Monday, October 24, 2011

 

This one is a response to a blog post by Eric Jacobson, who wrote up some notes from James Whittaker’s keynote at STARwest.

I myself did not attend STARwest, so I have only these notes to go on, but as I read them a few things struck me as being odd. Some of Mr. Whittaker’s assertions I agree with, though I think that his reasoning for them is flawed. From Eric’s blog:

testers do not get respect in their current roles…the reason we lack respect is all the recent software quality “game changers” have been thought of by programmers

The game changers that he listed were strange to me. If I read the post correctly, they are there to help back up the assertion that testers should be fearful for their jobs. Let me list them, then list reasons why after most of them my reaction was either ‘meh’, or ‘huh?’

  •  Today’s software is easier to update and fix problems in (easier than software from 10 years ago).

There are some words missing in this statement. I don’t know what they are, so let’s try some variations.

Some of today’s software is potentially easier to update an fix problems in (yadda yadda), provided that it is well designed, well structured, well written, well commented and well tested.

or maybe

Today’s software is easier to update and fix problems in (yadda yadda) due to the advancement of IDE’s and debuggers as well as frameworks built into some modern languages that assist structure and testability.

Well okay, both fair points. Both have the potential to increase software quality. I’m not seeing any particular threat to software testing here though.

Threat rating: Meh (with a side of Huh?).

  • Crash recovery – some software can fix itself.

Ok. Couple of things

1. So what?
2. ‘fix itself’ is a pretty nebulous statement. Maybe James said more on this and because I missed it, I can’t speak to it specifically.

Netflix servers have the Chaos Monkey that kills processes at random to help them work out how to make their product more robust. That’s cool. Does that mean their testers are on the way out? Other software that crashes can sometimes recover and or send logs back home so stuff can get traced and fixed. That’s also great, but the software has at that point, already crashed. If it keeps running, it may well be in a state that is detrimental to the end user. Whatever. This point is irrelevant to testing getting in the way of quality.

Threat rating: Huh?

  • Reduction of dependencies via standards (e.g., most HTML 5.0 websites work on all browsers now).

Would love to see the reference to the stats backing up this statement. Tell it to the banks and insurance companies and other ultra-conservative behemoths mired in their own red tape and bullshit processes. Those poor bastards are still on IE6 or 7. Aah Standards. Yep. Standards are great. I don’t know if this is an attempt to say that cross-browser testing is no longer required. It may make checking functionality across browsers a little easier, but again I’m not seeing a threat to testing here either.

Threat rating: Meh.

  • Continuous Builds – quicker availability of builds makes software easier to test but it has nothing to do with testers.

Eric already commented here that build tools have nothing to do with testers – well, not far beyond lighting up a suite of unit tests and potentially helping deliver builds to testers faster.

Threat rating: Meh

  • Initial Code Quality (e.g., TDD, unit tests, peer reviews)

These are all great things and if anything are complementary to software testing. TDD and unit testing can help Developers put better code in place sooner and find out about stuff that is broken at the unit level faster. Code reviews have the potential to help coders in the room become better at their craft (and if there are testers in the room, it can help them understand how coders think, how their code is put together and any bad habits that their colleagues might have).
Yep. It’s a positive for code quality. Not seeing any threat to testing here.

Threat rating: Meh

  • Convergence of the User and Test Community (e.g., crowd source testing, dog food testing).  Per James, “Testers have to act like users, users don’t have to act.”

Hmm, having had first-hand experience with crowd-sourced testing, I have to say this has a long, long, long way to go before it is any threat to the profession of software testing. It is a useful supplement to software testing, however using it as a replacement for software testing would be a recipe for pain.

Having end users test a site can be great and the feedback you receive can give you some great insights into how they want to use a product and what they want to achieve. To equate that with ‘Users are better because they don’t have to pretend’ is naive. There are a couple of problems. Firstly they’re not trained to look for problems. They may stumble across them and they may find things that your software testers do not. That doesn’t mean that testing is not a worthwhile activity. Secondly, users aren’t trained to describe problems in a concise or logical manner. There’s only so much troubleshooting you can do with the description ‘It’s shit and it doesn’t work. I paid good money for this, so fucking fix it’ (spelling and grammar corrected for the sake of legibility). They don’t generally have access to logs, or provide screenshots or video of how a problem came to be.

So – to my knowledge, not thought of by programmers and not a threat to testing.

Threat rating: Meh.

Eric then lists 4 additional ‘painful facts’ about testing (I’m assuming these were taken directly from Mr .Whittaker’s presentation)

1. Only the software matters.  People care about what was built, not who tested it.

Depends on who you mean by people (and by ‘matters’). End users? Sure – unless the software doesn’t do what they need, or crashes on them constantly. Then they’re all about who tested it. Funny about that. What about other people?
Again – how is this painful? Hindsight is 20/20 and it’s possible that valuable info from a software tester is less valuable to the audience after the fact.  A software tester’s job is to provide information to decision makers to help them make their decisions. Talented testers do this in such a way that the information is timely, relevant and valuable. Do that consistently and you can bet your arse that they’ll care about who tested it.

 

2. The value of the testing is the activity of testing, not the artifact.  Stop wasting your time creating bug reports and test cases.  Start harnessing the testing that already exists (e.g., beta testing).

The first sentence here is gold. The second sentence is a bit WTF? Mostly because the example given was ‘beta testing’ – were there no other examples of testing given? I’m all for ditching unnecessary test documentation. Do as much as you need and no more. I’m also all for harnessing existing testing (as distinct from checking, which should also be harnessed, but is not the same thing). It also doesn’t mean ‘when you’re done coding, throw it at your users’.

 

3.The only important artifact is the code.  Want your tests to matter?  Make them part of the code

I wish I had an analogy to better illustrate how useless this statement is. Important to whom? In relation to what other artefacts? This statement is probably referring to ‘checks‘, but it manages to also dismiss other forms of testing with a sweeping blanket statement that doesn’t actually mean anything. Wow.

If you have a bunch of checks, sure. Automate them. Build testability into your code, absolutely. Your code and probably product quality will be better for it and chances are your testers will thank you.

 

4. Bugs don’t count unless they get fixed.  Don’t waste time logging bugs.  Instead, keep the testers testing.

Aside from the notion that bugs count in any case, I couldn’t agree more, but again I fail to see how is this painful to testing. Log bugs if your developer can’t fix it when you tell em, be that in a bug tracker or on a post-it attached to a story. Whatever works, just get em fixed (you know, unless they don’t actually matter to the people that count).

I disagree with Mr. Whittaker about the reason that there is a lack of respect for software testers in the industry. The ‘game changers’ that he talks about are largely produced by programmers in order to make life easier for … programmers. Sure they’re also generally good for software quality, that’s not why we testers get a bum rap.

The reason we don’t get a whole lot of respect is because the industry is saturated with fake testers. Not only that, but there are complimentary industries *coughcoughcertificationcough* that make a tidy profit churning out crap material to the gullible and the ignorant, helping to produce the next generation of fake testers.

Now it may be that Mr. Whittaker is taking these testers to task when he says that developers are getting better at testing, but testers aren’t getting better at programming and “It’s only important that testing get done, not who gets it done.” If so, then great. I agree. The faster fake testing dies, the sooner the rest of us can get on with it. I might finally be able to hire someone without having to read through the resumes of 300 oxygen thieves.

I do think that his exhortation to “get a specialty and become an expert in some niche of testing (e.g., Security, Internationalization, Accessibility, Privacy) or learn how to code.” is just scaremongering for any tester who knows their craft, but hey, controversy puts bums on seats, so I suppose I can’t fault him too much for that. Sure, become a specialist if you have an interest in it. Knowing how to code certainly won’t hurt you, but I know a number of world-class testers who do not code and are testing generalists. I do not believe their livelihoods are in danger any time soon. As testers, we bring something different to the table. Some programmers are excellent testers. I like working with them. I encourage them to get better at testing. It helps me to help them look good. For coders, being able to think like a tester is a skill worth developing. That goes beyond being able to write unit tests.

So as far as controversial keynote speeches go, this was probably quite entertaining. As far as it containing any real substance to support an assertion that a skilled software tester should be concerned about losing their job to coders though – I just don’t see it.

6 Comments

  1. Scott Barber says:

    You and I interpret some of the comments differently, but I didn’t attend the keynote either. Some of our conclusions are also different, but I wouldn’t say that they contradict one another either because they are based on different interpretations of quotes out of context.

    If you haven’t already, you might want to take a look at my comments to Eric’s post (http://www.testthisblog.com/2011/10/james-whittakers-starwest-keynote.html).

    Looks like I should be consolidating those thoughts into a post of my own. :)

  2. Ben Kelly says:

    G’day Scott,
    Yeah I read your comments. As you say, I think we’re saying similar things and coming at it from slightly different angles. It’s inevitable (given that we weren’t there), that we’ll be taking stuff out of context and possibly misconstruing what was meant, although James Whittaker also posted a response to Eric’s blog and said it was a pretty good synopsis, FWIW.
    I think underpinning both of our responses is the skill of the individual tester. If you know what you’re doing and you are constantly educating yourself, then you’ll land on your feet even if there is a major industry shakeup. The poo-flinging script monkeys and the dead eyed robots will probably not be so lucky.
    Will keep an eye on my rss reader for your next blog entry :)

    1. Scott Barber says:

      Yeah, pretty much. Keep adding skill so you can keep adding value. As long as you’re adding value, you’ll have a job. Might be called a tester, might not… personally don’t care. :)

      [BK] Agreed.

  3. I must start with the disclaimer:
    I wasn’t at STARWest and so the only information I have about this keynote is what has been written both here and in Eric’s original post.

    “The ‘game changers’ that he talks about are largely produced by programmers in order to make life easier for … programmers”

    This pretty much sums up what I thought when I read it. It is not helped that the impression that I get about testing at Google is that it is very skewed towards the SDET idea, which indicates to me that James ends up looking closely at the programme type solutions since that is what they implement and skips the ideas and innovations around ‘manual’ testing. Basically, I think there is a bit of confirmation bias that infected this talk.

    1. Ben Kelly says:

      Thanks, Rosie.
      I was going to watch it and then update this post with my comments. Turns out it gave me enough material for another blog post :P

Leave a Reply