testjutsu

I’m going to be attending CITCON coming up in Melbourne at the end of June.

There will be a bunch of people from all different tech areas to discuss CI and the testing thereof. It’s an ‘OpenSpace‘ format, so I’m looking forward to seeing what sort of things people want to discuss and hear about.

It’s a free seminar and I have it on good authority from people who attended last year’s event that it is highly worthwhile. The event is limited to 150 places and I believe many of them have already been filled, so if you’re considering attending then I’d act sooner rather than later.

I hope to see you there.

After blogging about tester advocacy I began to wonder whether I could come up with a code of conduct that testers could use to guide their actions and decisions. I don’t think you’re likely to get a one-size-fits-all code, so perhaps I should say a collection of heuristics by which a tester can guide his own actions.

I’m not talking so much about the day-to-day activities of a tester, though that certainly plays a part. I’m more concerned by the actions in his relationships with his peers, his employer and the people whom his testing may represent. I was hoping that with some careful consideration, I’d be able to put together a concise list of points.

I started listing the sort of things that are handy to remind myself about, but I quickly realised that they aren’t really that useful in all situations.

I listed things like

I will pro-actively educate myself in the art of testing.
I will find out who matters and test and report accordingly.
I will use all the resources at my disposal
I will not be the gatekeeper of the decision to ship/not ship

The problem with these is that while they work for me right now, they may not work for testers in other places, or with differing levels of experience, different duties, different responsibilities. The other problem with a list like the one above is that you could easily expand it to a hundred points or more.

I don’t want a laundry list, or a bunch of ‘thou shalt’ commandments. I am also wary of locking the art of testing into some sort of cookie-cutter formula/recipe to be followed by rote, rather something that is applicable in all situations that any tester can use to guide their thinking and actions in a positive way. This appears to me no small task.

I put it to the MAST list to see what they had to say about it. There was some interesting and spirited discussion around the term ‘trusted advisor’, but what I took away from the conversation was that Paul Szymkowiak‘s suggestion the exercise of attempting to define such a code of conduct is perhaps as important (or more so) than the results.

It is still something that I am spending time thinking about, but if the journey turns out to be as important as the destination, then perhaps someone else out there might gain something if I document my endeavours here. To that effect, I shall do my best to post updates here as and when I make progress on this front.

I have been reading Matt’s blog for a while now. Since October last year, Matt has been periodically expanding and evolving his thinking about ‘technical debt’ – the small, seemingly inconsequential short-cuts and hacks that we tend to implement or let slide in order to hit deadlines or save us doing more work (at least in the short term). Like most bad habits (such as the gradual accrual of fiscal debt, say, on a credit card), these things tend to lead to large amounts of pain down the track.

Rather than regurgitate what Matt has been saying, let me point you at his work. I think it is worthy of consideration for anyone who has had to interact with another human being in a professional capacity, ever.

• Technical Debt #1
• Technical Debt #2
• Technical Debt #3
• Technical Debt #4
• Technical Debt #5
• Technical Debt #6

If you’ve skipped to this bit of text because you’re not up for six essays on an unfamiliar concept, let me recommend that you at least read #6 before checking out details on Matt’s upcoming Technical Debt Workshop.

If you’re in the vicinity and have something you can contribute, then why not do it? If you’re not, then keep an eye on Matt’s blog. I’ve no doubt there will be some very interesting outcomes from the workshop.

The company I work for has been using subversion for several years now and it suits our needs reasonably well. It’s generally pretty easy to use; it’s certainly an improvement on CVS, but on the other hand there are some things that aren’t all that great. The way externals were implemented leaves much room for improvement and svn tends not to play nicely with mixed user permissions (such as when files are written alongside or over checked-out code by www-data). Subversion has a relationship with posix ACLs that is rather less cordial than the one described in The Pogues’ Fairytale of New York, but these are perhaps war stories for another time. These errors at least have the decency to fail with the requisite amount of melodrama.

More recently we’ve come across some more subtle and potentially nasty issues that subversion barely bothers to notify you about, if at all.

The way our shop works, we have a single trunk from which branches are forked. Changes are made on-branch before being tested and re-integrated into the trunk. Problems can arise when you make changes to a file on one branch (A), and move the same file on another (B).

Subversion treats moving files as a ‘delete’ and an ‘add’ (delete the old file, add a new file with the same contents and a different name).  If you integrate branch A and then branch B, the newer changes from branch A will be lost (as the file is deleted) and the branch B version will be instated with its new name. SVN does not warn you that a newer instance has been overwritten – charming.

If on the other hand, you merge branch B first, then branch A, then because subversion can’t find the the file to merge with branch A (because it was renamed in B), it simply reports that the file is ‘skipped’. No mention that changes to that file could not be merged as the file was not found just ‘skipped <filename>’.

Easy enough to miss when you have several hundred files scrolling past you at a rapid rate of knots.

Apparently, the best you can do is be very, very careful when integrating branches that contain deleted or renamed files. I’ve taken to pointing any sizeable svn output to a file and grepping the hell out of it.

A similar thing happens when you use nested branches.  This one is perhaps more understandable, but the level of reporting is just as woeful. If you branch from the trunk, then make a second branch from the branch then care must be taken if you attempt to merge the second tier branch directly back to the trunk. If you have added files in the second tier branch, they too will simply be ‘skipped’ if you attempt such a merge.

Merging back to the first tier branch is no problem, and is probably what the system was designed for. We discovered this issue when a developer created the nested branch to make newer updates to a very old branch. A better solution would have been to create a fresh branch from the trunk and merge the changes from the old branch into it, before making the required modifications.

Hopefully these are useful to some of you other SVN users out there.

I used to think that as a software tester, ‘if you’re doing your job well, then no one notices’. This sort of thinking was something I was perpetuating by misunderstanding my role as (merely) someone who performs test execution. I could see the benefit of the work I was doing and assumed that because no one else seemed really interested, they didn’t care.

From my experience, there is a tendency for testers to be thought of as second-class dev-team citizens, if not overtly then subconsciously. A developers job is perhaps easier to visualise than that of a tester. I’m not saying their job is easier, but for some reason a coder implementing programming algorithms by writing code seems easier to picture than a tester implementing testing algorithms by… ‘you guys just click on stuff to make sure it works, right?’.

I spent a lot of time arguing for recognition of testers and questioning why other members of the dev team seemed not to understand how important testing was. It was frustrating to me. It seemed like more often than not, when people took notice it was because something had gone pear-shaped because it was my fault (I’m the quality guy, right?).

It slowly dawned on me that it wasn’t because they weren’t interested, but because the way I was conveying the how, what and why of testing was not compelling. My colleagues knew development. They understood development, but their experience with software testing was limited. It’s not that they didn’t value the test team, but the way I was reporting my testing wasn’t as relevant to them because it was more or less a binary ‘Yes I have found problems’ or ‘no I haven’t’. This had a dangerous side effect of making me a defacto deployment gatekeeper.

They valued the ‘yes’ or ‘no’ a great deal, but because that’s all I was providing, and because that was their experience with testing, that’s all that was expected. I was thinking hard about testing and how the product might break and what else might go wrong, but I wasn’t really focused on how to get that across to the people that mattered. They assumed that when I said I was done, I knew enough to know that was true. To be honest, at that point, I didn’t know enough to argue otherwise.

As my experience in testing grew, I realised that test execution is a small (but of course important) facet of a tester’s role. I began to see the testers role as multifaceted and I became part devils advocate, part psychopomp and possibly large amounts of the damned from various circles of Dante’s inferno . I pushed to be included earlier in the project lifecycle. I asked questions, made suggestions, asked for documentation and information and made sure the people I was working with were across the thinking I was doing about risk and risk mitigation. I started talking in terms of how the testing we were doing was covering the risks we’d raised (and sometimes how it wasn’t). I didn’t always go the most diplomatic way about it, but gradually things began to change for the better.

The role of a tester to me now is to really be a central information hub, and a representative for those who may not have a voice (such as the end user). Reporting the test effort should be a matter of knowing who your audience is (or more broadly, who matters) and what matters to them. The development group I work with now have come to expect assessments of risk, analysis of proposed solutions – the thought behind the effort is as important as the effort itself.

I suspect that my early experiences are not uncommon. I have heard and read many accounts of testers feeling undervalued and misunderstood, being blamed for errors that go into production, being the gatekeeper for the decision to deploy. I don’t think testing is nearly as understood as a profession as coding is and it’s something I’d like to change.

I think the onus is on the software tester to be proactive in promoting their effort in such a way that it adds value to those that matter; so that the people they are supporting understand exactly what the tester’s role is. I think there is a need for education both of non-testers, to show them the value of software testing, as well as education of software testers so they are able to convey a compelling story of what their testing is and does and why it matters.

My situation is of course going to differ from other testing shops out there, both in the style of work and the relationship with developers in the development or project group. There may be an element of ‘picking your battles’, or making change in small increments over time. Testers might face extreme reluctance or hostility to change. I think at that point it becomes a matter of the tester being clear on the sort of behaviour they are willing to commit/adhere to, and those that they will not. More on that soon.

I am reasonably certain that somewhere in the world, a tester is being blamed for not finding this.

I am a little pressed for time right now, but I have been thinking a lot lately about the perception of testers by the people they support and what that can mean in the context of the projects they work on, the companies they work for and testers as individuals. I think there’s a rich vein of blogworthy material to mine here, which I promise to attempt just as soon as time permits.

Writing for the screen has many similarities with software development. It is a highly collaborative affair, and one that requires patience, skill, clear communication and quite possibly a penchant for the masochistic.

A screenwriter has only 120 pages in which to tell their story. It sounds like a lot, but when you can write only what can be seen or heard, it rapidly becomes very challenging. Every word must drive the story forward.

The story has to be one that works on film. If the story depends on internal states (what people are thinking and feeling), it may be something that is better suited to a novel. If it’s dialogue-heavy, you might have a stage play on your hands. Not every story lends itself to being on film.

A writer of spec scripts (scripts that are unsolicited) is, whether they know it or not, writing for at least four audiences. Firstly the production company’s readers.

Readers are paid a pittance to wade through massive piles of screenplays that (literally) tens of thousands of other writers submit to Hollywood every year. They’re the ones who sift the gold from the shit (and 99.9% is shit). They’re hoping that the next 120 pages that they pick up is going to be the nugget they can plonk on a producer’s desk and say ‘we have to make this’, and of course they’re generally disappointed. Poor bastards.

As a screenwriter, you have to grab their attention from the first sentence – the first word if you can, then hold it. Keep them wanting to turn the pages even if it’s the 30′th script they’ve read that day and their eyes are red raw. A spec script has to be that good.

After that, the script has to grab the imagination of a director. If you take a look at a novice screenplay, it’s generally full of camera direction and references to music and such. This is a no-no for writing on spec.

It is the director’s job to call the shots, not the writers. Okay, so a spec script with this sort of direction will be binned by a reader before it ever gets to the director, but the point is, screenplay writing is collaborative and you need to know what other people’s jobs are, and let them do it. That doesn’t mean you can’t capture their mind’s eye and suggest implicitly what the right shot might be.

Let’s say for instance you have an action movie where you have a hostage strapped to an operating table about to be operated on, and your next scene is your protagonist tending his wounds after a battle. If the last sentence of your operating theatre scene ends with the hostage being anesthetised, and the first sentence of your protagonist scene is him having a bullet wound stitched shut, you’re not explicitly telling the director to superimpose the needles for the scene transition, but the inference is reasonably clear.

You’re not telling the director how to do his job, but there’s a suggestion there if he wants to use it.

The screenwriter needs to write for the actors as well. If you don’t have characters that test the range of an actor, that challenges them, they’re less likely to want the part. You need to know how to build tension and create conflict and show them the opportunity to display their acting chops.

The novice writer probably starts out with the movie-going audience in mind and of course they’re important too. You need to have a story that works on screen, that grips the interest and imagination of the viewer, holds it for 2 hours and leaves them feeling that the time they spent was worthwhile.
As a software tester reporting bugs, your role is not so terribly different (although hopefully on a smaller scale in terms of length). You also have several audiences and several relationships to manage.

You need to report your findings clearly and concisely with enough detail to adequately convey the bug and the environment and context surrounding it. Make it easy for the person reading your report. If there is some heavy lifting you can do for them, do it.

Is your bug worthy of reporting? Relate the problem to risk. Why is the bug worth fixing? What happens if we ship without it being fixed? Who benefits from it being fixed (or who is it detrimental to if not). Create a compelling case for the resolution of your bug.

Don’t tell a programmer how to do their job. If you have theories about the cause, or suspicions about a more serious underlying issue, then note it down, but preceding it with ‘you need to fix’ or ‘I would have written it like this…’, or ‘the whole thing needs rearchitecting’ is unlikely to win you many friends.

Like a screenwriter, once you submit your bug, it is out of your hands. You want to send it into the world with the best possible chance of success. A screenwriter wants their play to entertain (and also come back with a large cheque attached), and build their reputation as someone who produces quality work. You want your bug to come back resolved and build the same kind of reputation.

Understand who your audience is, what they are looking for, and make a point of providing it to them.

Cem Kaner has an excellent document on his site about bug advocacy. Well worth the read.

For any budding screenwriters in the audience, if you haven’t already devoured the wordplayer.com columns written by Ted Elliot and Terry Rossio, then your next eight hours are in this link.