Pull reporting & Push reporting

Tuesday, October 11, 2011

Communication in testing is something that has been occupying me of late. Not a surprise to anyone that reads my material with any sort of regularity. How we report our findings is a massive part of our job and yet when it comes to talking about software testing it seems to be the poor cousin – planning and execution seem to get a larger share of people’s attention. Maybe that’s fair enough. We probably spend a lot more time planning and executing stuff at the coalface than we do communicating about the testing we have done (or plan to do).

To me, effective test reporting is the glue that holds it all together. If you can’t (or don’t) report your findings effectively, in a way that your audience finds useful, then your testing is not effective.

It occurs to me that test reporting seems to fit within two categories. I call them Pull reporting and Push reporting.

Pull Reporting is where your audience comes to you for information. Generally they have questions that they want an answer to, such as ‘When will you be finished testing?’, ‘How stable is Product X?’, ‘Did you find any bugs in interface Y?’ and so on.

Setting aside for the moment the quality of these questions, let’s talk a bit about the sort of situations in which pull testing arises. Stand-up meetings, drive-by situation reports, weekly meetings are all examples of pull reporting. In each case you have an audience that (at least to some degree) has context in mind and is requesting specific information. In my experience it’s the drive-by sit-rep that can be the most challenging, often because I’m in the middle of working on something else and I need to context-switch in order to answer. I don’t know about you, but it takes me time to switch out of something I’m deeply focused on in order to answer something unrelated.

In this situation, there are a few things you can employ to buy yourself some time.

  • Remember to breathe. If nothing else, oxygen is good for your brain and taking a deep breath gives you a few more seconds to switch over and consider the question posed.
  • Ask your audience to repeat the question. If you were in the middle of something else, it’s possible you heard something akin to Charlie Brown’s teacher rather than a question. Ask them to repeat it.
  • If you don’t feel like you can effectively answer the question right this second, ask for time. The question might seem urgent, but it’s rare that the situation is so mission critical that you can’t say ‘Give me 5 minutes, I’ll come to your desk (or call you back etc) with what you’re after.
  • If you are ready to answer, go ahead and answer. If you are speculating about something, make sure that you identify that you are speculating. ‘I know x, y and z. It may be that a & b, but I’m not sure’
  • Poll them to see if you have given them the info they’re after. It shows them you actually care about what they’re asking and it gives you immediate feedback as to whether your report was effective.

The drive-by report is also a great opportunity for an ambush. Knowing that someone is focused elsewhere and will need to context switch – that’s a great time to go on the offensive if you immediately want someone on the back foot. It’s a dick move, but it works. Hopefully you’re not on the receiving end, but should it happen, then remember that not all questions need to be answered. Beware ‘why’ – especially when it’s accompanied by a pronoun as the third word (why did you… why didn’t you… why aren’t you etc). You may find it more constructive to turn those sorts of why questions into ‘what’ questions. ‘What has happened that is concerning you?’ Answer that, and then you can decide whether the why is worth answering. Suggesting a different venue and time for the discussion is probably also a good idea also.

If you do find yourself in a position where you can reel off a report on request, you might like to consider using Michael Kelly’s MCOASTER heuristic. It’s a handy list of items that your audience may be interested in. You don’t have to report on every single item. Pick the ones you think your audience wants to know about (and check in after to see if they want more).

Push reporting is the opposite in terms of who has context in mind. Push reporting is when you have information that you believe your audience should know, but they are not aware of it. In order to effectively convey the information you have, you need to understand what is important to them, what they know already and how to frame your information in such a way that it is easily digestible and meaningful to your audience.

Examples of Push reporting are Bug reports, Reporting showstoppers or mission-critical issues, Risk reporting. Time is generally on your side (though not always). Take the time to make sure that your audience is properly framed. Brief them on your subject matter before you launch into your report. Even if you have an appointment with an agenda for your report, you can’t assume your audience will immediately be ready to hear what you have to say.

Push reporting is where your ability to tell a compelling testing story comes to the fore. Our job is to present meaningful information to people that matter. If all I do is present that information, then I really feel that I’m not living up to my potential as a tester. I would rather present that information in such a way that it presents a compelling call to action.

In my presentation at CAST I used an example from the movie Hunt for Red October. In this clip, the sonar operator aboard the USS Dallas (Jonsey) presents information to the captain about what he has discovered.

The Hunt for Red October: Jonsey Reports

Keep in mind that the raw information that Jonsey has is

  • He knows he has heard a submarine that they don’t have previous records of.
  • He lost the sub but picked up a sound the computer tells him his magma displacement
  • He has done some testing and believes the sound is man made.
  • He has picked up the sound on three separate occasions.

That’s it. That’s the raw info he had. Everything else he presented to the captain he has put together from what he thinks that information probably means.

He could have just presented that information alone. If he had, it’s possible that the captain would have extrapolated the rest. They tend not to put morons at the helm of nuclear powered, nuclear armed naval vessels. Still, if this is all the captain gets, then the onus is on him to do the legwork. What Jonsey has done is taken raw information and put it into a format that greatly helps the captain see the significance of the information. The only thing left for the captain to do is to decide whether he buys Jonsey’s version of events and then act accordingly.

This is about as close to the perfect push report as you’re likely to see. It may be that your own push reports aren’t as high-stakes as hunting a rogue Russian sub, but the principle remains. If you have information that your audience needs to hear, take the time to put it into a format that helps them see the significance of that information.

Ask yourself what you want from revealing your information. Do you think certain action should be taken? How does your information support taking that action? Do you want to highlight risks associated with your information? What are some credible issues that could arise as a result? How does your information support that? It may be that you have information that you simply wish to relay in case it is useful. Nothing wrong with that, but think about the information you are presenting and whether or not there is more to it than what you have.

Remember also that however compelling your case, the final decision is unlikely to be yours. Suggest a recommended course of action. Don’t demand. There may be other factors that you are not aware of that have some bearing on the situation. It’s your job to find and present the information. Let them take responsibility for what happens next. That’s why they get paid the big bucks.

Elicit feedback. Just as with pull reporting, you want to find out whether the information you gave was useful. If not, why not? You can take the response you get and feed his back into your testing.

I think that the distinction betwen pull and push reporting is a useful one. It provides me a framework to think about how to report and what to include. I’m curious to hear your thoughts or how you think about test reporting.

Leave a Reply