EverythingMiscellaneousSoftware Testing

The Outside Context Problem

My regular reader might have noticed that I (re)read a book called ‘Excession‘ recently. Penned by a gent called Iain Banks, who in my opinion, can excrete pure gold.

It is a great read. For those of you already familiar with his work, it probably needs no introduction. For those yet to experience it – if you have any time for sci-fi novels at all, then your life is not complete until you have read this one as well as ‘The Player of Games‘ and better yet ‘Use of Weapons‘.

In Excession, he coins the term ‘Outside Context Problem’ that he describes thus:

An Outside Context Problem was the sort of thing most civilisations encountered just once, and which they tended to encounter rather in the same way a sentence encountered a full stop. The usual example given to illustrate an Outside Context Problem was imagining you were a tribe on a largish, fertile island; you’d tamed the land, invented the wheel or writing or whatever, the neighbours were cooperative or enslaved but at any rate peaceful and you were busy raising temples to yourself with all the excess productive capacity you had, you were in a position of near-absolute power and control which your hallowed ancestors could hardly have dreamed of and the whole situation was just running along nicely like a canoe on wet grass… when suddenly this bristling lump of iron appears sailless and trailing steam in the bay and these guys carrying long funny-looking sticks come ashore and announce you’ve just been discovered, you’re all subjects of the Emperor now, he’s keen on presents called tax and these bright-eyed holy men would like a word with your priests.

It even has its own wikipedia entry.

I’m sure that any tester that has been around the block once or twice has experienced a situation where they were blindsided by a problem (quite possibly after the system under test went to production) that made them think something along the lines of ‘Wow – never in a million years would I have thought of that as a possibility’.

The Outside Context Problem (OCP) is a little different to the SEP heuristic, in that rather than mentally deleting problems that appear to be in someone else’s jurisdiction, OCP describes those things that are nigh on impossible to factor into your risk analysis simply because you have no concept of it prior to it happening.

Of course this has particular relevance to software testing – we’re supposed to be the ones who can think outside the box; who can come up with the oddball issues that the developers and the stakeholders and the analysts didn’t think of. So how do we factor in OCPs to software testing?

Personally, I don’t know that there’s a definitive answer to that question. You can factor in extra testing time for that X factor, you can look at similar projects and products where applicable, to see if there were things that they didn’t account for, you can get advice from your peers, you can do all sorts of end-user testing to gauge how they might (mis)use the product and so on, but depending on what it is you’re testing you might have one, all or none of these things at your disposal. They still won’t guarantee you coverage of an OCP.

We encounter OCPs often when we are still learning and hopefully less and less over time, but you can’t ever completely eliminate the possibility of another one coming along. It’s why a one-size-fits-all standard simply won’t work for the software testing industry.

Beyond there being a multitude of known differences to factor in from one project to another, you have any number of unknowables that no amount of planning will allow you to say ‘I’ve covered all the bases’.

In terms of the OCP, perhaps the best you can do is cover your arse. If you can list the risks you have identified and present them to your audience, get sign off that the set of identified risks is acceptably comprehensive, then as long as you have done the best you can with the resources you have at hand, there is not much more that you can do.

That’s probably cold comfort for people working with virulent strains of bacteria or virii, or any sort of system where lives depend on getting it right the first time. You might be able to legally cover yourself, but you still have to sleep at night and I wonder how many people following a catastrophe (that they didn’t see coming) do not wonder ‘Is there something else I could have done?’

2 thoughts on “The Outside Context Problem

  1. Ben,
    You are right that there is no way to cover all bases.

    But I don’t like the backside-covering strategy (although I’ve done it πŸ™ ) because no other position is knowledgeable enough in testing matters to assess the risks as the tester team.
    Presenting a big powerpoint of test directions and asking ‘anyone can say if we missed a point?’ does not diminish the bad-feelings – or responsibility – when a problem comes after the product is released.

    If I may try an analogy…
    Let’s say the financial executive of a company feels uncertainty on his numbers… he can call all major executives from the other departments (marketing, development, legal…) and show the financial strategy.
    After a big accounting problem, it would not make sense for him to say ‘well, you all said it was ok!’ – Because he is the money professional, and even if it is nice to see him sharing his strategy with others, the responsibility is still at his table.

    Can you tell a bit more about your experiences with such cases?
    How do you conduct the arse-covering session? How was the reaction when a surprise problem arose (if there was any)?

    Thanks for the insightful readings!

  2. I wouldn’t necessarily say that no one else is as qualified to assess the risks. In fact, I would suggest that in many instances, a software testing team has a fairly narrow field of view.

    I believe the test team has a central role as an information resource to many other units, often outside the project team, and it may well allow them a wider perspective that say the developers have.

    That said, if the information flow is not reciprocal then I think the likelihood of an OCP is increased. Sales, marketing, executive, legal, are all departments of a company that I can see having unique insight into possible risks of a project. There may be many more depending on the nature of the company and or the nature of the product itself.

    I maintain that the best you can do to avoid an OCP is to gather as much information from the resources you have at your disposal, assess the risks you are able to, and circulate said risks to everyone you can identify that matters, (as opposed to presenting testing directions, which may be meaningless to them). Ask them if they can think of any other scenarios that might be a problem and factor those in. Reassess the risks as things change.

    That at least allows you to attempt to mitigate the risks you can identify.

    I think your analogy is an incomplete one. You say ‘an accounting problem’, but I am talking specifically about an Out of Context Problem.

    Let’s extend your analogy – Perhaps something like data loss because an apostrophe in someone’s surname wasn’t handled by their off-the-shelf accounting software?
    It might not occur to a finance exec to have the company’s testing team (assuming it had one), put the software through its paces. Even if he or she did, this problem might not crop up immediately. They might have assumed that the product was sufficiently tested by the software company’s test team – not an unreasonable assumption for a non-technical individual to make.

    Said exec may feel completely justified in saying ‘there was nothing more I could have done’ if they had taken the appropriate precautions that their training and experience afforded them. Moving forward, I’d be willing to bet said exec would advocate thorough testing of any new software product because, unlike before, he now has a reference that this sort of thing can happen.

Leave a Reply

Your email address will not be published. Required fields are marked *