Dimensions of Testability

Posted on Posted in Software Testing

In early 2015, Maria Kedemo asked me if I’d like to work with her on a workshop around testability. At the time, I wondered whether there was really enough in the subject to warrant a workshop. Testability seems like a pretty straightforward topic, right? It’s how easy something is to test. I’d seen James Bach’s early version of his testability model and while there were some things to think about, it still didn’t seem that deep.

Then we started discussing it and I realised I was wrong.

The concept of testability is an easy one to grasp, but when you look into things that can affect testability, that’s a very deep rabbit hole indeed. There are a number of existing models of testability around, but none of them expressed what we were looking for in a way that we wanted. It took us a long time to be able to articulate specifically what we wanted to express about testability, but ultimately we wanted to show the relationships between the many different things that affect testability. We came up with what we call the ‘Dimensions of Testability’ model.

Maria and I discussed, argued, reasoned and in between tore our own hair out to get to a place where we felt the model was useful. It is still a work in progress and likely always will be, but in this form, we think it shows some important relational aspects of software, the people that interact with it and each other and the surrounding environmental concerns that affect them. I plan to delve into each of these areas in greater detail in the future, but for now, let me post the model itself so that others might find it beneficial.

8 thoughts on “Dimensions of Testability

  1. Hi Ben, I was looking at your model and though you’re not offering any other description than the graph here, what struck me was the representation of the circles as mostly self-supporting entities. If I only focus for example on the Tester one, it just seems that the tester does have all skills & knowledge needed, and these skills, if one is in the contextually-fit mental & physical state, will be expressed in the tester’s work. That was my first reading of the left-hand side of the graph.
    Isn’t it often the case, at least in my practical experience, that the testability of a project is not correctly assessed by a tester lacking the expertise needed in a specific project? That some of the product’s testable features have been left out of the testing strategy not by their “own fault” but by not having been evaluated as testable? Isn’t in such a case testability at the boundary between declarative (and tacit) knowledge/skill and needed or required skill by the project-specifics? I just feel that the representation with this circles suggests there are only marginal overlaps between tester and product, on one hand, and if we take a within-subject perspective, between owned and needed skill. I have the same conflicting reading of the Product side of the graph: the product vision seems to be fully expressed in the product through the team’s effortful implementation of code in a given environment (I am simplifying my reading a bit). However, the testability challenge appears also when the product vision is only partially reflected in the product itself and there are strong pushes from the relationship side of your graph towards testing in the product vision frame, not in the frame of the product’s natural evolvement.
    My main concern is with this “free-floating” reading enabled by a simple graphical choice of en-circling: it seems to accommodate isolated explanations, but disengages thinking from relationships and interactivity. Maybe I’ll understand more when you add a bit of explanatory text to the image 🙂

    1. Hi Sorina,
      Thank you for commenting on the model. This is how we can evolve and improve.
      The way I see the model is actually a sphere of elements which we have included in our model. All of these elements interact and together create testability. It is however difficult to represent the model in a self explaining way to show the different dimensions and how they might affect one another. The model is a lot more complex than it looks and I agree it doesn’t really represent what we are aiming to explain. We are still looking for new ways of representing the model of elements which we think affects testability.
      The model is used as heuristics when addressing how easy it is to test in a specific context. It is still usable in it’s current form. As I see it, the model will always need explaining text with it no matter how we change it.

  2. Hi Ben,
    Over the past few months I’ve been thinking about testability a lot and I’ve been talking about it at work (lightning talk, other talks and discussions). I’ve come to the conclusion that the awareness of testability and how broad it is, is pretty low (at least where I am). Regardless of the reason, I’m now thinking of three things 1) how to help people understand and internalize testability awareness into their daily decision making (I think every decision counts towards decreasing or increasing testability) 2) what kind of model would help them to grasp testability AND decide what to do about it.3) how to express testability in business terms (i.e. create a business case for it).

    I’ve run a couple of small exercises where I ask people to brainstorm the question “what would make it easier to test something” and then tried to map the answers on the 5 categories present in James’ model. This exercise kind of serves its purpose in that it helps people understand the broadness of testability. However, I’ve observed that the exercise does little to show the dynamics and interrelations of aspects of testability, how different decisions will have an impact on testability day in and day out. I also feel like process needs more attention with regard to testability.

    Recently, I’ve adopted alternative names for the 5 categories (borrowed from Richard Bradshaw) because I want to communicate in a language that people in different domains can understand (switching “epistemic testability” for “known/unknown”).

    One thing I’ve also come to realize is that instead of having tester as a role be a prominent part of the model, it’s worthwhile having all and any people who test included. In your model tester is prominent and I’ve also considered tester as large part of testability. But then again… is it really so? Maybe our own biases put the tester in the center while it could be worthwhile to investigate the skills, knowledge and engagement of other roles who also test (esp. devs and analysts, I don’t have experience with product owners). I think their skills in and awareness of testing can have a huge impact on testability and quality overall.

    So I find your model as a good alternative to map the overall space, though it’s somewhat lacking the specific examples that I appreciate in James’ model.
    I’m toying with the idea of creating something else so hopefully I’ll be able to express what I mean with all of the above.

    thanks for your work on this! It’s good to have something to build on 🙂

    Helena

    1. Hi Helena,
      I am very happy to see you are one of the persons advocating for testability. The lack of awareness is exactly why Ben and I started the mission of bringing awareness to testability. Our model emerged from using and discussing James Bach’s model and we realized people in our workshops found it easier to understand our model as heuristics to figure out what might affect testability. I would say it is an addition to Jame’s model. You may use them both.
      One exercise we use in our workshop is exactly the one of coming up with things that would make testing easier ( what are the struggles you have right now). These are also mapped. Then we discuss them and we probe questions. What if we added more logs? About the dynamics we have a short but powerful practical exercise.We used to have a simulation to show the dynamics but it was too exhausting and took too long.

      Yes, the tester is a prominent part of the model, but note, the tester is anyone who tests. Thus skills, motivation, knowledge is closely tied to the person who tests. It is also why it is of significance for how easy it is to test. But imagine if you think you have all the knowledge you need to test a specific product. Then we have what is called perceived testability. You might think you have a high degree of testabilty but due to your ignorance you will not find the important bugs.

      We have not yet written much to explain the model, however you can read more about it in the article I wrote for Testing Trapeze:
      http://www.testingtrapezemagazine.com/magazine/december-2015/
      We intend to write more about testability and the model here on Ben’s blog and on my blog as well .

      Keep on bringing awareness to testability. I would love to hear more about the work you do.
      Thanks!
      /Maria

Leave a Reply

Your email address will not be published. Required fields are marked *