2016-07-07

A word about false-positive and false-negative test-results, and why having a 'negative test-result' means 'passing the test'

Hi,


I am writing this because I sense there is a (legitimate) misunderstanding about some vocabulary we are using and I would like to try and clarify things. I will use an analogy from everyday life, that you might be familiar with, and I will explain why a POSITIVE test is a FAILED test, whereas a NEGATIVE test is a PASSED test. If it’s already making sense to you, you can pass, otherwise read on. There shouldn’t be any confusion about what is a false-positive test, and what is a false-negative test, which is which, and why they are critical.

First, notice that the usual phrasing is “one is taking a test”, or “one is being tested for something”.

So, when software is taking a test, software is tested “for” regressions.

Let me explain.




RnD and QA-oriented people, when coding automated tests, we are testing software “for” regressions. An analogy being: just like a chemical-test might reveal the presence of a molecule in your blood, a regression-test might reveal some incoherent or wrong behavior coming from the system-under-test.

So the software is really tested “for” regression(s) just like a person is tested “for” pregnancy, alcohol, narcotics or a disease. And, finally, the result of a test comes in two flavors only: positive or negative. And the meaning of this “positive” or “negative” status is:

-          Positive = Anomaly Detected,
-          Negative = Nothing To Report.

Except that we, humans, when dealing with a consumer, we negate this result and we call a negative test, a PASSED test, and we call a positive test, a FAILED test.

But the logic behind it remains the same and we, RnD and QA-oriented people, we must be familiar with it. There are only 4 typical situations, by the end of an anomaly search:

a) we might find some anomaly and be able to reproduce it (positive test),
b) we might not find any, because there aren’t any anomalies (negative test)
c) we might see some anomaly that isn't there, and then reconsider and force the status to PASS (false-positive test, status FORCED),
d) we might miss some (false-negative test).




All things considered,

-          Situation c), on top of causing a tremendous overhead on the QA-oriented people, it might make the product absolutely impossible to tests (eg. like when using non-deterministic algorithms or data structures),
-          Situation d), on top of causing an increased overhead on both RnD and QA-oriented people later in time, it also means “bad press” for the company at one level or another.

Working iteratively at minimizing the occurrences of cases c) and d) is what makes the whole integrated system reliable and our lives easier in the long run. 

No comments:

Post a Comment