Skip to main content

Automation Effectiveness Formula
or
What's The Value of Test Automation

Preface

The question is raised again and again in different forms:

  • What's the value of test automation?
  • Where do you gain from automated tests?
  • How many bugs did you find with your automated tests?
  • Automation? Why bother?... and so on.

Here is an attempt to put an end to it, to give an ultimate answer. :) To create a formula. The formula for test automation effectiveness. :)

Seriously, it's not the formula which is important, but the things you consider to be important to calculate effectiveness.

I am assuming that the only real value is engineering time. We want to do more with less afford. "Bang for the buck" that is. What we also have to assume here is: all the testing we automate is useful. Other words, it would be executed manually anyway.

Formula

So

AutomationTime - time spent on a test suite automation including test creation and maintanance.

ManualTime - time that would be spend repeating the test suite manually.

Let's define effectiveness:

(1) Effectiveness = ManualTime / AutomationTime

Take a deep breath here ... 

For the same test suite, more time we would spend performing it manually - bigger the effectiveness of automation. Less time we spend on the suite automation - bigger the effectiveness. Bigger the value is, better we're doing. If the value is less then 1, we're in trouble. 

Now, let's look closer: 

(2) AutomationTime = TestCreationTime + ReleaseTestMaintananceTime * (NumberOfReleases - 1)

where

TestCreationTime - time needed to create the tests

ReleaseTestMaintananceTime - time needed to adopt tests to the product changes

NumberOfTestCycles - number of test executions. Most tests gets executed only several times for release. Some tests gets executed daily, though. Naturally, the effectiveness is going to be different for a different group of tests. "-1" comes from assumption that tests are created for the first release so there is no maintenance needed for the first release.

(3) ManualTime = OneManualExecutionTime * NumberOfTestCycles * NumberOfPlatforms

where

OneManualExecutionTime - time needed to run all automated tests manually once.

NumberOfPlatforms - number of platforms we are executing the tests for. That includes not only platforms but all the variants we're executing the tests for. For example, all variations of deployment servers counted too, if the tests are created to work with various servers.

Hence, here is the formula:


Effectiveness = OneManualExecutionTime * NumberOfTestCycles * NumberOfPlatforms
TestCreationTime + ReleaseTestMaintananceTime * (NumberOfTestCycles - 1)

Estimations

Everything below is a rough estimation. Reality is believed to be better than the estimation.

For experienced test developers in case of infrastructure is ready, TestCreationTime / OneManualExecutionTime value could be 3 or even less. 5 sounds more realistic. Some team nowadays even learned to generate tests, so, for them, that value would be close to zero.

ReleaseTestMaintananceTime in average is believed to be less then half of OneManualExecutionTime for sure. Actual time could vary from zero to, sometimes, 1.2 times of OneManualExecutionTime.

That gives:


Effectiveness ~= OneManualExecutionTime * NumberOfTestCycles * NumberOfPlatforms =
OneManualExecutionTime * 5 + (OneManualExecutionTime /  2) * (NumberOfTestCycles - 1)


= OneManualExecutionTime * NumberOfTestCycles * NumberOfPlatforms =
OneManualExecutionTime * (5 + (NumberOfTestCycles - 1) / 2))


= NumberOfTestCycles * NumberOfPlatforms =
((10 + NumberOfTestCycles - 1) / 2)


= 2 * NumberOfTestCycles * NumberOfPlatforms
(NumberOfTestCycles + 9)


Here is a table demonstrating calculated values of estimated testing effectiveness for different numbers of test cycles and platforms.

32

3.12

4.68

6.24

7.8

16

1.65

2.34

3.12

3.9

8

0.78

1.17

1.65

1.95

4

0.39

0.59

0.78

0.98

2

0.2

0.29

0.39

0.49

Test cycles/Platforms

2

3

4

5

Appendix

One thing to note is that with the number of test cycles being really big, the approximate formula of effectiveness is:


Effectiveness ~= OneManualExecutionTime * NumberOfPlatforms =
ReleaseTestMaintananceTime


= NumberOfPlatforms * (OneManualExecutionTime / ReleaseTestMaintananceTime)


Last ratio is indeed the main driver of the automation effectiveness. In the estimation above the ratio is represented by the "2" multiplier in the numerator.

Since OneManualExecutionTime is a given value for any given test suite, we can say that the effectiveness, roughly, is defined by number of platforms (proportional) and test maintenance time (inversely proportional).

Shura

Aug 16 2006

 
 
Close
loading
Please Confirm
Close