Main attributes of test automation

Below are some of the attributes of test automation that can be measured:

Maintainability

• Definition : The effort needed to update the test automation suites for each new release.
• Possible measurements : The possible measurements can be e.g. the average work effort in hours to update a test suite.

Reliability

• Definition: The accuracy and repeatability of your test automation.
• Possible measurements: Number of times a test failed due to defects in the tests or in the test scripts.

Flexibility

• Definition : The ease of working with all the different kinds of automation test ware.
• Possible measurements : The time and effort needed to identify, locate, restore, combine and execute the different test automation test ware.

Efficiency

• Definition : The total cost related to the effort needed for the automation.
• Possible measurements : Monitoring over time the total cost of automated testing, i.e. resources, material, etc.

Portability

• Definition : The ability of the automated test to run on different environments.
• Possible measurements : The effort and time needed to set-up and run test automation in a new environment.

Robustness

• Definition : The effectiveness of automation on an unstable or rapidly changing system.
• Possible measurements : Number of tests failed due to unexpected events.

Usability

• Definition : The extent to which automation can be used by different types of users (Developers, non-technical people or other users etc.,)
• Possible measurements : The time needed to train users to become confident and productive with test automation.

Measurements may be quite different from project to project and one cannot know what is best unless one has clearly understood the objectives of the project.

For example, for software that is regularly changing, with frequent releases on many platforms, the important attributes will be ease of maintaining the tests and - of course - portability.