Zombie Testing

    Mindlessly  executing a program with the intent of filling spreadsheets with useless data, writing thick boiler plate bug reports, and meeting company metrics.  Find bugs is a secondary pursuit.  Any requirements verifications should be done to the letter; there’s no reason to check the intent of the requirements.

It’s true that zombies walk among us!  Your fellow software testers might be zombie testers!  But how do you know?  I’m going to lay out some indicators that might out your colleague as a zombie:

• They act with mindless disregard toward their job (Passive not Proactive).

Testers that have turned display a mindless disregard for their job.  They do not think; their brains are not buzzing with activity!  They are not analyzing requirements, looking for ambiguities.  They are not listening to conversations about new software implementation.  They are not engaged with investigating bugs to find the root cause.  They don’t communicate their unsubstantiated concerns and hunches to the developers or the project managers.  Zombie testers never pick up on disconnects between technical people and customers, nor would they intervene to increase understanding anyway. They particularly like it when these test cases are written by a QA manager and just executed by them.  Less thinking = better!

• They protect the status quo, rejecting new methods of testing.

Not only are their minds soft, but they will seek to eat the brains of all who challenge the group think.  They stifle creativity.  They have a “not invented here” mentality.  Zombie testers will continue to look for bugs using the same heuristics, and they will continue to believe that they are being effective.  Anyone that questions this status quo will be dealt with severely.  Past successes will be used to give credence to their methods; they will point to the relative stability of the group’s code base.  They will reject radical new testing paradigms like Exploratory Testing in favor of set, predetermined test cases.  Using their super lethargic strength, they will cursh all dissenters until they are assimilated as zombie testers too.  Stay clear!  Their complacency is overpowering!

• They don’t question developers, other testers, managers, authority figures, or process; they just march slowly and aimlessly in stride with everyone else.

A tester that doesn’t challenge anyone or anything displays traits of potential zombification.  Zombie testers will never go against software developers; they will assume that since the developer is an eccentric genius they must know what they are doing.  After all, complicated and over-engineered code is better; right?  When the developer makes statements like “I know what the business wants more than they do”, a zombie tester will agree that this technical person knows the business better than the business people.  Zombie testers won’t disagree with other testers that have more experience than them; after all, these senior testers have written more tests than them.  These senior testers also protect the status quo, which is great!  Zombie testers won’t disagree with their supervisors, even if their supervisor comes in with some crappie half-baked idea that he learned at a conference.  They won’t question the guru testers that have written the books.  Never mind the fact that some of those grey hair gurus haven’t tested since COBOL was en vogue.

• They place too much faith in automated testing.

To a testing zombie, automated tests are the end all be all!  Every test requires a hammer, and our automated test platform is a big ass hammer.  Mindlessly crank these things out.  It’s easy!  Just model each test on the countless examples that are already in the massive suites.  There’s something so therapeutic about not having to think while writing test scripts.  It’s like watching TV.  These zombies love to push the button on the suite and watch as all their tests come up green.  They are all green so that must mean that the tests are all valid, cover the code correctly, and actually test what we want to test.  If the tester starts to question the integrity of the testing harness or decides to manually check something, he or she is probably not a zombie.  He or she is probably a healthy tester doing their job.

• They are more proud of the pretty documentation than the actual bugs they found.

One time, Bill had an interview with a tester candidate that was truly excited about the organization and prettiness of her bug reports.  He told me that in his head, he was thinking, “NO!  I don’t care about that.  I want you to find bugs, not write reports!”  If your testers really start to glow and brag about their documentation, then they might have already turned.  If they are overly critical about the way you write up your bugs or insist that you use boiler plate sheets, then be standoffish.  They might be feasting on your brains at any moment!  Tester zombies are slow, but they can be fast to protect the status quo.  If your tester is  not excited about they bugs they found or the innovative approaches they took to finding them, be very concerned!  Note: zombie testers tend to value metrics, methods, procedures, and systems more than they value making a difference in the quality of the software.  If they constantly quote material from the ISTQB certification test, then analyze how pedantic their tone is.  The more pedantic their tone, the more they’ve turned.

Main attributes of test automation

Below are some of the attributes of test automation that can be measured:

Maintainability

• Definition : The effort needed to update the test automation suites for each new release.
• Possible measurements : The possible measurements can be e.g. the average work effort in hours to update a test suite.

Reliability

• Definition: The accuracy and repeatability of your test automation.
• Possible measurements: Number of times a test failed due to defects in the tests or in the test scripts.

Flexibility

• Definition : The ease of working with all the different kinds of automation test ware.
• Possible measurements : The time and effort needed to identify, locate, restore, combine and execute the different test automation test ware.

Efficiency

• Definition : The total cost related to the effort needed for the automation.
• Possible measurements : Monitoring over time the total cost of automated testing, i.e. resources, material, etc.

Portability

• Definition : The ability of the automated test to run on different environments.
• Possible measurements : The effort and time needed to set-up and run test automation in a new environment.

Robustness

• Definition : The effectiveness of automation on an unstable or rapidly changing system.
• Possible measurements : Number of tests failed due to unexpected events.

Usability

• Definition : The extent to which automation can be used by different types of users (Developers, non-technical people or other users etc.,)
• Possible measurements : The time needed to train users to become confident and productive with test automation.

Measurements may be quite different from project to project and one cannot know what is best unless one has clearly understood the objectives of the project.

For example, for software that is regularly changing, with frequent releases on many platforms, the important attributes will be ease of maintaining the tests and - of course - portability.

Difference between Smoke testing and Sanity Testing

Sanity Testing :

1. Sanity Testing is done after Smoke Testing is done, to Check the least functionality of the application meets the Requirements.

2. Testing the build in such a way that the basic GUI functionalities are in sane with the requirement specifications.i.e testing whether the links are working properly ,whether the UI is acceptible or not.

3. Sanity testing is a very basic check to see if all software components compile with each

     other without a problem. This is just to make sure that developers have not defined          

     conflicting or multiple functions or global variable definitions

Smoke testing: 

 
1. It is a random testing on the build for the basic functionality

2. After Receiving the Build from the Development Site, the Basic and Generally Identified Test Cases will be Tested. If these Testcases Passed without any Issues then, the Further down Testing Efforts will be Planned. If it fails then the Build is sent back to the Development Team.

But finally these two are almost same except that Smoke testing will be done by developers and Sanity testing will be done by testers

After doing installation testing, we do Smoke testing to check whether the build is read for testing. Then we do Sanity testing to check whether minimum functionality is working or not. If the minimum functionality it self is failed then at this stage only the build can be rejected to dev back.

What is Six Sigma

Six Sigma refers to a philosophy, goal, or methodology used to reduce waste and improve the quality, cost and time performance of any business. Sigma is a Greek letter used to indicate the amount of variation or defect level in a product.

A typical company today might be performing at the three sigma level, meaning they are experiencing one defect out of 16 opportunities. This would equate to about 67,000 defects per million opportunities. A better company might be at the four sigma level or one defect per 160 opportunities. Not bad, but still over 6,000 errors per million.

A performance level of six sigma is equal to 3.4 defects per million opportunities

The Testing Estimation Process

        One of the most difficult and critical activities in IT is the estimation process.  I believe that it occurs because when we say that one project will be accomplished in such time by at such cost, it must happen.  If it does not happen, several things may follow: from peers’ comments and senior management’s warnings to being fired depending on the reasons and seriousness of the failure.

Before even thinking of moving to Systems test at my organization, I always heard from the development group members that the estimations made by the Systems test group were too long and expensive.  Then, when I arrived at my new seat, I tried to understand the testing estimation process.

The testing estimation process in place was quite simple.  The inputs for the process, provided by the development team, were: the size of the development team and the number of working days needed for building a solution before starting systems tests.

The testing estimation process said that the number of testing engineers would be half of the number of development engineers and one third of the number of development working days.

A spreadsheet was created in order to find out the estimation and calculate the duration of tests and testing costs.  They are based on the following formulas:

Testing working days  =  (Development working days)  /  3.

Testing engineers  =  (Development engineers)  /  2.

Testing costs  =  Testing working days  _*  Testing engineers  _*  person daily costs.

As the process was only playing with numbers, it was not necessary to register anywhere how the estimation was obtained.

To exemplify how the process worked, if one development team said that to deliver a solution for systems testing it would need 4 engineers and 66 working days then, the systems test would need 2 engineers (half) and 21 working days (one third).  So, the solution would be ready for delivery to the customer after 87 (66+21) working days.

Just to be clear, in testing time, it was not included the time for developing the testcases and preparing the testing environment.  Normally, it would need an extra 10 days for the testing team.

The new testing estimation process

Besides being simple, that process worked fine for different projects and years.  But, I was not happy with this approach and my officemates from the development group were not, either.  Metrics, project analogies, expertise, requirements, nothing were being used to support the estimation process.

I mentioned my thoughts to the testing group.  We could not stand the estimation process for very long.  I, myself, was not convinced to support it any more.  Then, some rules were implemented in order to establish a new process.

Those rules are being shared below.  I know that they are not complete and it was not my intention  for estimating but, from now, I have strong arguments to discuss my estimation when someone doubts my numbers.

The Rules

1^st Rule: Estimation shall be always based on the software requirements

All estimation should be based on what would be tested, i.e., the software requirements.

Normally, the software requirements were only established by the development team without any or just a little participation from the testing team.  After the specification have been established and the project costs and duration have been estimated, the development team asks how long would take for testing the solution.  The answer should be said almost right away.

Then, the software requirements shall be read and understood by the testing team, too.  Without the testing participation, no serious estimation can be considered.

2^nd Rule: Estimation shall be based on expert judgment

Before estimating, the testing team classifies the requirements in the following categories:

¨      Critical:  The development team has little knowledge in how to implement it;

¨      High:  The development team has good knowledge in how to implement it but it is not an easy task;

¨      Normal:  The development team has good knowledge in how to implement.

The experts in each requirement should say how long it would take for testing them.  The categories would help the experts in estimating the effort for testing the requirements.

3^rd Rule: Estimation shall be based on previous projects

All estimation should be based on previous projects. If a new project has similar requirements from a previous one, the estimation is based on that project.

4^th Rule: Estimation shall be based on metrics

My organization has created an OPD, Organization Process Database, where the project metrics are recorded.  We have recorded metrics from three years ago obtained from dozens of projects.

The number of requirements is the basic information for estimating a testing project.  From it, my organization has metrics that guide us to estimate a testing project.  The table below shows the metrics used to estimate a testing project.  The team size is 01 testing engineer.

	Metric	Value
1	Number of testcases created for each requirement	4,53
2	Number of testcases developed by Working day	14,47
3	Number of testcases executed by Working day	10,20
4	Number of ARs for testcase	0,77
5	Number of ARs verified by Working day	24,64

For instance, if we have a project with 70 functional requirements and a testing team size of 2 engineers, we reach the following estimates:

Metric	Value
Number of testcases – based on metric 1	317,10
Preparation phase – based on metric 2	11 working days
Execution phase – based on metric 3	16 working days
Number of ARs – based on metric 4	244 ARs
Regression phase – based on metric 5	6 working days

The testing duration is estimated in 22 (16+6) working days.  Plus, 11 working days for preparing it.

5^th Rule: Estimation shall never forget the past

I have not sent away the past.  The testing team continues using the old process and the spreadsheet.  After the estimation is done following the new rules, the testing team estimates again using the old process in order to compare both results.

Normally, the results from the new estimate process are cheaper and faster than the old one in about 20 to 25%. If the testing team gets a different percentage, the testing team returns to the process in order to understand if something was missed.

6^th Rule: Estimation shall be recorded

All decisions should be recorded.  It is very important because if requirements change for any reason, the records would help the testing team to estimate again.  The testing team would not need to return for all steps and take the same decisions again.  Sometimes, it is an opportunity to adjust the estimation made earlier.

7^th Rule: Estimation shall be supported by tools

A new spreadsheet has been created containing metrics that help to reach the estimation quickly.  The spreadsheet calculates automatically the costs and duration for each testing phase. 

There is also a letter template that contains some sections such as: cost table, risks, and free notes to be filled out.  This letter is sent to the customer. It also shows the different options for testing that can help the customer decides which kind of test he needs.

8^th Rule: Estimation shall always be verified

Finally, All estimation should be verified. I’ve created another spreadsheet for recording the estimations.  The estimation is compared to the previous ones recorded in a spreadsheet to see if they have similar trend.  If the estimation has any deviation from the recorded ones, then a re-estimation should be made.