Wednesday, November 24, 2010

Use-Case Modeling for Requirements Specifications


Many software teams are discovering that mixing use-case modeling techniques for requirements expression along with traditional methods of documenting specific requirements within a “software requirements specification” (SRS) document provides an efficient means to record the complete set of detailed requirements for a system or application to be built.
The Rational Unified Process  provides the following definitions:
A requirement describes a condition or capability to which a system must conform; either derived directly from user needs, or stated in a contract, standard, specification, or other formally imposed document.
A use case describes a sequence of actions a system performs that yields an observable result of value to a particular actor.
In our experience, we have found that a fully specified set of use cases for a system often does a great job of stating many of the requirements for that system. But just as often there are also a significant number of requirements that do not fit well within this modeling technique.  Especially for non-functional requirements (e.g., specifications for usability, reliability, performance, maintainability, supportability), it is usually best to use the good-ol’ tried-and-true traditional method for stating requirements.
Traditionally, requirements specified in an SRS are simple declarative statements written in a text-based natural-language style (e.g., “ When the user logs in, the system displays the splash screen described in Figure x.”).  Use cases should be viewed merely as an alternative for specifying requirements; moreover they describe  complete threads, or scenarios, through the system which provide additional value in understanding the intended behavior of the system.  Use-case modeling is a hallmark of the Unified Modeling Language (UML) and the Rational Unified Process (RUP) as well as being a central feature provided by visual modeling tools, such as Rational Rose.  In most circumstances use cases should simply be applied as an added form of expression which increases understandability, as opposed to simply replacing the  specification of  requirements in the traditional fashion.
We need to be able to support the traditional requirements approach in all of our processes, specifications and tools since it may well will be needed whether or not a project decides to use the use-case methodology (e.g., in specifying requirements that do not fit easily in the use-case model, especially non-functional requirements).  It does not have to be an "either-or". Neither approach is "best" in all circumstances - although it could definitely be argued as to which approach might be "better" for a particular application.
A common concern that we hear from development teams first trying to apply use-case modeling is that they do not want to maintain redundant specifications of their requirements - one expression in use-case form and another in the traditional  SRS form.  They like use cases for working with their customers to specify the functional requirements of a system in the customers’ language, however they are often more familiar with the more traditional form of expression (and perhaps they are required by their boss,  or management or  an outside agency) to produce a Software Requirements Specification (SRS) that contains the “complete” set of detailed software requirements for the system.  A key concept of good requirements management advocates minimizing redundancy, so this is definitely a valid concern that should be addressed.
To facilitate the combination of traditional SRS methods and use cases, we have defined a simple construct called the “SRS Package”. This “package” pulls together the complete set of software requirements for a system (or portion of a system) which may be contained in a single document, multiple documents, a requirements repository (consisting of the requirements’ text, attributes and traceability), use case specifications and even the graphical use case model diagrams which describes relationships amongst the use cases and actors.  We’ve found this simple combination to be effective in most all application development projects, and it’s an easy way to kickstart your teams ability to realize the additional benefits provided by the use case technique. 
The Purpose of a Software Requirements Specification
The SRS provides an organized way to collect all software requirements surrounding a project (or perhaps at a subsystem or feature level) into one document. In these days of automation and visual modeling, we often find ourselves with several different tools for collecting these requirements. The complete set of requirements may indeed be found in several different artifacts accessible by multiple tools. For example, you might find it useful to collect some (or all) of the functional requirements of your system in use cases and you might find it handy to use a tool appropriate to the needs of defining the use-case model.  On the other hand, you might find it appropriate to collect traditionally stated natural-language requirements (such as non-functional requirements, design constraints, etc.) with a word processing tool in “supplementary specifications”. And a requirements management tool must be able to access all requirements for maintaining requirement attributes and traceability. 
Arguably, some combination of the use case and traditional specification technique should be used in all projects.  To facilitate this, we will collect the requirements for our SRS in a UML “package” construct that may include a single document, multiple documents, use case specifications and even the graphical use case model which describes relationships amongst the use cases. 
The “SRS package” controls the evolution of the system throughout the development and release phases of the project. In the Rational Unified Process, the following workers use the SRS Package:
·         The system analyst creates and maintains the Vision, the use-case model overview and supplementary specifications, which serve as input to the SRS and are the communication medium between the system analyst, the customer, and other developers.
·         The use-case specifier creates and maintains the individual use cases of the use-case model and other components of the SRS package,
·         Designers use the SRS Package as a reference when defining responsibilities, operations, and attributes on classes, and when adjusting classes to the implementation environment.
·         Implementers refer to the SRS Package for input when implementing classes.
·         The project manager refers to the SRS Package for input when planning iterations.
·         The testers use the SRS Package to verify system compliance.

From Vision to SRS
In the Requirements workflow of the Rational Unified Process, our requirements artifact structure starts with an artifact called the Vision which describes the user's or customer's view of the product to be developed, specified at the level of key user needs and features of the system.

The SRS Package is obviously related to the Vision document. Indeed, the Vision document serves as the input to the SRS. But the two artifacts serve different needs and are typically written by different authors. At this stage in the project, the focus of the project moves from the broad statement of user needs, goals and objectives, target markets and features of the system to the details of how these features are going to be implemented in the solution.
What we need now is a collection, or package, of artifacts that describes the complete external behavior of the system - i.e., an artifact that says specifically: “Here is what the system has to do to deliver those features.” That is what we refer to as the SRS Package.
A Living Artifact
The SRS Package is an active, living artifact. Indeed it plays a number of crucial roles as the developers embark upon their implementation effort: It serves as a basis of communication between all parties - i.e., between the developers themselves, and between the developers and the external groups (users and other stakeholders) with whom they must communicate. Formally or informally, it represents a contractual agreement between the various parties - if it’s not in the SRS Package, the developers shouldn’t be working on it. And if it is in the SRS Package, then they are accountable to deliver that functionality.
The Project Manager’s Reference Standard
The SRS serves as the project manager’s reference standard. The project manager is unlikely to have the time, energy, or skills to read the code being generated by the developers and compare that directly to the Vision document. He or she must use the SRS as the standard reference for discussions with the project team.
As noted earlier, it serves as input to the design and implementation groups. Depending on how the project is organized, the implementers may have been involved in the earlier problem-solving and feature-definition activities that ultimately produced the Vision document. But it’s the SRS they need to focus on for deciding what their code must do. It serves as input to the software testing and QA groups. These groups should also have been involved in some of the earlier discussions, and it’s obviously helpful for them to have a good understanding of the “vision” laid out in the Vision documents. But their charter is to create test cases and QA procedures to ensure that the developed system does indeed fulfill the requirements laid out in the SRS. The SRS also serves as the standard reference for their planning and testing activities.
Sample Templates for an SRS Package
In support of these concepts, two separate SRS document templates have been developed and are enclosed:
·         An SRS template for use with projects using use-case modeling, and
·         An SRS template for projects using only traditional requirements specification techniques.
See the Rational Unified Process (v5.5) for more information on these and other artifacts, along with best practices for developing software with UML.

Methodology for an effective regression testing


1. What is regression testing?

Regression testing is selective retesting of the system with an objective to ensure the bug fixes work and those bug fixes have not caused any un-intended effects in the system.

2. Types of regression testing

There are two types of regression testing that are proposed here even though it is not being practiced or popular.

 A "final regression testing" is being done to validate the gold master builds and "Regression testing" being done to validate the product & failed test cases between system test cycles.

The final regression test cycle is conducted on an "unchanged build for a period of x days" or for a period, which was agreed as the "cook-time" for release. The product is continuously exercised for the complete duration of this cook-time. Some of the test cases are even repeated to find out whether there are failures in the final product that will reach the customer. All the bug fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing which ensures "the same build of the product that was tested reaches the customer".

A normal regression testing can use the builds for a period that is exactly needed for the test cases to be executed. However unchanged build is highly recommended for each cycle of regression testing.


3. How to select test cases for regression testing?

It was found that some of the defects reported by customers in the past were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is really an art and not that easy.

  1. Requires knowledge on the bug fixes and how it affect the system
  2. Includes the area of frequent defects
  3. Includes the area which has undergone many/recent code changes
  4. Includes the area which is highly visible to the users
  5. Includes the core features of the product which are mandatory requirements of the customer

Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an Extreme defect can have no or a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.

When selecting the test cases we should not select more test cases, which are bound to fail and has no or less relevance to the bug fixes. You need to select more positive test cases than negative test cases for final regression test cycle as this may create some confusion and unexpected heat. It is also recommended that the regular test cycles before regression testing should have right mix of both positive and negative test cases. Negative test cases here I mean those test cases which are introduced newly with an intent to break the system.

It is noticed that several companies have "constant test cases set" for regression testing and they are executed irrespective of the number and type of bug fixes. Sometimes this approach may not find all side effects in the system and in sometimes it may be observed that the effort spend on executing test cases for regression testing can be minimized if some analysis is done to find out what test cases are relevant and what are not.

It is a good approach to plan and act for regression testing from the beginning of project before the test cycles. One of the ideas is to classify the test cases into various Priorities based on importance and customer usage. Here I am suggesting the test cases be classified into three categories;

  • Priority-0 – Sanity test cases which checks basic functionality and are run for pre-system acceptance and when product goes thru major change. These test cases deliver a very high project value to both engineering dept and to customers.

  • Priority-1 – Uses the basic and normal setup and these test cases deliver high project value to both engineering and to customers.

  • Priority-2 – These test cases deliver moderate project value. Executed part of ST cycle and selected for regression testing on need basis.

There could be several right approaches to regression testing which needs to be decided on "case to case" basis;

  • Case 1: If the criticality and impact of the bug fixes are LOW, then it is enough a test engineer selects a few test cases from TCDB and executes them. These test cases can fall under any Priority (0, 1 or 2).

  • Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.

  • Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.

  • Case 4: One can also go thru the complete log of changes happened (can be obtained from CM engineer) because of bug fixes and select the test cases to conduct regression testing. This is an elaborate process but can give very good results.

4. Resetting the test cases for regression testing

In a big product release involving several rounds of testing, it is very important to note down what test cases were executed with what build and related information. This is called test case result history. In many organizations not all types of testing and all test cases were repeated for each cycle. In such cases resetting the test cases become very critical for the success of regression testing. Resetting a test case is nothing but setting a flag called NOTRUN or EXECUTE AGAIN with zero base thinking.

RESET of test case, are not expected to be done often. Resetting of the test cases needs to be done with following considerations;
a.     When there is a major change in the product
b.     When there is a change in the build procedure which affect the product
c.     Large release cycle where some test cases were not executed for a long time
d.     You are in the final regression test cycle with a few selected test cases
e.     Where there is a situation the expected results of the test cases could be quite different from previous cycles

When the above guidelines are not met, you may want to RERUN the test cases rather than resetting the results of the test cases. There are only few differences between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think zero base and expect different result than what was obtained in earlier cycles and therefore those test cases affect the completion rate of testing. In case of RERUN the management need not worry about completion rate as those test cases can be considered complete except for a formality check and are expected to give same results.

To give you an example, if there is a change in Installation of a product, which does not affect the product functionality, then the change can be tested independently by rerunning some test cases and we don't have to RESET the test cases.

RESET is also decided based on how stable the functionalities are. If you are in Priority-1 and have reached a stage of comfort level on Priority-0 (say for example more than 95% pass rate) then you don't RESET Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in Priority-2 test phase.

4.1 Pre-system test cycle phase

For pre-system acceptance only Priority-0 test cases are used. For each build that is entering the system test, the build number is selected and all test cases in Priority-0 are reset to NOT RUN. The system test cycle starts only if all pre-system test cases (Priority-0) pass. Test manager or CCB, can decide exceptions if any.

4.2 System test cycle – Priority-1 testing phase

After pre-system acceptance is over, Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are high. A RESET procedure during this phase may affect all Priority-0 and Priority-1 test cases and these test cases are reset to NOTRUN in TCDB.

4.3 System test cycle – Priority-2 testing phase

Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass % as defined in test plan. In this phase several builds are used. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are very high. A RESET procedure during this phase may affect Priority-0, Priority-1 and Priority-2 test cases. 

4.4 In what way regression testing is related to above three phases?

Regression testing is normally done after Priority-2 testing or for the next release involving only few changes. Resetting test cases during the above phases are not called as regression testing as in my assumption regression comes into picture only after the product is stable. A testing for a release can be decided either by saying a regression testing is sufficient or we can do all phases of testing starting from Priority-0 to Priority-2.

A regression testing for a release can use test cases from all priorities (as mentioned before). A regression testing involving multiple priorities of test cases also needs the test cases executed in strict order i.e. Priority-0 test cases are executed first, Priority-1 next and Priority-2 test cases.

4.5 Why we need to RESET the test cases?

Regression testing uses good number of test cases, which would have been executed already and associated with some results and assumptions on the result. A RESET procedure makes them to NOTRUN so that it gives a clear picture about how much of testing is still remaining, and reflect the results of the regression testing on Zero base.

If test cases are not RESET, then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds can be used in each phase of the testing and a gut-feel that if something passed in the past builds, it will pass in future builds also. Regression testing doesn't go with an assumption that "Future is an extension of the past".

5. How to conclude the results of a regression testing?

Regression testing uses only one build for testing(if not it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;
a.     If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.
b.     If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.
c.     If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn't be considered for the pass %. This may also mean that such test cases shouldn't be selected for regression.
d.     If the result of a particular test case is FAIL using the previous builds but works with a documented workaround and
a.     if you are satisfied with the workaround then it should considered as PASS for both system test cycle and regression test cycle
b.     If you are not satisfied with the workaround then it should be considered as FAIL for a system test cycle but can be considered as PASS for regression test cycle.

6. Can we apply the regression test guidelines for patch/upgrade releases?.

The regression guidelines are applicable for both cases where
a.     You are doing a major release of a product, executed all system test cycles and planning a regression test cycle for bug fixes.
b.     You are doing a minor release of a product (CSPs, patches …etc) having only bug fixes, and you are planning for regression test cycles to take care of those bug fixes.

There can be multiple cycles of regression testing that can be planned for each release, if bug fixes come in phases or to take care of some bug fixes not working with specific build.

7. How do I find out which test case to be executed for a particular defect fix?

When failing a test case it is a good practice to enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.

Quality Assurance & Testing ”Some Myths”


Quality Assurance is comparatively new phenomenon as to testing. I would like here to be specific while defining quality, its assurance and differences from traditional testing methodologies.

Glance at Software Industry and Quality artifacts:

Unlike the industrial revolution in “West” software revolution didn’t take centuries to sow the seed and then boom with a vertical graph, instead it took the software industry less almost half a century to evolve, flourish, and experience the crests and troughs. Quite rightly this is the offspring of baby boomers generation, post world war scenarios, super powers’ marathon, and industrial revolution.

Although the software industry learned fast but still the path it followed was same as was taken by its predecessor industries. While saying that I mean look at the traditional industry, it took them centuries to understand and digest the need of quality product and satisfied customer. Similarly software industry started without having any knowledge of the customer’s exact needs, without any framework, without any diversified targets.

I think many of the readers would agree with me if I say that software industry had the sole aim of superiority of weapons, in its infant days. Fact of the mater is the today’s software industry owes a lot to the “Department of Defence” (DoD) and its subsidiaries. All the buzzwords we utter today like CMM, Malcom Baldrige (MBNQA), and SPICE are the product and requirement of DoD.

To stream line the process of its software manufacturing, and outsourcing, DoD came up with these different sorts of parameters, standards, and tools to measure where the organization stands which intend to do business with DoD by providing the solution towards the software needs of DoD.

But still do you know what is quality? How and why would different people and organizations have different set of processes and standards? If they all don’t have the same framework then who is performing the actual quality work and how can we categorize them.

To deduce the actual trend and definition of quality lets discuss a few examples. For NASA software’s reliability, and performance matters most. For Mr. X software is good enough if it looks good, GUIs are aesthetic; performance and reliability are secondary issues for him in the initial stage. Hence for two different customers primary and secondary issues swap. Same is the case with the quality of these soft wares, if NASA’s software is not very aesthetic in look and feel but it performs excellently with accuracy and is 100% reliable they would say it is of high quality whereas if Mr. X is made to use a very reliable but difficult to use software with crude interfaces, he would definitely going to resist. For Mr. X the quality parameters are not the same as for NASA.

So we conclude that quality is a relative term and it is a variable, which keeps on changing. That’s why I keep on saying “Quality lies in the eyes of beholder”.

Now if we try to define a few major quality artifacts we would have a checklist that would look some thing like this:

Quality software must

ü  Meet the requirements of user
ü  Have sufficient and up to date documentation.
ü  Be cost effective.
ü  Be user friendly
ü  Easy to learn
ü  Reliable
ü  Manageable
ü  Maintainable
ü  Be Secure

But REMEMBER!

“To every quality artifact there is an equal and opposite resisting force”