Guidelines: Important Decisions in Test

The following decisions should be made regarding the Test discipline's workflow:

Document the decisions in the Development Case, under the headings Disciplines, Test, Workflow.

Decide what artifacts to use and how to use each of them. The table below describes those artifacts you must have and those used in other cases. For more detailed information on how to tailor each artifact, and a discussion of the advantages and disadvantages of that specific artifact, read the section titled "Tailoring" for each artifact.

For each artifact, decide how the artifact should be used: Must have, Should have, Could have or Won't have. For more details, see Guidelines: Classifying Artifacts.

Artifact	Brief Tailoring Comments (see the artifacts for details)
Test Case	Must have. The question is to what extent are you going to create them.
Test Classes in the Design Model	Should have. Use this, if necessary, when you develop the test components.
Test Components in the Implementation Model	Must have. You will always develop components for testing.
Test Model	Should have.
Test Packages in the Design Model	Should have. Use this if you need to organize the test classes.
Test Plan	Must have. You need a test plan, unless the testing is trivial.
Test Procedure	Must have. This artifact is the actual test—the question here is not if, but how and what kind of structure (format and detail) you are going to use.
Test Results	Must have. This artifact is the raw data captured during the execution of test and used to evaluate the test and to calculate the different key measures of test. Considerations include what type of data needs to be captured (based upon the key measures to be reported) and what tools are required to capture the data.
Test Script	Must have for automated testing. Could have. Based upon your structuring of the test procedures, determine if the test scripts will implement and execute entire test cases or use cases, or if they need to be combined with others to completely implement and execute a test cases or use cases.
Test Subsystems in the Implementation Model	Should have. Use this if you need to organize the components.
Workload Analysis Document	Must have for Performance Tests. Should have if the structure (format and content) of the document is what you consider as being configurable.

Tailor each artifact by performing the steps described in the Activity: Develop Development Case, under the heading "Tailor Artifacts per Discipline".

Decide How to Review Artifacts

This section gives some guidelines to help you decide how you should review the test artifacts. For more details, see Guidelines: Review Levels.

Test cases

Test cases are created by the test team and are usually treated as Informal, meaning they are approved by someone within the test team. This is a natural decision, especially when members of the test team participated in the Requirements discipline. If the test cases are developed by testers who have not been part of the Requirements discipline, then the test cases are usually informally approved by project team members who participated in it.

If a customer wants to validate a product before it's released, some test cases could be selected as the basis for that validation. These test cases are treated as Formal-External.

Test procedures

Test procedures should be treated as Informal artifacts—approved by someone within the test team and under configuration management control.

When test automation or external review are required, they are treated as Formal-Internal or Formal-External.

Test scripts

Test scripts are usually treated as Informal; that is, they are approved by someone within the test team.

If the test scripts are to be used by many testers, and shared or reused for many different tests, they should be treated as Formal-Internal.

Test artifacts in design and implementation

In the design model there are two test artifacts, test classes and test packages. In the implementation model there are two test artifacts, test subsystems and test components.

These artifacts are like design and implementation artifacts, however, they're created for the purpose of testing. The natural place to keep them is with the design and implementation artifacts. Remember to label them in such a way that they are clearly separated from the design and implementation of the system.

Defects

Defects are usually treated as Informal and are usually handled in a defect-tracking system. On a small project, you can manage the defects as a simple list, for example, using your favorite e-mail system. This is only manageable for small systems—when the amount of defects grows, you need to start using a defect-tracking system.

Another decision to make is whether you need to separate the handling of defects, also known as symptoms, from faults, which are the actual errors. In small systems, you may manage to track only the defects and implicitly handle the faults. However, as the system grows, you must separate the management of defects from faults. For example, several defects may be caused by the same fault. Therefore, if a fault is fixed, it's necessary to find the reported defects and inform those users who submitted the defects, which is only possible if defects and faults are managed separately.

Test plan

In any project where the testing is non-trivial, you need a test plan. In many cases, it's treated as Informal; that is, it's not reviewed and approved. If testing is important, it could be treated as Formal-Internal or even Formal-External.

Decide on Test Approval Criteria

You must decide who is responsible for determining if an iteration has met its test criteria. This individual, or group, then decides if the iteration has met its test criteria, if all tests were completed successfully, and if the system fulfills the stipulated criteria.

This is an important decision—you cannot reach a goal if you don't know what it is. It's important to note that for early iterations it may not be necessary for the tests pass; only that they have been identified, developed, and run.