Tool Mentor: Executing Tests Using Rational Robot

Tool Mentors > Rational Robot Tool Mentors > Executing Tests Using Rational Robot

Purpose

This tool mentor describes how to use Rational Robot to execute testing (playing back the test scripts), and to analyze the results.

Related Rational Unified Process™ information:

Activity: Execute Test, refer to the section titled "Execute Test Procedures"
Activity: Perform Unit Tests

Overview

This tool mentor is applicable when running Microsoft Windows 95/98/2000/NT 4.0.

To execute the test scripts recorded or programmed using Rational Robot, play back the test script.

Play the script back
View the results in the TestManager log
Analyze verification point results with the Comparators

1. Play the script back

When you play back a script, Rational Robot repeats your recorded actions and automates the execution of the test. With automation, you can test each new build of your application faster and more thoroughly than by testing it manually. This decreases testing time and increases both coverage and overall consistency.

Test scripts are played back at several times during the test life cycle:

Test Implementation — Play back the scripts to verify that they work as intended, using the same build of the application-under-test that you used to record. This verifies the baseline of expected behavior for the application-under-test.
Test Execution and Regression — Play back the scripts to compare the latest build of the application-under-test to the baseline established during test implementation. Ongoing play back of all test scripts during the development cycle reveals any differences that may have been introduced into the application since the last build. These differences can be evaluated to determine if they are actual defects or deliberate changes.

Before playback, Rational Robot automatically compiles a script if it has changed since it last ran.

For information on preparing the test environment for script playback, see Tool Mentor: Setting up the Test Environment in Rational Robot.

To play back a script:

Click the Playback Script button on the toolbar. The Playback dialog box will appear.
Type a script name or select it from the list.

To change the Playback options, click GUI Options.

When finished, click OK. For information on playback options, see Tool Mentor: Setting up the Test Environment in Rational Robot.

Click OK to continue.
If the Specify Log Information dialog box appears, do the following:

Select a build from the list. To create a new build, click the Build button on the right.

Select a log folder from the list. To create a new log folder, click the Log Folder button on the right.

Accept the default log name, which is the same as the script name, or type a new name.

Click OK.

If a prompt appears asking if you want to overwrite the log, click Yes.

Note: To stop playback of a script, press the F11 key. Rational Robot recognizes the F11 key only when playing back Object-Oriented commands. The F11 key does not stop playback during low-level actions.

2. View the results in the TestManager log

After playback completes, use the TestManager log to view the playback results, including verification point failures, procedural failures, aborts, and any additional playback information.

To control the log information and display of the log, you can set options in the Log tab in the GUI Playback Options dialog box:

To update your project with the playback results, select Output playback results to log.
To display the log automatically after playback, select View log after playback.

If this is not selected, you can display the log after playback by clicking Tools > Rational Test > Rational TestManager.

If the playback of the test scripts resulted in any errors, the specific cause of the error must be identified.

Scripts which contain errors are identified in the log by the word Fail (shown in red) in the Results column, or Warning (shown in yellow). There are several classes of errors to be addressed: fatal errors, script errors, and verification point failures. Fatal errors typically require that the test environment should be investigated and the failure resolved. Script errors typically require test script maintenance. Verification point failures (during test implementation) typically require changes to the verification point parameters or indicate a difference between the desired state of the application-under-test when the script was recorded versus when it was played back. See the next section for analyzing verification points.

Additional information regarding the error condition may be viewed in the Log Event Properties dialog box. Open this box by selecting a log event and clicking View > Properties.

For each script in which a failure (or warning) occurred, review the information regarding the failure and if necessary, edit the test script (see Section 3. Edit the test script if necessary above). The test script (and Rational Robot) may be opened from the log by clicking View > Script.

After editing the script, the environment should be re-set to its appropriate initial state (see Tool Mentor: Setting Up the Test Environment in Rational Robot and the tests re-executed (Play the script back).

3. Analyze the verification point results with the Comparators

Use the Comparators to analyze verification point results. To open a Comparator from the TestManager log:

In the Log Event column, select a verification point and click View > Verification Point. The appropriate Comparator will appear. The Comparators may also be opened from Rational Robot by double-clicking a verification point in the Asset pane. However, when you open a Comparator this way, you can view only the baseline file.
Differences between the baseline and the actual results will be highlighted.
Review the differences and determine the appropriate course of action, including:
(Note: the available actions will differ based upon the type of verification point.)

Copy property to baseline — copy the current individual actual property and over-write the baseline property.

Copy all properties to baseline — copy all the actual property values and over-write all the baseline properties.

Edit individual property (menu, value, and so on) by double-clicking the property (menu, value, and so forth) and entering a different value.

Editing the properties list (to identify which properties should be captured by the verification point).

Create or modify masks.

When finished making the appropriate changes in the Comparator, click File > Save Baseline to save the changes, and click the Exit button to close the Comparator.

After editing the script or verification points, the environment should be reset to its appropriate initial state (see Tool Mentor: Setting Up the Test Environment in Rational Robot) and the tests re-executed (Play the script back).

Rational Unified Process