<- previous index next ->
There are many types of software testing
Black Box testing:
Treat the software as a "black box".
Supply inputs, check outputs.
Test every item in the requirements specification.
White Box testing:
You have access to the source code.
To get 100% path coverage, you will probably
need automation. See "STEST" below.
Additional test may be needed to be sure
every data structure is used in all the
ways they may be used.
Unit Testing:
Test ever file as it is coded.
A) Test as expected in normal use
B) Stress test with bad input and overloaded input.
C) Exercise every path and data structure
D) Expect one or more errors during integration
Integration Testing:
When doing a bottom up build, not the best development method,
combine existing files that call or are called by the new
file, module, sub system.
When doing top down build, integrate each file, module,
subsystem as it is available.
Do not trust that Unit Testing has all the defects removed.
System Testing:
Provide real world data in a simulated environment
that would be expected when in production use.
Rerun previous tests that are still applicable.
Acceptance Testing:
Software Quality Assurance review of test procedures
and test report.
You may have a customer who has been involved.
Possibly release a Alpha Test version.
Possible release a Beta Test version.
Automation helps reduce testing cost and improves reliability.
STEST SQA tool for metrics and testing
This program computes quality metrics in terms of structured testing
statistics. It computes the cyclomatic complexity for a total program. It
also produces the total count, mean, standard deviation, minimum count,
maximum count, and name of the routine with the maximum count for many
program characteristics.
In addition the program can produce an instrumented version of
a program. The instrumented version can be run with one or more sets of
test data. Either statement coverage or path coverage can be selected.
Cumulative and incremental coverage statistics are printed.
This program is valuable during the software testing process.
It is a useful tool for evaluating the completeness of a test suite. The
output can be used to locate untested areas of a program and thus aids
in generating good test data.
STEST DIRECTIVES
The input is a collection of directive lines and file-specs.
Directive lines begin with a semicolon (;) in column 1 followed by the
command, with only the first four characters of the command being used.
If no semicolon is found in column 1, the line is assumed to be a file-
spec. A file-spec may contain a directory name, a list of
files, or any combination of directory and file names. To get more
information on a specific directive, key / followed by the directive
name.
/FORTRAN
The ;FORTRAN directive specifies the language is Fortran.
Format: ;FORTRAN
/JOVIAL
The ;JOVIAL directive specifies the language is Jovial J73/I.
Format: ;JOVIAL
/J73
The ;J73 directive specifies the language is Jovial J73,
MIL-STD-1589B. ( under development )
Format: ;J73
/1750A
The ;1750A directive specifies the language is MIL-STD-1750A
assembly language. ( under development )
Format: ;1750A
/ADA
The ;ADA directive specifies the language is ADA MIL-STD-1815A
Format: ;ADA ( under development )
/AFOR
The ;AFOR directive specifies the language is FORTRAN
for the MIL-STD-1750A target (embedded computer version)
Format: ;AFOR ( under development )
/TITLE
The ;TITLE directive allows the user to insert a title(up
to 60 characters). The default is to leave the title blank.
Format: ;TITLE
/PRDUMP
The ;PRDUMP directive directs the program to generate a
listing of predicates (e.g. .GE.). The default is '0' which says
'do not list predicates'. In order to get the listing of predicates,
a '1' must be used on the directive card.
Format: ;PRDUMP 1
/END
When the ;END directive is encountered, the program processes
the current set of source files and looks ahead for additional source
files.
When an end-of-file is encountered, the STEST program outputs
the listing of structured testing statistics. Because of this feature,
more than one language may be processed in one run.
Format: ;END
/SCOVER
The ;SCOVER directive commands the generation of a unique
version of each source file, instrumented to record the number of
times each statement has been executed. The default is '0' which says
'do not produce instrumented version of source files'. In order to get
the instrumented version, a '1' must be used on the directive line.
The conditions for using this directive are:
1. The language MUST be Fortran.
2. A ;DIR directive MUST be used.
3. The ;PCOVER directive MUST NOT be used simultaneously.
Format: ;SCOVER 1
/PCOVER
The ;PCOVER directive commands the generation of a unique
version of each source file, instrumented to record the number of
times each path has been executed. The default is '0' which says
'do not produce instrumented version of source files'. In order to get
the instrumented version, a '1' must be used on the directive line.
The conditions for using this directive are:
1. The language MUST be Fortran.( Ada and AFOR soon)
2. A ;DIR directive MUST be used.
3. The ;SCOVER directive MUST NOT be used simultaneously.
Format: ;PCOVER 1
/DIR
The ;DIR directive allows the user to specify a directory name
(up to 40 characters) where the instrumented version of source files
are to be placed. Since the instrumented source files have the same
names as the original source files, it is advisable to have the
instrumented versions in a different directory.
This directive is used only in conjunction with either the
;SCOVER or ;PCOVER directive.
Format: ;DIR .
/UPDATE
The ;UPDATE directive allows the user to re-instrument one or
more routines of a previously instrumented program. This directive is
used when a change has been made to one or more routines in a program
and the user wants to re-instrument the modified routines without
re-processing the entire program.
The conditions for using this directive are:
1. The language MUST be Fortran.
2. Either the ;PCOVER or ;SCOVER directive (whichever is
applicable) MUST be used.
3. The ;DIR directive MUST be used.
4. The directory name MUST be the same as the one used when
the program was originally instrumented.
Format: ;UPDATE 1
STEST INSTRUMENTING
The following steps are required to create an instrumented version
of a program:
1. Create a directory for holding the instrumented source files.
2. Run STEST.
3. Get into instrumented directory as default.
4. Execute ZZCOMP command file to compile instrumented routines.
5. Get a copy of CAS$LIB:ZZZPRE.OBJ. (This file must be linked with the
instrumented object files.)
6. If path coverage is being measured, get a copy of
CAS$LIB:ZZPPOS.OBJ.
If statement coverage is being measured, get a copy of
CAS$LIB:ZZSPOS.OBJ.
(The appropriate file must be linked with the instrumented object
files.)
7. Link instrumented object files.
8. Variations on the above for Ada and AFOR COMING SOON
STEST INSTRUMENTING FILES
When STEST generates an instrumented version of a program, several
new files are created in the directory specified by the ;DIR directive.
An instrumented version of each source file is produced. These files have
the same names as the original source files.
ZZCOMP.COM is the command file generated to compile the instrumented
routines.
An instrumentation data table file, ZZZZZZ.DAT, is also produced.
It is used by the coverage statistics reporting routine.
STEST INSTRUMENTING STATISTICS
The instrumented version of the program can be run with one or
more sets of test data. Upon completion of each test run, the coverage
statistics are printed.
STEST INSTRUMENTING STATISTICS PATH_COVERAGE
The statistics consist of three parts. The first part is a table
of untested paths. The second part of the output (printed for test runs
subsequent to the first run) is a table of paths executed during the
current run only. These two tables have the same format. Column one
lists the routine name and column two lists the line number of the path.
Clarifying information is printed after the line number and varies
depending upon the statement type.
Two special cases concerning line numbers should be noted. The
path following an entry statement has been assigned a line number of -1.
The first path in a routine has been assigned a line number of 0.
The final part of the statistics is a table showing the
percentage of paths tested. Column one lists all the routines and column
two lists the percentage of paths tested.
See example to get more information on the output.
STEST INSTRUMENTING STATISTICS PATH_COVERAGE EXAMPLE
PATH COVERAGE METRICS
ROUTINE PATH NOT EXECUTED
ACORE COMPLETELY TESTED
ASGFIL 50 TRUE
ASGFIL 66 TRUE <-- means true path not
taken
BSSWRL EVERY PATH NOT EXECUTED
TOTAL NUMBER OF PATHS NOT EXECUTED: 29
ROUTINE PERCENTAGE OF PATHS TESTED
ACORE 100.00
ASGFIL 0.70
BSSWRL 0.00
STEST INSTRUMENTING STATISTICS STATEMENT_COVERAGE
The statistics consist of three parts. The first part is a table
of untested statements. The second part of the output (printed for
test runs subsequent to the first run) is a table of statements
executed during the current run only. These two tables have the same
format. Column one lists the routine name and column two lists the
line number of the executable statement. In the case of a 'logical IF'
statement, the word 'true' or 'false' is printed after the line number
to indicate the value of the logical expression.
The final part of the statistics is a table showing the
percentage of statements tested. Column one lists all the routines and
column two lists the percentage of statements tested.
STEST INSTRUMENTING STATISTICS REPORTS
Cumulative coverage statistics may be recovered and printed out
by performing the following steps.
1. Get into instrumented directory as default.
2. For path coverage statistics, use the following format:
@CAS$:PGENER output (where 'output' is where the listing is to go)
For statement coverage statistics, use the following format:
@CAS$:SGENER output (where 'output' is where the listing is to go)
STEST USAGE (Run stream on DEC VAX 11/780)
@CAS$:STEST
;FORTRAN
;PRDUMP 1
;PCOVER 1
;DIR [Directory]
[Directory] Process all files of type .FOR
;END
;END
(The set of directive cards will vary depending upon user needs.)
The alternate format is:
@CAS$:STEST file output
where 'file' contains all the input directives required and
'output' is where the output is to go.
STEST OUTPUT
The output of the STEST program is a printout consisting of:
1. Listing of user commands.
2. Table of structured testing statistics. (The total count,
mean, standard deviation, minimum count, maximum count,
and name of the routine with the maximum count are printed
for many program characteristics.)
and optionally:
3. Listing of program predicates.
last updated for DEC VAX VMS before December 1997
<- previous index next ->