I-0008: Testing Is Not Expected To Find All Flaws

NUMBER:               I-0008
STATUS:               Reposted for External Review

TITLE:                Testing Is Not Expected To Find All Flaws

FIRST POST:            [criteria 2344]
MOST RECENT REPOST:    [criteria 2466]

REQUIREMENT:          Security Testing
CRITERIA CLASSES:     C1, C2, B1, B2, B3, A1
DOCUMENT(S):          <None>
RELATED TO:
     I-0336           Security Testing Must Be Comprehensive

STATEMENT:

This provides technical guidance regarding the entire security testing requirement.

Security testing is not expected to uncover all possible flaws.

PROJECTED IMPACT:

Negligible impact anticipated.

SUPPORT:

"Testing can show the presence of bugs, but not their absence" (Edsger W. Dijkstra).

All known approaches to absolute demonstrations of correctness are not only impractical, they are impossible. It is not meaningful to expect that any form of testing for a complex product or system will uncover all possible problems.

Once testing has been completed, any subsequent discovery of a flaw in an evaluated product is not considered evidence that the product is unsatisfactory or that the evaluation was incomplete. Rather, it is a recognition that testing cannot be complete. Fielded systems have the discovered flaw addressed through the RAMP process, and testing for the flaw often becomes part of the regression test for the system.