PD-0120: Parameter Validation Testing

This decision represents a long-term technical decision based on an OD, and may not be the same as the final results of the source OD. With respect to published criteria documentation and scheme documents, it provides suggested guidance on evaluation direction, but is not authoritative. Authoritative decisions are provided through the published criteria documents and published scheme and international interpretations thereof. With respect to published PPs, PDs are authoritative corrections to the PP, based on input from the PP author (if available), that are in force until the publication of the next revision of that PP.

Issue

It is unclear whether, at EAL4, the CEM requires that parameter validation testing be conducted for all evaluations. Such testing would include testing interfaces for:

1. Impermissible values

2. Bad parameters

3. Strings too long

4. Numeric values too large

5. Incorrect syntax

If such testing is required:

1. What is the derivation of the requirement (both in terms of the CC Part 3 and the CEM)? Does such testing fall under the normal testing requirements (i.e., ATE)? Or is it viewed as part of vulnerability testing (AVA)?

2. What is the breadth of such testing? I.e., does it need to be applied to all interfaces, or only to certain interfaces? If the latter, how is the set determined: is it a random sample; or is it a set based on some form of analysis, such as implementation subset analysis?

3. What is the approach to such testing? Is actual testing required, or can it be done through alternative analysis approaches?

Overarching all of these questions is the issue of security-relevance. Does "normal testing" include security-relevant tests? Do the requirements apply only to security-relevant interfaces? Should penetration testing be confined only to security-relevant inputs?

Resolution

This issue is complicated by the fact that the CC and methodology separate the functional specification, functional testing, and vulnerability analysis requirements and treat them as if they were completely isolated and unrelated, when, in reality, they are inter-dependent.

Basically: Developer testing can be expected (but is not required) to include a lot of parameter validation testing. However, the penetration testing created by the developer and the evaluator is expected to include testing of parameters that may have vulnerabilities. If the vulnerability analysis identifies security relevant problems in the TOE interfaces, then penetration testing of those interfaces should occur.

With respect to the specific Common Criteria components:

1. ADV_FSP.2: The functional specification

   The functional specification describes the security behavior of the TSFI, in terms of the interfaces presented. These interfaces are the means to invoke service from the TSF, and the responses (including error responses) to such service requests can reveal information.

   Interfaces may have variations on invocation of the interface, called aspects. For example, a UNIX command line interface might have flags that can be set, parameters that may be defined, etc., all of which are considered aspects. Interfaces to the TSF are either security-relevant or non-security-relevant. Security-relevant interfaces have security-relevant aspects and may also have non-security-relevant aspects.

   PD-0077 defines the necessary contents of the functional specification for ADV_FSP.2. It states that there may be some (non-security-relevant) interfaces that need not be specified to the same detail as other (security-relevant) interfaces, but that all interfaces and their effects must be included. The only exception to this is indirect error messages, which need not be documented.

2. ATE_COV.2: Security Functional testing by the developer.

   The test coverage requirements are meant to ensure that the developer has adequately tested the interfaces as described in the functional specification. When ATE_COV.2 is claimed, the developer is required to test the security-relevant aspects of the security-relevant interfaces; the evaluator ensures these have been tested. The developer must supply explanations of why untested aspects of the security-relevant interfaces were found to be non-security-relevant. It is worth noting that, in many cases, it may be easier (i.e. more direct, less work) for the developer simply to test all aspects than to provide the necessary justification for each untested aspect.

   While the developer tests the security-relevant aspects of the security-relevant interfaces under ATE_COV.2, the evaluator's task is to rerun a subset of these tests as part of ATE_IND.2 (using specifically, the methodology for ATE_IND.2-9 and ATE_IND.2-10). As mentioned in the Sampling annex (B.2) of the methodology, if the soundness and accuracy of the tests reveal consistency, then the sample size to be rerun can be smaller than if there is no consistency seen as testing progresses.

3. AVA_VLA.2: Penetration Testing.

   While functional testing is the verification that the TSF performs the advertised security-relevant behavior as specified; penetration testing explores the TSF's unadvertised behavior. This typically results from inputs outside the specification. Such inputs, in making use of the non-security-relevant aspects of the security-relevant interfaces, and those inputs to any non-security-relevant interfaces, would be covered by penetration testing.

Support

The CC specifies functional testing in the ATE family and penetration testing in the VLA family. The former is performed by the vendor and the latter by the evaluator. In regard to the specific requirements:

• ATE_COV.2-2 only deals with expected behavior.

• ATE_COV.2-4 covers the correspondence between the function specification and the tests.

• ATE_IND.2-4 (including paragraphs 1549, 1550.1.1, 1551.a and c) defines a subset of the developer's tests.

• AVA_VLA is specific as to the checking the evaluator performs on the developer's vulnerability analysis. This includes what should be a large amount of cross-checking. Furthermore, the family requires the evaluator to create a test suite built upon the developer's suite in the construction of which the evaluator has to attempt to include vulnerabilities dismissed by the developer and others not considered at all by the developer. The CEM includes much more guidance as to what the evaluator has to look for and where to look. All tests created by the evaluator have to be described in the ETR for that work unit. For a product with many capabilities one would expect this to be an extensive effort. The validator should expect to see a large amount of justification as to the disposition of the various postulated vulnerabilities; as well as the actual tests constructed.

This issue grows out of the question of the differences between functional testing and vulnerability testing. The confusion arises when the distinction between the two is blurred. Functional testing attempts to verify that the TSF performs the advertised behavior as specified (see the discussion in PD-0077); whereas penetration testing explores the TSF's (unadvertised) behavior that results from inputs outside the specification.

For example, the command-line syntax (or interface) for the Unix "kill" command is:

kill [ -s signal | -p ] [ -a ] pid ... 
	kill -l [ signal ]

The full functional specification captured in the man page (not included here for space) includes a description of what each switch does, the potential values of switch arguments, the output of the command, and an indication of any error messages or codes returned by the command.

The ATE testing of this command would be guided by the SFRs of the ST, and would be expected to include an attempt to use each of the documented flags that were determined to be security relevant (for example, -s if processes were both subjects and objects of the TOE's security policy). Use of valid and invalid values for "signal" and "pid" would be tested solely to exercise any security relevant error testing highlighted in the functional specification. The goal of this testing would be to verify that the security-relevant aspects of the command work as expected. Exercising this command in any way not advertised (attempting a -z flag, for example, or using invalid parameter values if no error messages were listed) would be considered vulnerability testing. Bear in mind that both kinds of testing are to be performed; however the CC and CEM require us to distinguish them.

At some point during vulnerability testing the objectives for the TOE will have to be considered. While finding unexpected and insecure behavior is the point of the testing (and even fun in some sense) the imperfections so exposed must contradict at least one security objective. That is, not all flaws found are fodder for the failure file. To be written-up as vulnerabilities they must represent a threat that can succeed against an objective.

There are a number of different types of testing that the developers of major software products perform. They include configuration, installation, functional, and performance testing. Functional testing includes security and regression testing. Security tests are those tests that verify that the product's security policy is being enforced. Regression tests are tests added to the test suites to verify that updates to the product intended to fix problems found therein actually fix those problems and have no other unintended side effects. Note that regression test suites can and should have additions made to them on a continual basis as changes are made to the product.

When a software developer embarks on an evaluation attempt, they will review their functional test suite to determine how well it performs security testing. They most likely will find they have to create a whole new test suite or greatly enhance the existing security tests because the test suite they have is lacking in coverage and depth. During the evaluation process they will present the new suite to satisfy the ATE_COV, ATE_DPT, and ATE_FUN requirements. It is then the evaluator's task to determine whether the new test suite is sufficient according to those requirements.

If the developer has regression tests that involve security policy or security functions, one might think that they would be included in the security test suite submitted to their evaluation lab. Yet as long as what one might call the "base security test suite" meets the CC requirements, there is no requirement for the developer to do so.

While the developer's regression tests wouldn't be used themselves to satisfy the vulnerability analysis requirements, the analysis that led to them could be. However that effort may be considered to be "in the past," and a new search for vulnerabilities should to be made during the time-span of the evaluation.

ATE_FUN.1 states "The objective is for the developer to demonstrate that all security functions perform as specified." The key phrase is "as specified". The evaluators have access to the developer's functional specifications. The ATE_FUN requirements demand that the evaluators ascertain that the developer has developed and executes tests verifying that the functional spec has been implemented. In all probability, the developer's functional specification does not go into very much detail concerning all possible valid inputs for the security functions. In this case, the evaluators will have to decide whether or not the developer's test suite performs adequate testing of input parameters. The validators will review the evaluator's decision and the evaluators will be well advised to show a good correspondence between any input parameter specifications and the tests in the test suite(s). If there is relatively little in the functional specification, the validators would have to see next to no testing in order to judge the requirements as failures. The validators may suggest more parameter checking, but their only recourse will be to make mention of a lack of security function input protection in the Validator's Report.

As to the rest of the ATE families, the ATE_COV requirements are concerned with coverage of the various functions, not coverage of all the possible inputs. The ATE_DPT requirements are based upon the various design documents that hardly ever concern themselves with valid input. For the ATE_IND requirements, the evaluators are restricted to the developer's test suite. There is nothing there in the ATE_IND components allowing the evaluator to criticize the number and variety of tests provided.

While there may be relatively little leverage for the evaluator and validator in the ATE families, the AVA_VLA family refers to commonly available external sources. Especially if the developer has created patches for currently published vulnerabilities, then the evaluator and validator should insist upon seeing testing that includes all the possible functions that may have that vulnerability. The developer can be expected to present arguments based on the code tree that explain why at least some existing tests apply to more than one function. The analysis required is done by the developer, though the guidance is "to confirm that no identified security vulnerabilities can be exploited in the intended environment for the TOE and that the TOE is resistant to obvious penetration attacks." One would expect the evaluator and validator to check this thoroughly.

Modification History:

2005-08-23

PD Created. (August 2005 ODRB Agenda Item 3.a.i)

2005-11-07

Updated PD in response to comments received by the ODRB (November 2005 ODRB Agenda Item 4.c.i)

References:

• CEM v2.2

Related NIs:

• None

Related CCIMB-INTERPs:

• None

Source OD: 0244