United States Election Assistance Comittee

Register to Vote!

Use the National Mail Voter Registration Form to register to vote, update your registration information with a new name or address, or register with a political party.

Note: If you wish to vote absentee and are a uniformed service member or family member or a citizen living outside the U.S., contact the Federal Voting Assistance Program to register to vote.

EAC Newsletters
and Updates

Sign up to receive information about EAC activities including public meetings, webcasts, reports and grants.

Give Us Your Feedback

Share your feedback on EAC policy proposalsElection Resource Library materials, and OpenEAC activities. Give feedback on general issues, including the Web site, through our Contact Us page.

Military and Overseas Voters

EAC has several projects under way to assist states in serving military and overseas citizens who register and vote absentee under the Uniformed and Overseas Citizens Absentee Voting Act. Learn more

Chapter 5: Test Plan

This chapter defines required content for the test plan, which is to be prepared by the test lab. It does not specify an overall organization for the test plan, nor does it enumerate all of the content that would be reasonable and customary for a test lab to include. Test labs are encouraged to apply relevant external standards, such as [IEEE97] and [IEEE98] or their logical successors, to determine the organization and content of test plans, provided that the information described in this chapter does appear in the result.

The purpose of the test plan is to document the test lab's development of the complete or partial test suites. To some extent, the test plan is determined by the Testing Requirements (Part 3). The test plan must document the test suites so that the results of testing are reproducible.

Prior to development of any test plan, the test lab must obtain the Technical Data Package (TDP) from the manufacturer submitting the voting system for conformity assessment. The TDP contains information necessary to the development of the test plan, such as the manufacturer's hardware specifications, application logic specifications, operating manual, and maintenance manual.

2 Comments

Comment by M. Santos (Voting System Test Laboratory)

The intention of the test plan, among other things, is to document the intent of the test cases and what each one will validate at an intermediate level, not a detailed level. The detailed level occurs at the test case creation phase, which occurs after the test plan phase. The intent of the test cases themselves is to detail, in a reproducible manner, what is being validated in each individual test case. Shouldn't the test cases be a deliverable of the test report?

Comment by Cem Kaner (Academic)

The test plan shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)

5.1 Test plan contents

5.1-A Test plan references

The test lab SHALL list all documents that contain material used in preparing the test plan.

Applies To: Voting system

Source: [VVSG2005] II.A.1.1

1 Comment

Comment by Cem Kaner (Academic)

All test plan references shall be public records unless they are proprietary materials supplied with COTS software embedded in the voting system. .......... (Affiliation Note: IEEE representative to TGDC)
5.1-B Test plan, implementation statement

The test lab SHALL include a copy of the implementation statement provided by the manufacturer.

Applies To: Voting system

Source: Revision of [VVSG2005] II.A.1

5.1-B.1 Test plan, clarifications to implementation statement

The test lab SHALL document any interpretations made by the test lab to fully identify the implementation under test and the scope of assessment that is desired.

Applies To: Voting system

5.1-C Test plan, inventory of materials delivered

The test lab SHALL enumerate the materials delivered by the manufacturer to the test lab to enable conformity assessment to occur.

Applies To: Voting system

DISCUSSION

Materials include hardware, software, the TDP, evidence of prior certifications, test ballots, test data, etc.

Source: [VVSG2005] II.A.3

5.1-C.1 Test plan, specificity of inventory

Materials SHALL be identified by specific version, serial number, etc., if they are versioned or numbered, and the quantity of each SHALL be noted.

Applies To: Voting system

5.1-D Test plan, previous work

The test lab SHALL document all prior certifications, reviews, tests, or other conditions that impact the test lab's determination of the scope of conformity assessment, and document said impact.

Applies To: Voting system

DISCUSSION

The test lab may recognize certifications, reviews, and tests conducted by other labs, whether they are accredited for voting system conformity assessment or not, as making some portions of the voting system test campaign redundant. For example, a COTS computer should already have been certified to comply with the Rules and Regulations of the Federal Communications Commission, Part 15, Subpart B requirements for both radiated and conducted emissions and need not be retested for that. Also, if a slightly modified system is submitted for reassessment, the test lab's finding that some or all of the test campaign need not be repeated would be documented under this requirement.

Sometimes new systems use a combination of new devices interfaced with the devices of a previously tested system. For example, a manufacturer can submit a voting system for conformity assessment that has a new DRE voting device, but that integrates the election management subsystem from a previously tested system. In this situation, the accredited test lab may design and perform a test procedure that draws on the results of testing performed previously on reused subsystems. However, irrespective of previous testing performed, the scope of testing is expected to cover:

  1. All functionality performed by new devices;
  2. All functionality performed by modified devices;
  3. Functionality that is accomplished using any interfaces to new devices, or that shares inputs or outputs from new devices;
  4. All functionality related to vote tabulation and election results reporting; and
  5. All functionality related to audit trail maintenance.

Source: [VVSG2005] II.3.2.4, II.A.2, II.B.1.2

1 Comment

Comment by Brian V. Jarvis (Local Election Official)

This section requires significant clarification because it currently leaves too much leeway and too much room for interpretation. The test lab should only recognize certifications, reviews, etc. if there is a "direct artifact" that exists proving the satisfactory results of those certifications, reviews, etc. actually exist. (A direct artifact is the tangible output resulting directly from implementation of a practice.) The example given where "a COTS computer 'should already have' been certified..." cannot be considered sufficent evidence. Additionally, the next sentence gives too much test relief in the case of a "slightly modified system." (What is 'slightly' and who objectively defines it?) If any of the system design or architecture or interfaces have changed, this should require the complete repeat (e.g., re-testing, regression testing) of the test campaign for the entire system.
5.1-E Test plan, reproducible testing

The test lab SHALL provide the information needed to reproduce the testing that it performs, including facility requirements, test set-up, test sequence, test operations procedures, data recording requirements, and pass criteria.

Applies To: Voting system

Source: Condensed from [VVSG2005] II.A.5 and 6

2 Comments

Comment by Carolyn Coggins (Voting System Test Laboratory)

Due to the exceptional number of settings, options, and configurations of a voting system, the requirement to "reproduce testing" is beyond the scope of the test plan. The level of detail required to accurately reproduce testing is appropriately identified as an element of the Test Case, not the Test Plan.

Comment by Cem Kaner (Academic)

The test plan should document a set of reproducible test cases that provide the traditional level of coverage of product requirements. However, testing should also include exploratory ("open-ended") testing of the functionality, performance, usability and other quality attributes of the system. The test documentation should provide a final report that describes the strategy of the exploratory testing, outlines the testing actually done and describes the results. However, because this testing will (and should) vary from build to build of the software, it should not be documented to the level of detail of a traditional reproducible test. .......... (Affiliation Note: IEEE representative to TGDC)
5.1-E.1 Test plan, standard test suites

For applicable tests that are specified in Part 3, the test lab SHALL document the implementation details that determine how the standard tests are realized for the implementation under test.

Applies To: Voting system

Source: New requirement

1 Comment

Comment by Carolyn Coggins (Voting System Test Laboratory)

"Implementation details" is vague. Identify the specific information that needs to be captured.
5.1-E.2 Test plan, public test suites

For tests that the test lab is adopting from publicly available test suites, the test lab SHALL identify the public reference and document the implementation details that determine how the public tests are realized for the implementation under test.

Applies To: Voting system

Source: New requirement

5.1-E.3 Test plan, other test suites

For all other tests, the test lab SHALL incorporate all relevant information into the test plan as needed to reproduce the testing.

Applies To: Voting system

Source: New requirement

1 Comment

Comment by Carolyn Coggins (Voting System Test Laboratory)

Redundant requirement; see comment for 5.1 -E
5.1-F Test plan, responsible parties

The test lab SHALL identify the parties responsible for conducting the conformity assessment, including subcontracted test labs and engineers assigned to the task.

Applies To: Voting system

Source: New requirement

1 Comment

Comment by Traci Mapps (Voting System Test Laboratory)

What is the purpose of providing names of engineers assigned to the task?