United States Election Assistance Comittee

Register to Vote!

Use the National Mail Voter Registration Form to register to vote, update your registration information with a new name or address, or register with a political party.

Note: If you wish to vote absentee and are a uniformed service member or family member or a citizen living outside the U.S., contact the Federal Voting Assistance Program to register to vote.

EAC Newsletters
and Updates

Sign up to receive information about EAC activities including public meetings, webcasts, reports and grants.

Give Us Your Feedback

Share your feedback on EAC policy proposalsElection Resource Library materials, and OpenEAC activities. Give feedback on general issues, including the Web site, through our Contact Us page.

Military and Overseas Voters

EAC has several projects under way to assist states in serving military and overseas citizens who register and vote absentee under the Uniformed and Overseas Citizens Absentee Voting Act. Learn more

Chapter 6: Test Report

6.1 Test report contents

Reporting performance test results for usability is covered under Part 1: 3.2.1.1 "Overall performance metrics".

1 Comment

Comment by Cem Kaner (Academic)

The test report shall be a public record .......... (Affiliation Note: IEEE representative to TGDC)
6.1-A Test report, include revision history

For modifications to previously tested systems, the test lab SHALL include the test reports that are precedential to the current evaluation.

Applies To: Voting system

DISCUSSION

It is anticipated that the test report will be delivered in electronic form, so the volume of data should not be a problem.

Source: New requirement

6.1-B Test report, include test plan as amended

The test lab SHALL include a copy of the test plan, amended to reflect any changes that were allowed during the course of the testing campaign.

Applies To: Voting system

DISCUSSION

Test plans must be updated whenever a change to a voting system requires deviation from the original test plan.

6.1-C Test report, implementation statement as amended

The test lab SHALL include the implementation statement submitted by the manufacturer, amended to reflect any changes that were allowed during the course of the testing campaign.

Applies To: Voting system

DISCUSSION

Because minor defects in a system may be corrected during the course of the testing campaign, the system that completes the conformity assessment process might not be identical to the one for which an implementation statement was submitted. The product identification for the revised system must be different. Also, if a system fails a test for a particular voting variation, the manufacturer and test lab may agree to eliminate that voting variation from the list of classes to which conformity assessment is desired rather than correct the system.

3 Comments

Comment by Brian V. Jarvis (Local Election Official)

If any defects (major or minor) are corrected during the course of the testing campaign, the delivered baseline is considered changed (and no longer baselined). Also, it should be assured that only the manufacturer (and not the testing lab) is permitted to make those corrections to the software. After making the necessary changes, as well as updating all related deliverable and internal software work products, the manufacturer would then be permitted to re-submit the system to the test lab for conformity assessment so that the submitted system is always the same one for which the implementation statement was submitted. Also, if a system fails a test for a particular voting variation, resulting in the elimination of that voting variation from the list of classes to which conformity assessment is desired then, one of two outcomes must result: (1) the defective code must be explicitly deactivated and documented as such in the delivered documentation or (2) the defective code constitutes "dead code" and must be deleted from the system; dead code can never, ever be submitted as part of a delivered baseline (and the system must be corrected).

Comment by Diane Gray (Voting System Test Laboratory)

Discussion seems to conflict with Voting System Test and Certification Manual Section 4.5 and VVSG 2005 Volume I Section 1.6.4.

Comment by Janet Baer (Voter)

All computer systems are subject to subtle errors. Moreover, computer systems can malfunction or be deliberately corrupted at any stage of their design, manufacture, and use. For example, when ordering products from Staples, we got a blue screen of death twice! That was yesterday. The methods used to corrupt computer systems can be extremely difficult to foresee and detect. Therefore, it is crucial to the integrity of elections that voting systems provide a means of recording and recovering voter intent that does not depend on the reliability of software. We need a paper trail.
6.1-D Test report, witness build

The test lab SHALL include a copy of the record of the final (witnessed) build and sufficient description of the build process to reproduce it.

Applies To: Voting system

DISCUSSION

See Part 3: 2.6.1 "Voting system software version recommended for certification".

Source: New requirement

6.1-E Test report, setup validation info

The test lab SHALL identify the repository for software reference information and include the unique identifier assigned to the software reference information by the repository.

Applies To: Voting system

Source: New requirement

1 Comment

Comment by Carolyn Coggins (Voting System Test Laboratory)

"The VVSG should provide the matrix that satisfies this requirement. The overall design of the VVSG would be improved if it was contained in a matrix form, and instead of having all of this redundant content there were columns that identified the systems to which individual requirements applied. It's visually easy to understand and read. "
6.1-F Test report, summary finding

The test lab SHALL include a summary finding of whether or not the implementation under test satisfies all applicable, mandatory ("SHALL") requirements of the Voluntary Voting System Guidelines.

Applies To: Voting system

6.1-G Test report, reasons for adverse opinion

If the test lab finds that the implementation under test does not satisfy all applicable, mandatory ("SHALL") requirements of the Voluntary Voting System Guidelines, the test lab SHALL identify each of the specific requirements that is not satisfied.

Applies To: Voting system

6.1-H Test report, evidence supporting adverse opinion

For each unsatisfied mandatory requirement, the test lab SHALL describe the inspections or tests that detected the nonconformities and include applicable evidence (e.g., vote data report, citation of logic error in source code).

Applies To: Voting system

6.1-I Test report, anomalies

The test lab SHALL summarize all failures, errors, nonconformities and anomalies that were observed during conformity assessment, no matter how minor.

Applies To: Voting system

2 Comments

Comment by Carolyn Coggins (Voting System Test Laboratory)

There is no such thing as a minor failure, non-conformity etc. A system must pass all requirements or it fails.

Comment by Diane Gray (Voting System Test Laboratory)

Please clarify: presume typos, incorrect page numbers, etc. considered outside the realm of minor anomalies?
6.1-I.1 Test report, deficiencies corrected during test campaign

The test lab SHALL identify those deficiencies that were corrected during the course of the testing campaign and identify the inspections or tests that confirm that the deficiencies were corrected.

Applies To: Voting system

DISCUSSION

For minor defects of a localized nature, the test lab may permit the manufacturer to correct the fault without incurring a complete regression test of the system. However, a certifying authority may require that revised documents be submitted whenever changes are made.

1 Comment

Comment by Diane Gray (Voting System Test Laboratory)

Define "a certifying authority". Are the revised documents in addition to any documented changes required by this VVSG?
6.1-J Test report, benchmarks

For requirements that specify benchmarks, the test lab SHALL report the result of the measurement for the implementation under test.

Applies To: Voting system

6.1-J.1 Test report, failure rate

The test lab SHALL report the observed cumulative failure rate and the failure rate that was demonstrated with 90 % confidence for each type of device, for each applicable failure type in Part 1: Table 6-3 (Part 1: 6.3.1.5 "Requirements").

Applies To: Voting device

DISCUSSION

See also Part 3: 5.3.2 "Critical values". "Type of device" refers to the different models produced by the manufacturer. These are not the same as device classes. The system may include several different models of the same class, and a given model may belong to more than one class.

1 Comment

Comment by Traci Mapps (Voting System Test Laboratory)

Please define how the 90% confidence rate is determined.
6.1-J.2 Test report, error rate

The test lab SHALL report the observed cumulative report total error rate and the report total error rate that was demonstrated with 90 % confidence for the system as a whole.

Applies To: Voting system

DISCUSSION

See Part 3: 5.3.4 "Accuracy".

1 Comment

Comment by Cem Kaner (Academic)

The report total error rate is not a statistically meaningful measurement. It is easy to manipulate this statistic by including a large pool of easy-to-pass tests. This is an example of the fundamental difference between hardware and software reliability. We expect the hardware device to fail on a random basis when we present identical tests, but we should not expect that of the software. Software failures are more likely to be triggered by combinations of data and sequence that have not been previously tested. .......... (Affiliation Note: IEEE representative to TGDC)
6.1-J.3 Test report, misfeed rate

For paper-based tabulators and EBMs, this SHALL include the observed cumulative misfeed rate and the misfeed rate that was demonstrated with 90 % confidence for each type of device.

Applies To: Paper-based device Λ Tabulator, EBM

DISCUSSION

See Part 3: 5.3.5 "Misfeed rate".

6.1-K Test report, ballot tabulation rate

For paper-based tabulators, the test lab SHALL report the ballot tabulation rate used in tests.

Applies To: Paper-based device Λ Tabulator

DISCUSSION

Stress tests might use a higher rate than other tests.

1 Comment

Comment by Carolyn Coggins (Voting System Test Laboratory)

Is the rate the number of ballots in a test or the number of ballots in a specific period of time? Rates are very different for different tests.
6.1-L Test report, shoulds that were not done

The test lab SHALL identify each applicable, non-mandatory ("SHOULD") requirement to which nonconformity was demonstrated.

Applies To: Voting system

DISCUSSION

Test labs are not required to test every "should" requirement; however, if they do, they must report the results.

2 Comments

Comment by Carolyn Coggins (Voting System Test Laboratory)

Is this correctly stated? Isn't it a demonstration of conformity? Doesn't testing non-conformity mean the labs test everything to confirm it's non-conforming?

Comment by Diane Gray (Voting System Test Laboratory)

"Should" requirements would normally be tested only if the vendor stated the voting system performed it. What is the basis for this requirement?
6.1-M Test report, waived tests

The test lab SHALL identify all tests that were waived.

Applies To: Voting system

DISCUSSION

A test is waived if the documented assumptions of an applicable test are not met by the implementation under test. A test that pertains to a system or device class that was not claimed in the implementation statement is implicitly assigned the verdict Not Applicable.

1 Comment

Comment by Brian V. Jarvis (Local Election Official)

Recommend enhancing the requirement (or creating a new requirement) to indicate that the test lab shall not waive a mandatory ("SHALL") requirement.
6.1-N Test report, timeline

The test lab SHALL include a timeline of the testing campaign as it actually occurred.

Applies To: Voting system

2 Comments

Comment by Brian V. Jarvis (Local Election Official)

Recommend enhancing this requirement with specifics defining what artifacts/data are necessary to document the timeline of the testing campaign.

Comment by Diane Gray (Voting System Test Laboratory)

How specific should this timeline be: is date range adequate, or are further details required?
6.1-O Test report, compensatory procedures

The test lab SHALL list any specific election management practices that are required for the voting system to satisfy the requirements of the VVSG.

Applies To: Voting system

DISCUSSION

For example, if additional procedures must be followed in order to safeguard the secrecy of the vote, these must be documented. If a system requires unusually onerous procedural compensations because customary system safeguards are absent, this may impact certification decisions.

Source: New requirement

2 Comments

Comment by Brian V. Jarvis (Local Election Official)

Due to the significant public scrutiny that voting machines are going to receive for the forseeable future, my recommendation is that no certification should be given if any additional procedures must be defined during testing in order to satisfy stated requirements. Otherwise, this is clear indication that either the system was not sufficiently tested by the manufacturer prior to delivery to the testing lab or that the testing lab developed deficient testing procedures. Either case is unacceptable. Testing must be spotless or the testing must stop, the appropriate testing documentation updated and then, testing restarted (not simply resumed). Both development and testing procesess must be free of defects. End-to-end testing must be free of any defects.

Comment by Carolyn Coggins (Voting System Test Laboratory)

This is extremely broad. Every system has specific management practices. Also where is this identified? In the matrix? Discrepancies? For it to be effective it has to be in a consistent location.
6.1-P Test report, warrant of accepting change control responsibility

If any changes to the system are required to complete conformity assessment, the test lab SHALL include a signed warrant from the manufacturer that those changes will be included in the product that is delivered to customers.

Applies To: Voting system

Source: New requirement

2 Comments

Comment by Brian V. Jarvis (Local Election Official)

From a Quality Assurance point of view, this is not a good policy (nor a good requirement). A signed warrant is not good enough. (Trust but verify.) If the manufacturer is allowed to make changes or fixes to the system, the system should go through another round of end-to-end testing. There should be absolute assurance that the product that is delivered to customers is the identical product that was qualification tested. That's just the way Quality Assurance works and there's no way around it. I've been a Quality Manager for years and I've never seen a product release process work like you've defined.

Comment by Diane Gray (Voting System Test Laboratory)

What entity is responsible for ensuring the changes are included in the delivered product?
6.1-Q Test report, issues list

The test lab SHALL list and explain any concerns that SHOULD be brought to the attention of readers and/or the VVSG interpretations and maintenance processes.

Applies To: Voting system

DISCUSSION

Any unresolved concerns may be documented in the test report. "Concerns" would include ambiguities in the VVSG, interpretation conflicts, requirements that appear to do more harm than good, loopholes in the VVSG (where it is possible to satisfy the technical requirements while failing to satisfy their intent), and other issues whose resolution would require action by outside authorities.

2 Comments

Comment by Diane Gray (Voting System Test Laboratory)

Will any follow-up action be required to ensure unresolved concerns are addressed?

Comment by Carolyn Coggins (Voting System Test Laboratory)

VVSG interpretations are the role of the EAC. This should be resolved prior to issuance of the report.