Chapter 2: Conformity Assessment Process
2.1 Overview
Conformity assessment encompasses the examination and testing of software and firmware; tests of hardware under conditions simulating the intended storage, operation, transportation, and maintenance environments; inspection and evaluation of system documentation; and operational tests to validate system performance and functioning under normal and abnormal conditions. Conformity assessment also evaluates the completeness of the manufacturer's developmental test program, including the sufficiency of manufacturer tests conducted to demonstrate compliance with stated system design and performance specifications, and the manufacturer's documented quality assurance and configuration management practices. The assessment addresses individual system components or elements as well as the integrated system as a whole.
Beginning in 1994, the National Association of State Election Directors (NASED) began accrediting Independent Test Authorities for the purpose of conducting qualification testing of voting systems. The qualification testing process was originally based on the 1990 voting system standards and evolved to encompass the new requirements contained in the 2002 version of the standards.
The Help America Vote Act (HAVA) directs the U.S. Election Assistance Commission (EAC) to provide for the testing, certification, decertification, and recertification of voting system hardware and software by accredited laboratories. HAVA also introduces different terminology for these functions. Under the EAC process, test labs are "accredited" and voting systems are "certified." The term "standards" has been replaced with the term "VVSG."
Conformity assessment may be performed by one or more accredited test labs that together perform the full scope of tests required. Assessment may be coordinated across accredited test labs so that equipment and materials tested by one accredited test lab can be used in the tests performed by another accredited test lab.
When multiple accredited test labs are being used, the development of the test plan (see Part 2: Chapter 5: “Test Plan (test lab)”) and the test report (see Part 2: Chapter 6: ”Test Report (test lab)”) must be coordinated by a lead accredited test lab. The lead test lab is responsible for ensuring that all testing has been performed and documented in accordance with the VVSG and is ultimately responsible for the summary finding of conformance (see Requirement Part 2: 6.1-F).
Whether one or more accredited test labs are used, the testing generally consists of three phases:
- Pre-test activities;
- Chapter 3: overview of general testing approaches;
- Testing; and
- Post-test activities.
2.2 Scope of Assessment
The conformity assessment process is intended to discover vulnerabilities that, should they appear in actual election use, could result in failure to complete election operations in a satisfactory manner. This involves
- Operational accuracy in the recording and processing of voting data, as measured by report total error rate;
- Operational failures or the number of unrecoverable failures under conditions simulating the intended storage, operation, transportation, and maintenance environments for voting systems;
- System performance and function under normal and abnormal conditions; and
- Completeness and accuracy of the system documentation and configuration management records to enable purchasing jurisdictions to effectively install, test, and operate the system.
Conformity assessment involves several different kinds of testing, including
- Inspections, where the conformity of the voting system and manufacturer practices for configuration management and quality assurance are evaluated via expert review;
- Hardware testing, where the ability of the system to tolerate the physical conditions of its operation, transportation and storage is evaluated;
- Functional testing, where the conformity of the voting system's observable behaviors is evaluated;
- Performance testing, where the satisfaction of specified benchmarks is either evaluated in specific tests or monitored concurrent with other testing;
- Usability testing, where the performance is evaluated with human test subjects; and
- Vulnerability testing, where the system's resistance to attack is evaluated.
Voting system hardware, software, communications and documentation are examined and tested to determine suitability for elections use. Examination and testing address the broad range of system functionality and components, including system functionality for pre-voting, voting, and post-voting functions. All products for election use are tested in accordance with the applicable procedures.
Tests are conducted for new systems seeking initial testing as well as for modified versions of systems that have been previously tested.
Not all systems are required to complete every category of testing. Consistent with Requirement Part 2: 5.1-D, the test lab may find that proven performance of COTS hardware, software and communications components in commercial applications other than elections obviates the need for certain specific evaluations. However, as most functional testing exercises the complete system, COTS components are always tested together with other components of the voting system. Similarly, if a previous version of the same system has been tested, the test lab may find that complete retesting would be redundant, but some tests that exercise the entire system are always conducted. The background and rationale for these decisions regarding the scope of testing must be documented in the test plan.
The accredited test lab determines which tests are necessary to reassess a modified system based on a review of the nature and scope of changes and other submitted information including the system documentation, manufacturer test documentation, configuration management records, and quality assurance information. The accredited test lab may determine that a modified system is subject only to limited retesting if the manufacturer demonstrates that the change does not affect demonstrated compliance with these VVSG for:
- Performance of voting system functions;
- Voting system security and privacy;
- Overall flow of system control; and
- The manner in which ballots are defined and interpreted, or voting data are processed.
Limited testing is intended to facilitate the correction of defects, the incorporation of improvements, the enhancement of portability and flexibility, and the integration of vote counting software with other systems and election software.
In all cases, the system documentation and configuration management records are examined to confirm that they completely and accurately reflect the components and component versions that comprise the voting system.
2.3 Testing Sequence
Tests and inspections required by these VVSG need not be conducted in any particular order. Test labs should organize the test campaign to maximize overall testing effectiveness, to test in as efficient a manner as possible, and to minimize the amount of regression testing that is incurred when nonconformities are found and corrected. Test anomalies and errors are communicated to the system manufacturer throughout the process.
2.4 Pre-Test Activities
Pre-test activities include the request for initiation of testing and the pre-test preparation.
2.4.1 Initiation of testing
Conformity assessment is conducted at the request of the manufacturer. The manufacturer must:
- Request the performance of conformity assessment from among the accredited testing laboratories;
- Enter into formal agreement with the accredited test lab for the performance of testing; and
- Prepare and submit materials required for testing consistent with the requirements of the VVSG.
Conformity assessment is conducted for the initial version of a voting system as well as for all subsequent revisions to the system that are to be used in elections. As described in Part 3: 2.2 “Scope of Assessment”, the nature and scope of testing for system changes or new versions is determined by the accredited test lab based on the nature and scope of the modifications to the system and on the quality of system documentation and configuration management records submitted by the manufacturer.
2.4.2 Pre-test preparation
Pre-test preparation encompasses the following activities:
- The manufacturer and accredited test lab enter into an agreement for the testing to be performed by the accredited test lab;
- The manufacturer prepares and submits a TDP to the accredited test lab. The TDP consists of the materials described in Part 2: Chapter 3: ”Technical Data Package (manufacturer)”;
- The accredited test lab performs an initial review of the TDP for completeness and clarity and requests additional information as required;
- The manufacturer provides additional information if requested by the accredited test lab;
- The test lab witnesses the production of the implementation for testing;
- The manufacturer delivers to the accredited test lab all hardware and software needed to perform testing.
2.4.2.1 Documentation submitted by manufacturer
The manufacturerSHALL submit to the test lab a Technical Data Package conforming to the requirements of Part 2: Chapter 3: ”Technical Data Package (manufacturer)”.
Applies to: Voting system
DISCUSSION
The manufacturer must submit all the documentation necessary for the identification of the full system configuration submitted for evaluation and for the development of an appropriate test plan by the accredited test lab for conducting conformity assessment. This documentation collectively is referred to as the Technical Data Package (TDP). The TDP provides information that defines the voting system's design, method of operation, and related resources. It provides a system overview and documents the system's functionality, hardware, software, security, test specifications, operations procedures, maintenance procedures, and personnel deployment and training requirements. It also documents the manufacturer's configuration management plan and quality assurance program. If another version of the system was previously tested, the TDP would also include appropriate system change notes.
Source: [VVSG2005] II.1.5
2.4.2.2 Voting equipment submitted by manufacturer
Manufacturers may seek to market a complete voting system or an interoperable component of a voting system. In all instances, manufacturers must submit for testing the specific system configuration that will be offered to jurisdictions or that comprises the component to be marketed plus the other components with which the component is to be used. Under no circumstances will a component be assessed except as part of a complete voting system, and that assessment is valid only when that component is used with that same system (see Part 1: 2.3 “Conformance Designations”).
If needed for compliance with Part 3: 2.4.3.4 “Unmodified COTS verification”, the manufacturer SHALL supply the system with the COTS components omitted, for subsequent integration performed by or witnessed by the test lab.
Applies to: Voting system
DISCUSSION
See Part 3: 2.4.3.4 “Unmodified COTS verification”.
Source: New requirement.
The hardware submitted for conformity assessment SHALL be equivalent, in form and function, to the actual production version of the hardware units specified for use in the TDP.
Applies to: Voting system
Source: [VVSG2005] II.1.6.a
The firmware and software submitted for conformity assessment SHALL be the exact firmware and software that will be used in production units.
Applies to: Voting system
Source: [VVSG2005] II.1.6.b
Developmental prototypes SHALL NOT be submitted unless the manufacturer can show that the equipment to be tested is equivalent to standard production units both in performance and construction.
Applies to: Voting system
Source: [VVSG2005] II.1.6.c
Benchmark directory listings SHALL be submitted for all software/firmware elements (and associated documentation) included in the manufacturer's release as they would normally be installed upon setup and installation.
Applies to: Voting system
Source: [VVSG2005] II.1.6.d
2.4.3 Initial system build by test lab
The following requirements describe how test labs are to perform build of voting system software by the test lab.
Previously built voting system software being updated may be able to use the requirements found Part 3: 2.4.3.3 “Updating previously built voting system software executable code” to create the updated executable code including application logic, border logic, and third party logic.
2.4.3.1 Build environment establishment
The test lab SHALL assemble the build environment(s) used to create executable code including application logic, border logic, and third party logic.
Applies to: Voting system
Source: [EAC06] Section 5.6.1.2 and [VVSG2005] II.1.8.2.4
At least one representative from the manufacturer SHALL witness the assembly of the build environment.
Applies to: Voting system
Source: [EAC06] Section 5.6 and [VVSG2005] II.1.8.2.4
A representative from the test lab SHALL create a build environment establishment record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location the build environment was established; names, affiliations, and signatures of all people present; copies of the procedures used to assemble the build environment; list of software and hardware used to establish the build environment; and the voting system associated with the build environment.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL obtain the software and hardware required to establish the build environment.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-C documents the software and hardware required to assemble the build environment.
The test lab SHALL obtain COTS software and hardware required to assemble the build environment from the open market.
Applies to: Voting system
DISCUSSION
Note: manufacturers are required to supply non-COTS hardware and software as part of Requirement Part 3: 2.4.2.2-A.
The test lab SHALL remove any previously stored information on erasable storage media in preparation for using the media to assemble the build environment.
Applies to: Voting system
DISCUSSION
The purpose of this requirement is to prepare erasable storage media for use by the build environment. The requirement does not require the prevention of previously stored information leakage or recovery. Simply deleting files from file systems, flashing memory cards, and removing electrical power from volatile memory satisfies this requirement.
Source: [EAC06] Section 5.6.1.1
The test lab SHALL use the procedures found in the TDP to assemble the build environment.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-D documents the procedures to assemble the build environment. Test lab personnel can have manufacturers provide guidance during the assembly of the build environment, but test lab personnel must perform the actual assembly.
Source: [EAC06] Section 5.6.1.2
The test lab SHALL document as part of the build environment establishment record the reason for any deviation from assembly procedures found in the TDP.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-D documents the procedures used to assemble the build environment.
Source: [EAC06] Section 5.9
When digital signatures are associated with software, the test lab SHALL verify digital signatures before using the software for the build environment.
Applies to: Voting system
DISCUSSION
The digital signatures associated with the build environment may be from the manufacturer of the software, National Software Reference Library (NSRL), or other authoritative sources.
Source: [EAC06] Section 5.6.2.1
The test lab SHALL record as part of the build environment establishment record the results of digital signature verification including who generated the signature.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL copy the binary image of the assembled build environment to unalterable storage media.
Applies to: Voting system
DISCUSSION
This requirement creates a snapshot of the build environment before it is used to build the voting system software executable code. Unalterable storage media includes technology such as a CD-R, but not CD-RW.
The test lab SHALL create a digital signature for the binary image of the build environment, and include the digital signature on the unalterable storage media with the binary image.
Applies to: Voting system
Source: [EAC06] Section 5.6.1.3
2.4.3.2 Build of voting system software executable code
Previously built voting system software being updated may be able to use Requirement Part 3: 2.4.3.3 to create the updated executable code including application logic, border logic, and third party logic.
The test lab SHALL build the executable code including application logic, border logic, and third party logic of the voting system using the established build environment.
Applies to: Voting system
DISCUSSION
The build environment is established using the requirements in Part 3: 2.4.3.1 “Build environment establishment”.Source: [EAC06] and [VVSG2005] II.1.8.2.4
At least one representative from the manufacturer SHALL witness the build of executable code including application logic, border logic, and third party logic of the voting system.
Applies to: Voting system
Source: [EAC06] Section 5.6
A representative from the test lab SHALL create an executable code build record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location of the build; names, affiliations, and signatures of all people present; filenames of the source code and resulting executable code; voting system software version; name and version of the voting system (including certification number, if possible); and copies of the procedures used to build the voting system software executable code.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL validate manufacturer digital signatures on voting system software source code before placing source code on the build environment.
Applies to: Voting system
DISCUSSION
Requirement Part 3: 2.6.2.4-D requires manufacturers to provide voting system software source code with digital signatures as part of the TDP.
Source: [EAC06] Section 5.6.2.1
The results of digital signature validation including who generated the signature SHALL be part of the executable code build record for voting system software.
Applies to: Voting system
Source: [EAC06] Section 5.9
The test lab SHALL use the procedures found in the TDP to build the voting system software executable code including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-E documents the procedures to build voting system software executable code. Test lab personnel can have manufacturers provide guidance during the build of the voting system executable code, but test lab personnel must perform the actual build.
Source: [EAC06] Section 5.6.3
The test lab SHALL document as part of the executable code build record the reason for any deviation from build procedures found in the TDP.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-E documents the procedures to build voting system software executable code.
Source: [EAC06] Section 5.9
After voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL copy the binary image of the build environment (including source and executable code) to unalterable storage media.
Applies to: Voting system
DISCUSSION
This requirement creates a snapshot of the build environment after it has been used to build voting system software executable code. Unalterable storage media includes technology such as a CD-R, but not CD-RW.
Source: [EAC06] Section 5.6.2.3
After voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL create a digital signature for the binary image of the build environment (including source and executable code), and include the digital signature on the unalterable storage media with the binary image.
Applies to: Voting system
Source: [EAC06] Section 5.6.2.2
2.4.3.3 Updating previously built voting system software executable code
The following voting system software build requirements apply when updates to previously built voting system software has occurred. These requirements assume the original build environment can be used to create the updated software and a significant portion of original software is not being updated. If the original build environment cannot be used or a significant portion of the original software is being updated, then the requirements of Part 3: 2.4.3.1 “Build environment establishment” and Part 3: 2.4.3.2 ”Build of voting system software executable code”.
At least one representative from the manufacturer SHALL witness the establishment of the post build environment associated with the previously built voting system software, and the build of the updated voting system software executable code including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
This requirement does not modify the requirement found in Section 5.6 of the EAC Testing and Certification Program Manual [EAC06] requiring a representative from both the manufacturer and test lab to be present during the build.
Source: [EAC06] Section 5.6
The test lab SHALL establish the build environment using the original post build environment binary image associated with the previously built voting system software.
Applies to: Voting system
DISCUSSION
Requirements Part 3: 2.4.3.2-A.7 and Part 3: 2.4.3.2-A.8 create the post build binary image of the original built voting system software developed by the manufacturer. If the test lab does not posses the required hardware and software to create the build environment then Requirements Part 3: 2.4.3.2-A.7 and Part 3: 2.4.3.2-A.8 apply. This requirement extends the requirement found in [EAC06] Section 5.6.4.1 and 5.6.4.2 by explicitly stating the original build environment needs to be established.
Source: [EAC06] Section 5.6.4.1 and 5.6.4.2
The test lab SHALL remove previously stored information on erasable storage media in preparation for using the media to establish the build environment.
Applies to: Voting system
DISCUSSION
The purpose of this requirement is to prepare the erasable storage media for use by the original post build environment. The requirement does not require the prevention of previously stored information leakage or recovery. Simply deleting files from the file system, flash memory cards, and removing electrical power from volatile memory satisfy this requirement.
Source: [EAC06] Section 5.6.1.1
The test lab SHALL verify the digital signature of the original post build binary image associated with the previously built voting system software before using the binary image to establish the build environment.
Applies to: Voting system
DISCUSSION
This requirement does not modify the requirement found in Section 5.6.4.1 of the EAC Testing and Certification Program Manual [EAC06] that states the file signature of the build environment needs to be verified before use.
Source: [EAC06] Section 5.6.4.1
The result of digital signature verification including who generated the signature SHALL be part of the original post build environment establishment record.
Applies to: Voting system
Source: [EAC06] Section 5.9
A representative from the test lab SHALL create an original post build environment establishment record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location the original post build environment was established; names, affiliations, and signatures of all people present; copies of the procedures used to assemble the original post build environment; list of software and hardware used to establish the original post build environment; and the voting system associated with the original post build environment.
Applies to: Voting system
DISCUSSION
This requirement updates the requirement found in Section 5.9 of the EAC Testing and Certification Program Manual [EAC06] by specifying the information needed to be documented when establishing the build environment.
Source: [EAC06] Section 5.9
The test lab SHALL build the executable code including application logic, border logic, and third party logic of the updated voting system software.
Applies to: Voting system
DISCUSSION
This requirement does not modify the requirement found in Section 5.6.4.2 of the EAC Testing and Certification Program Manual [EAC06] that states the executable files are created; and extends the requirement found at Section 1.8.2.4 of [VVSG2005] Volume II in [VVSG2005] by requiring the use of the build environment established in Part 3: 2.4.3.1 “Build environment establishment”.
Source: [EAC06] Section 5.6.4.2 and [VVSG2005] II.1.8.2.4
The test lab SHALL validate manufacturer digital signatures on updated voting system software source code before placing the updated source code on the build environment.
Applies to: Voting system
DISCUSSION
This requirement modifies the requirement found in Section 5.6.4.2 of the EAC Testing and Certification Program Manual [EAC06] by constraining the verification to digital signature from a “file signature” (which could be a hash value or digital signature); extends 5.6.2.1 by specifying the verification to happen before software is installed on the build environment; and does not call for the digital signature of the build environment to be verified before installing the source code.
Source: [EAC06] Section 5.6.4.2
The result of digital signature verification including who generated the signature SHALL be part of the updated voting system software build record.
Applies to: Voting system
DISCUSSION
Requirement Part 3: 2.6.2.4-D requires manufacturers to provide voting system software source code with digital signatures as part of the TDP. This requirement updates the requirement found in Section 5.9 of the EAC Testing and Certification Program Manual [EAC06] by specifying the results of digital signature verification needs to be documented as part of the record when building the executable code.
Source: [EAC06] Section 5.9
The test lab SHALL use the procedures found in the TDP to build the updated voting system software executable code including application logic, border logic, and third party logic.
Applies to: Voting system
DISCUSSION
Requirement Part 2: 3.5.4-G documents the procedures to build the updated voting system software executable code. Test labs can have manufacturers assist in building of the updated voting system software executable code. This requirement extends the requirement found in Section 5.6.4.2 of the [EAC06] by specifying the use of the manufacturer supplied procedures to build the updated voting system software.
Source: [EAC06] Section 5.6.4.2
A representative from the test lab SHALL create an executable code build record that includes at a minimum: a unique identifier (such as a serial number) for the record; a list of unique identifiers of unalterable storage media associated with the record; the time, date, and location of the build; names, affiliations, and signatures of all people present; filenames of the source code and resulting executable code; voting system software version; name and version of the voting system (including certification number, if possible); and copies of the procedures used to build the updated voting system software executable code.
Applies to: Voting system
DISCUSSION
This requirement updates the requirement found in Section 5.9 of the [EAC06] by specifying the information needed to be documented when creating updated executable code.
Source: [EAC06] Section 5.9
After updated voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL copy the binary image of the updated build environment (including source and executable code) to unalterable storage media.
Applies to: Voting system
DISCUSSION
This requirement creates a snapshot of the updated build environment after it has been used to build the updated voting system software executable code. Unalterable storage media includes technology such as a CD-R, but not CD-RW. This requirement differs from the requirement found in Section 5.6.2.3 of the [EAC06] by creating the binary image after, instead of before, the updated software executable code has been built.
Source: [EAC06] Section 5.6.2.3
After updated voting system software executable code including application logic, border logic, and third party logic has been built, the test lab SHALL create a digital signature for the binary image of the updated build environment (including source and executable code), and include the digital signature on the unalterable storage media with the binary image.
Applies to: Voting system
DISCUSSION
This requirement differs from the requirement found in Section 5.6.2.2 of the [EAC06] by creating a digital signature on the binary image after the software executable code has been built as opposed to a “file signature” which could be a hash value or digital signature before the software executable code is built; although requirement 5.6.3.1 of the EAC Testing and Certification Program Manual requires “file signatures” for updated executable code.
Source: [EAC06] Section 5.6.2.2
2.4.3.4 Unmodified COTS verification
The following requirements describe how test labs are to verify that products identified as COTS are unmodified when used by the voting system.
The manufacturer SHALL document the procedures used to assemble and configure unmodified COTS components into the system supplied in Requirement Part 3: 2.4.2.2-A.
Applies to: Voting system
DISCUSSION
Test labs will assemble and configure unmodified COTS components into the voting system using the documentation provided by this requirement. Requirement Part 2: 4.4.1-A subitem e identifies all COTS components in the voting system, and Requirement Part 2: 3.8-D requires configuration data for unmodified COTS to be documented.
Source: COTS verification process per STS and CRT consensus, June 2006
Test labs SHALL obtain COTS components identified in Requirement Part 2: 4.4.1-A subitem 5 from open market suppliers of COTS components.
Applies to: Voting system
DISCUSSION
Test labs will procure the COTS components “off-the-shelf” from suppliers of the COTS components.
At least one representative from the test lab and manufacturer SHALL witness the assembly and configuration of the COTS components into the voting system.
Applies to: Voting system
The test lab SHALL assemble and configure the COTS components into the voting system.
Applies to: Voting system
The test lab SHALL document and maintain a record of the COTS assembly and configuration that includes, at a minimum: a unique identifier for each record; the time and date and location of the voting system build; names, affiliations, and signatures of all people present; copies of the procedures used to assemble and configure the COTS components; and identification of the voting system.
Applies to: Voting system
The test lab SHALL document deviations from the manufacturer documentation submitted for assembly and configuration of the COTS components.
Applies to: Voting system
2.5 Testing
Testing encompasses the preparation of a test plan, the establishment of the appropriate test conditions, the use of appropriate test fixtures, the witness of the system build and installation, the maintenance of test data, and the evaluation of the data resulting from tests and examinations.
2.5.1 Test plan
The accredited test lab SHALL prepare a test plan to define all tests and procedures required to assess conformity to the VVSG, including:
- Verifying or checking equipment operational status by means of manufacturer operating procedures;
- Establishing the test environment or the special environment required to perform each test;
- Initiating and completing operating modes or conditions necessary to evaluate the specific performance characteristics under test;
- Measuring and recording the value or range of values for the characteristics to be tested, demonstrating expected performance levels;
- Verifying, as above, that the equipment is still in normal condition and status after all required measurements have been obtained;
- Confirming that documentation submitted by the manufacturer corresponds to the actual configuration and operation of the system; and
- Confirming that documented manufacturer practices for quality assurance and configuration management comply with the VVSG.
Applies to: Voting system
DISCUSSION
Requirements on the content of the test plan are contained in Part 2: Chapter 5: ”Test Plan (test lab)”.Source: [VVSG2005] II.1.8.2.1
2.5.2 Test conditions
The accredited test lab may perform the tests in any facility capable of supporting the test environment.
Preparations for testing, arrangement of equipment, verification of equipment status, and the execution of procedures SHALL be witnessed by at least one independent, qualified observer, who SHALL attest that all test and data acquisition requirements have been satisfied.
Applies to: Voting system
Source: [VSS2002] II.9.6.2.2.a
When a test is to be performed at "standard" or "ambient" conditions, this SHALL refer to a nominal laboratory or office environment with a temperature in the range of 20.0 °C to 23.9 °C (68 °F to 75 °F) and prevailing atmospheric pressure and relative humidity.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.2.b
When a test is to be performed at conditions other than "standard" or "ambient," the test SHALL be performed at the required temperature and electrical supply voltage, regulated within the following tolerances:
- Temperature ± 2.2 °C (± 4 °F)
- AC electrical supply voltage ± 2 V
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.2.c
2.5.3 Test fixtures
Except as provided in Requirement Part 3: 2.5.3-B, the test lab SHALL NOT use simulation devices or software that bypass portions of the voting system that would be exercised in an actual election.
Applies to: Voting system
DISCUSSION
Devices or software that closely and validly simulate actual election use of the system are permissible. If a tabulator is specified to count paper ballots that are manually-marked with a specific writing utensil, it is not valid to substitute ballots that were mechanically marked by a printer. However, ballots that were marked according to manufacturer instructions can sometimes be recycled through a tabulator without invalidating the test. Limitations on this practice are provided in Requirement Part 3: 5.2.3-D.
The test lab may bypass the user interface of an interactive device in the case of environmental tests that:
- Would require subjecting test "voters" to unsafe or unhealthy conditions; or
- Would be invalidated by the presence of a test "voter."
The test lab may bypass the user interface of an interactive device in the case of environmental tests that:
Applies to: Voting system
2.5.4 Test data requirements
A test log of the procedure SHALL be maintained. This log SHALL identify the system and equipment by model and serial number.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.5.a
Test environment conditions SHALL be recorded.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.5.b
All operating steps, the identity and quantity of simulated ballots, annotations of output reports, the elapsed time for each procedure step, observations of equipment performance, and, in the case of non-operating hardware tests, the condition of the equipment SHALL be recorded.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.5.c
2.5.5 Test practices
The accredited test lab SHALL conduct the examinations and tests defined in the test plan to determine compliance with the voting system requirements described in Part 1 and Part 2.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.6
If any failure, malfunction or data error is detected, its occurrence and the duration of operating time preceding it SHALL be recorded for inclusion in the analysis of data obtained from the test.
Applies to: Voting system
Source: [VVSG2005] II.1.8.2.6.a
If a logic defect is responsible for the incorrect recording, tabulation, or reporting of a vote, the test campaign SHALL be terminated and the system SHALL be rejected.
Applies to: Voting system
DISCUSSION
Conformity assessment is not quality assurance. If a critical software defect is found, the system cannot be considered trustworthy even after the known fault is corrected, because the cases that the test lab does not have the opportunity to test can be expected to conceal similar faults. Any subsequent testing of a system based on or derived from the rejected system requires a new application and starting over.
Source: [GPO90] 7.1.1, [VSS2002] Overview, [VVSG2005] II.1.8.2.6.b
If a logic defect is found that is not responsible for the incorrect recording, tabulation, or reporting of a vote, the test campaign SHALL be suspended and the system returned to the manufacturer for correction and quality assurance.
Applies to: Voting system
DISCUSSION
Rejection may be a foregone conclusion if sufficient evidence has been collected to show that the reliability benchmark is not satisfied (see Part 3: 5.3.3 “Reliability”). Notwithstanding that, the manufacturer will be given the opportunity to correct noncritical software defects. Revisions to the software must be performed within the manufacturer's quality assurance and configuration management processes and must undergo manufacturer regression testing before the conformity assessment process is resumed. When it is resumed, the test plan should be revised to include regression testing for the change that was made.
Source: [VVSG2005] II.1.8.2.6.b, clarified and strengthened
If the anomaly is other than a logic defect, and if corrective action is taken to restore the equipment to a fully operational condition within eight hours, then the test campaign may be resumed at the point of suspension.
Applies to: Voting system
DISCUSSION
Rejection may be a foregone conclusion if sufficient evidence has been collected to show that the reliability benchmark is not satisfied (see Part 3: 5.3.3 “Reliability”). Notwithstanding that, the manufacturer may replace a component that has suffered a random failure, or the manufacturer may opt to suspend the test campaign in order to correct a hardware design defect that caused a nonrandom failure.
Source: [VVSG2005] II.1.8.2.6.c
If the test campaign is suspended for an extended period of time, the accredited test lab SHALL maintain a record of the procedures that have been satisfactorily completed. When testing is resumed at a later date, repetition of the successfully completed procedures may be waived provided that no design or manufacturing change has been made that would invalidate the earlier test results.
Applies to: Voting system
DISCUSSION
The considerations for resumption of testing are similar to those of Requirement Part 2: 5.1-D.
Source: [VVSG2005] II.1.8.2.6.d
The test campaign may resume after a deficiency is found if:
- The manufacturer submits a design, manufacturing, or packaging change notice to correct the deficiency, together with test data to verify the adequacy of the change;
- The examiner of the equipment agrees that the proposed change is responsive to the full scope of the deficiency;
- Any previously failed tests are passed by the revised system; and
- The manufacturer attests that the change will be incorporated into all existing and future production units.
Applies to: Voting system
DISCUSSION
Consistent with configuration management, the corrected system is formally a different system from the one that failed. The failure of the previous version is never "purged" entirely; rather, a new revision of the system is found not to suffer the same defect.
Source: [VVSG2005] II.1.8.2.6.e, clarified
2.6 Post-Test Activities
2.6.1 Voting system software version recommended for certification
The following requirements specify the version of the voting system software executable code including application logic, border logic, and third party logic that test labs included as part of a specific voting system recommended for certification.
2.6.1.1 Voting system software version
The test lab SHALL include voting system software executable code including application logic, border logic, and third party logic resulting from either an initial or final test lab build as part of the specific voting system recommended for certification.
Applies to: Voting system
DISCUSSION
The term “test lab build” refers to the voting system software executable code (including application logic, border logic, and third party logic) resulting from the test lab creating the executable code using (a) test lab procured equipment and build tools (such as compilers, linkers, etc.) and (b) source code and build procedures provided by the manufacturers. Note the test lab build is the result of using the requirements found in Part 3: 2.4.3 “Initial system build by test lab”.
Source: [VVSG2005] II.1.8.4.2
When no updates or modifications to the voting system software executable code including application logic, border logic, and third party logic has occurred since the initial test lab build, the test lab SHALL submit the executable code from the initial test lab build as part of the specific voting system recommended for certification.
Applies to: Voting system
When updates or modifications to the voting system software executable code including application logic, border logic, and third party logic has occurred since the initial test lab build, the test lab SHALL submit the executable code from a final test lab build as part of the specific voting system recommended for certification.
Applies to: Voting system
When required by Requirement Part 3: 2.6.1.1-A.2, the test lab SHALL use the requirements found in Part 3: 2.4.3 “Initial system build by test lab” to create a final test lab build of voting system software executable code including application logic, border logic, and third party logic
Applies to: Voting system
Source: [VVSG2005] II.1.8.4.2
2.6.2 Software distribution requirements for repositories, test labs, and manufacturers
The following requirements describe how voting system software must be distributed by test labs, voting system software manufacturers, and repositories such as the National Software Reference Laboratory (NSRL) to support traceability back to a reference version of the voting system software from a test lab, manufacturer, or repository. This traceability provides the basis for verifying that software installed on programmed devices of the voting system is certified voting system software. Although these requirements apply only to test labs, manufacturers, and repositories, other organizations that distributed voting system software such as jurisdictions may apply these requirements to support traceability back to reference versions of voting system software they distribute.
2.6.2.1 Software distribution package requirements
Software distribution packages are used to distribute software between different parties. Software distribution packages contain software from voting system manufacturers, third party manufacturers, test labs, repositories, and jurisdictions. The software contained on software distribution packages include voting application software, election specific software, installation software, third party software, and software integrity information.
Test labs, manufacturers, and repositories SHALL establish software distribution package master copies from which copies are created and distributed.
Applies