Notices of Clarification

The EAC finds that based upon the language of Section 4.3 of the VSTCM a manufacturer must submit a certification application prior to conducting any certification testing. Any testing occurring after the execution of a contract or agreement for certification testing between a Voting System Test Laboratory and a registered manufacturer is presumed to be certification testing.

When a VSTL considers doing business outside the certification context, it must consider the implications of such a decision in light of the prohibitions related to conflicts of interest and prohibited practices. VSTL’s may not test voting systems if they have a conflict of interest between their responsibilities under the certification program and their financial interests. Having an outside contractual or fiduciary relationship with a manufacture whose product the VSTL is responsible for testing under EAC’s Certification program is a conflict of interest. VSTLs must also be cognizant of the impact of prohibited practices. A laboratory may not be involved in both the development of a voting system and the certification of a system. Voting system development includes any testing, consultation or design work performed in order to ready a system for the market place or the certification process. Generally, any testing performed on behalf of a voting system manufacture that was not otherwise performed pursuant to an EAC certification will be considered developmental. When a VSTL is involved with the development of a voting system, it is prohibited from future testing of that system under EAC Program.

The EAC seeks to encourage concurrent state and Federal testing. This policy will serve to lower overall testing costs by promoting state use of EAC accredited laboratories to conduct the more expensive and time consuming State testing requirements. These requirements could include items such as volume testing, any State specific environmental testing, and other State specific functionality testing. When states authorize a VSTL to perform concurrent testing, they are responsible for monitoring these testing procedures as they would if the testing was being conducted by the State itself. Additionally, the State remains responsible for State certification actions based upon the outcome of testing conducted by a VSTL. Concurrent state testing is not subject to EAC Certification or oversight.

EAC Certification Program Manual requires, as a condition of registration, that each manufacturer provide the EAC with a “list of all manufacturing and/or assembly facilities used by the manufacturer.”(Section 2.3.1.7 of the EAC Voting System Testing and Certification Program Manual). For the purposes of Section 2.3.1.7, “manufacturing and/or assembly facilities” applies to facilities that provide the following manufacturing services: 1. Final system configuration and loading of programs for customer delivery. 2. Manufacturing of component units of a voting system. 3. Manufacturing of major sub-assemblies of the voting system.

Under EAC Laboratory Accreditation Program: 1. The lead VSTL is fully responsible for all contracting with third party laboratories for testing under EAC’s Certification Program. Manufacturers shall not directly manage, control or compensate a subcontracted laboratory. 2. The use of or selection of third party laboratories is at the sole discretion of the lead VSTL. Third party laboratories are subcontractors to the lead VSTL. 3. The lead VSTL shall directly manage the testing project, including the intake and distribution of the manufacturer’s documentation, management of units under test, and the assessment and management of the testing process.

The EAC concludes that to insure voting systems subject to certification are tested in the most thorough manner possible, the integrity of the program requires that prior testing is only presumed valid when conducted by a third party laboratory while under the direction of an EAC accredited VSTL. In order, however, to allow voting systems currently in the testing process to move forward during this critical time, the EAC will allow the use of non-core environmental and EMC testing undertaken and completed within one year prior to the implementation of our program (January 1, 2005 to December 31, 2006) under the following conditions:- . VSTL’s must submit all such non-core environmental and EMC testing to the EAC for review- the VSTL or its designated sub contractor laboratory shall re-run the electrostatic disruption test - The results of this ESD test shall be submitted to the EAC for review and approval prior to the EAC accepting any prior environmental or EMC testing.

This clarification is issued to allow a manufacturer to more easily change a Mark of Certification when a product has been recertified to a different version of EAC standards and to allow the removal of the Mark of Certification in instances where a machine has been decertified. In addition, this clarification will accommodate voting system manufacturers who do very small batch production runs and need additional flexibility in producing and affixing the Mark. This clarification amends the general requirements outlined in Section 5.15 of the Testing and Certification Program Manual. This clarification also provides further information on distribution of the Mark of Certification and options for compliance actions for misuse of the Mark.

Conformance testing must begin with confirmation that a system functions as documented. However, testing must also show that the system will recognize and respond appropriately to incorrect as well as correct data and procedures as currently specified in the 2002 VSS and 2005 VVSG. In addition, testing should ensure that the system is robust and resistant against common user and technical sources of error. All reports available should accurately report the results of all valid votes. Audit records will include information showing the appearance of invalid or questionable data that were rejected so that potential recount related issues may be resolved. Testing should also be responsive to requirements that may not be adequately defined in the current published standards, especially those that show up under State testing and/or during actual elections. The goal is to catch as many errors as possible in testing before they show up in an election environment.

Clarification appears necessary to further describe the expected depth and completeness of the test plan that is necessary to help ensure that a comprehensive test campaign is carried out and to help ensure that test campaigns are consistent among different VSTLs. The test plan shall document the strategy and plan for testing each section of the applicable voluntary voting system guidelines and is to be used as a key tool to manage the test campaign and to verify that a voting system or component meets all VVSG and program defined requirements. The test plan shall be written with completeness and clarity that will allow all constituents to understand what testing will be conducted, to assess each group of VVSG requirements, and to assure that the test plan will remain a living document throughout the life of the test campaign. The objective is to address each section (in both Volume I and Volume II of the 2005 VVSG) in detail, and to clearly and succinctly describe the strategy and/or approach for testing each section

The EAC’s Program Manual states in part, “Participation [in testing] includes but is not limited to the observation of testing by the Manufacturer.” The purpose of this statement is to reiterate that under no circumstances can a voting system manufacturer be present in the testing room while certification testing is being conducted. This prohibition is in place to ensure that the conformance testing being conducted is an independent evaluation of the system to the standards without influence or interference by the manufacturer. The EAC does recognize that in some cases there is value in allowing manufacturers to witness a particular test or a re-creation of a test in order to allow them to comment on the proper system set up or operation. However, any such participation must be (1) at the discretion of the VSTL, (2) supervised by the VSTL and (3) clearly documented in order to maintain laboratory independence.- halting an active certification test and bringing the manufacturer into the testing room for a re-creation of the test being performed- creating for the manufacturer either a closed circuit video feed or a web cam feed to allow for real-time correspondence during testing- supervised access prior and during the testing to perform unscheduled and non-routine maintenance

 A change to a voting system’s TDP can be considered de minimis provided the change meets the requirements of section 3.5 of the EAC’s Program Manual. Changes to the TDP that have the potential to effect the reliability, functionality, capability, and operability of a system will not be considered de minimis and must pass modification testing before being approved.

The goal of this NOC is to guide the production of test reports that are clearer and more comprehensive by documenting:- All the components, and necessary information that comprise the version and configuration of hardware, software and COTS needed for the evaluation of the voting system;- The completeness and comprehensiveness of the testing performed; - The adequacy of the testing performed; and- The results of all the testing performed.All such information needs to be provided with clarity, completeness and without ambiguity, so that a wide range of readers and users of the document will be able to understand the evaluation that supports a system’s certification

Test Plans submitted for modifications to previously EAC certified voting systems should be brief and structured to minimize test plan development and review, while enabling the EAC to maintain solid control of the certification process. The test plan shall concisely document the strategy and plan for testing those sections of the VVVSG applicable to the modification or modifications submitted. The test plan shall be written with clarity that will allow all constituents to understand what testing will be conducted, to verify compliance to VVSG requirements, and to assure that the test plan will remain a living document throughout the life of the test campaign for the modification.

In order for a data change to be found de minimis, the change must not materially alter the system’s reliability, functionality, capability, or operation (i.e. meeting the requirements of section 3.5 of the Cerification Manual).  Under no circumstance shall a change to the system’s data be considered de minimis if it has a reasonable and identifiable potential to impact the system’s performance and compliance with applicable voting system standards.

The replacement of certain obsolete or end-of-life COTS components in an EAC certified voting system with equivalent components is permitted. Using the process outlined above, the equivalence of the replacement components can be demonstrated and be considered for a de minimis classification provided the change meets the requirements of section 3.5 of the EAC’s Program Manual and this clarification. The EAC will continue to monitor the use of COTS products in voting systems and make appropriate changes to its procedures in instances in which it sees minimal risk to voting system integrity and potential cost savings.

System validation is an extremely important pre-election and post-election activity. The system identification tools required by the manual are intended for Federal, State, and local officials to identify and verify that the equipment used in elections is unmodified from its EAC certified version.

This notice of clarification is to clarify the way discrepancies are listed in the Test Report. Discrepancies noted during testing shall be entered in the test report as individual occurrences, not batched based on test case or component. The only discrepancies that may be batched or grouped are source code discrepancies and TDP discrepancies. In addition, even though source code TDP discrepancies may be batched or grouped, the number of occurrences of each identified discrepancy must be tallied and the total number noted in the section for that batch (i.e. source code or TDP).

This notice of clarification is to clarify the information needed to describe changes to a system accepted in to the testing and certification program as a modification. In addition to the general description currently given as part of the system overview for modifications, each Test Plan and Test Report must include a detailed listing of all the changes made to the modified system. This can be included in the body of the Test Plan and Test Report (if it is a small number of changes); or referred to in the system overview and included as an Appendix (if it is a large number of changes). The detailed description shall include: name of component and/or module, version number and brief description of the change.

This NOC outlines the process under which the EAC, VSTL and voting system manufacturer can come to a mutually acceptable agreement regarding the testing of any new technology that is submitted for certification under the extensions clause of the VVSG and Section 3.2.2.4 of the Program Manual - via TTA meetings (as detailed in this NOC).

There is a discrepancy between the VSTL manual and the Cert manual that requires clarification. The statement in question (“The TRR does not apply to modifications.”) is in the VSTL manual and was overlooked during the revision and editing of the manuals.  The TRR applies to applications to test full systems and modifications.  Additionally, if the application is for a system modification, the VSTL is responsible for reviewing no less than 1% of code that has changed in every software package, module or product.

The initial practical application of the TRR process has lead the EAC to conclude that one item needs to be modified in the TRR in order to achieve the stated purpose of this program requirement. In order to be truly ready for testing, a voting system must be able to do substantially more than simply read a fully filled square or oval for optical scan systems.

Based upon VSTL feedback and recommendations, EAC shall receive Virtual Machines (appliances) from the VSTL for the trusted build. Trusted builds shall include this virtual machine and any related items, so that the system can be constructed or restored on another machine. Trusted builds shall be in the Open Virtualization Format (OVF)

Tight project schedules and other realities of voting system testing and certification have generally made manufacturers choose to move forward with “at risk” testing performed well before test plan approval. This reality conflicts with the requirement that the VSTL inform the EAC in writing once test cases have been completed. The EAC has generally not seen test cases until after test plan approval, even for "at risk" testing that may have occurred much earlier. In order to remedy this situation and to allow the EAC to more productively and appropriately comment on test cases before they have been run, the following clarification is noted: require the VSTLs to notify the EAC in writing and to upload test cases to the VRT when requested by the EAC before any tests have been run using the particular test case or test cases. This process will allow EAC to review test cases proximate to when they will be run and will hopefully eliminate any issues arising from after-the-fact test case review that might cause meaningful and potentially harmful delays in the testing and ultimate certification of the voting system

The EAC will use each manufacturer's quality assurance system and its internal procedures for controlling changes to, and versions of, it's voting systems when determining when a voting system is a "new" system.

The EAC has determined that allowing software de minimis changes is necessary to confront a rapidly evolving cybersecurity threat environment as well as allowing Manufacturers to quickly respond to changing jurisdictional requirements, where those requirements can be addressed with minor changes.

The intent of section 3.8 in the Voting System Test Laboratory Manual, version 2.0 is to make clear that the VSTL accreditation is subject to reassessment by NVLAP and the EAC with a period of approximately two years, at the discretion of the EAC. It is not intended to describe the EAC VSTL accreditation revocation process, which is described in chapter 5 of the Voting System Test Laboratory Manual, version 2.0.