5.1.2 Electromagnetic compatibility (EMC) emissions limits

Testing of voting systems for EMC emission limits will be conducted using the black box testing approach, which "ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions” [IEEE00].

It will be necessary to subject voting systems to a regimen of tests to demonstrate compliance with emission limits. The tests should include most, if not all disturbances that might be expected to be emitted from the implementation under test, unless compliance with mandatory limits such as FCC regulations is explicitly stated for the implementation under test.

5.1.2.1 Conducted emissions limits

5.1.2.1.1 Power port – low/high frequency ranges

As discussed in Part 1: 6.3.5 “Electromagnetic Compatibility (EMC) emission limits”, the relative importance of low-frequency harmonic emissions and the current drawn by other loads in the polling place will result in a negligible percentage of harmonics at the point of common connection, as discussed in [IEEE92]. Thus, no test is required to assess the harmonic emission of a voting station.

High-frequency emission limits have been established by Federal Regulations [FCC07] as a condition for offering equipment to the US market. In such cases, part of the requirements include affixing a label or notice stating that the equipment complies with the stipulated limits. Therefore, the VVSG does not suggest performing a redundant test.

5.1.2.1.2 Communications (Telephone) port

5.1.2.2 Radiated emissions

5.1.3 Other (non-EMC) industry-mandated requirements

5.1.3.1 Dielectric stresses

5.1.3.2 Leakage via grounding port

5.1.3.3 Safety

The presence of a listing label (required by authorities having jurisdiction) referring to a safety standard, such as [UL05], makes repeating the test regimen unnecessary. Details on the safety considerations are addressed in Part 1: 3.2.8.2 “Safety”.

5.1.3.4 Label of compliance

Some industry mandated requirements require demonstration of compliance, while for others the manufacturer affixes of label of compliance, which then makes repeating the tests unnecessary and economically not justifiable.

5.1.4 Non-operating environmental testing

This type of testing is designed to assess the robustness of voting systems during storage between elections and during transporting between the storage facility and the polling place.

Such testing is intended to simulate exposure to physical shock and vibration associated with handling and transportation of voting systems between a jurisdiction's storage facility and polling places. The testing additionally simulates the temperature and humidity conditions that may be encountered during storage in an uncontrolled warehouse environment or precinct environment. The procedures and conditions of this testing correspond to those of MIL-STD-810D, "Environmental Test Methods and Engineering Guidelines."

5.1.5 Operating environmental testing

This type of testing is designed to assess the robustness of voting systems during operation.

5.2 Functional Testing

Functional testing is performed to confirm the functional capabilities of a voting system. The accredited test lab designs and performs procedures to test a voting system against the requirements outlined in Part 1. Additions or variations in testing may be appropriate depending on the system's use of specific technologies and configurations, the system capabilities, and the outcomes of previous testing.

Functional tests cover the full range of system operations. They include tests of fully integrated system components, internal and external system interfaces, usability and accessibility, and security. During this process, election management functions, ballot-counting logic, and system capacity are exercised.

The accredited test lab tests the interface of all system modules and subsystems with each other against the manufacturer's specifications. For systems that use telecommunications capabilities, components that are located at the poll site or separate vote counting site are tested for effective interface, accurate vote transmission, failure detection, and failure recovery. For voting systems that use telecommunications lines or networks that are not under the control of the manufacturer (e.g., public telephone networks), the accredited test lab tests the interface of manufacturer-supplied components with these external components for effective interface, vote transmission, failure detection, and failure recovery.

The security tests focus on the ability of the system to detect, prevent, log, and recover from a broad range of security risks. The range of risks tested is determined by the design of the system and potential exposure to risk. Regardless of system design and risk profile, all systems are tested for effective access control and physical data security. For systems that use public telecommunications networks to transmit election management data or election results (such as ballots or tabulated results), security tests are conducted to ensure that the system provides the necessary identity-proofing, confidentiality, and integrity of transmitted data. The tests determine if the system is capable of detecting, logging, preventing, and recovering from types of attacks known at the time the system is submitted for qualification. The accredited test lab may meet these testing requirements by confirming the proper implementation of proven commercial security software.

5.2.1 General guidelines

5.2.1.1 General test template

Most tests will follow this general template. Different tests will elaborate on the general template in different ways, depending on what is being tested.

1. Establish initial state (clean out data from previous tests, verify resident software/firmware);
2. Program election and prepare ballots and/or ballot styles;
3. Generate pre-election audit reports;
4. Configure voting devices;
5. Run system readiness tests;
6. Generate system readiness audit reports;
7. Precinct count only:
   1. Open poll;
   2. Run precinct count test ballots; and
   3. Close poll.
8. Run central count test ballots (central count / absentee ballots only);
9. Generate in-process audit reports;
10. Generate data reports for the specified reporting contexts;
11. Inspect ballot counters; and
12. Inspect reports.

5.2.1.2 General pass criteria

5.2.2 Structural coverage (white-box testing)

This section specifies requirements for "white-box" (glass-box, clear-box) testing of voting system logic.

For voting systems that reuse components or subsystems from previously tested systems, the test lab may, per Requirement Part 2: 5.1-D, find it unnecessary to repeat instruction, branch, and interface testing on the previously tested, unmodified components. However, the test lab must fully test all new or modified components and perform what regression testing is necessary to ensure that the complete system remains compliant.

5.2.3 Functional coverage (black-box testing)

All voting system logic, including any embedded in COTS components, is subject to functional testing.

For voting systems that reuse components or subsystems from previously tested systems, the test lab may, per Requirement Part 2: 5.1-D, find it unnecessary to repeat functional testing on the previously tested, unmodified components. However, the test lab must fully test all new or modified components and perform what regression testing is necessary to ensure that the complete system remains compliant.

5.3 Benchmarks

5.3.1 General method

Reliability, accuracy, and misfeed rate are measured using ratios, each of which is the number of some kind of event (failures, errors, or misfeeds, respectively) divided by some measure of voting volume. The test method discussed here is applicable generically to all three ratios; hence, this discussion will refer to events and volume without specifying a particular definition of either.

By keeping track of the number of events and the volume over the course of a test campaign, one can trivially calculate the observed cumulative event rate by dividing the number of events by the volume. However, the observed event rate is not necessarily a good indication of the true event rate. The true event rate describes the expected performance of the system in the field, but it cannot be observed in a test campaign of finite duration, using a finite-sized sample. Consequently, the true event rate can only be estimated using statistical methods.

In accordance with the current practice in voting system testing, the system submitted for testing is assumed to be a representative sample, so the variability of devices of the same type is out of scope.

The test method makes the simplifying assumption that events occur in a Poisson distribution, which means that the probability of an event occurring is assumed to be the same for each unit of volume processed. In reality, there are random events that satisfy this assumption but there are also nonrandom events that do not. For example, a logic error in tabulation software might be triggered every time a particular voting option is used. Consequently, a test campaign that exercised that voting option often would be more likely to indicate rejection based on reliability or accuracy than a test campaign that used different tests. However, since these VVSG require absolute correctness of tabulation logic, the only undesirable outcome is the one in which the system containing the logic error is accepted. Other evaluations specified in these VVSG, such as functional testing and logic verification, are better suited to detecting systems that produce nonrandom errors and failures. Thus, when all specified evaluations are used together, the different test method complement each other and the limitation of this particular test method with respect to nonrandom events is not bothersome.

For simplicity, all three cases (failures, errors, and misfeeds) are modeled using a continuous distribution (Poisson) rather than a discrete distribution (Binomial). In this application, where the probability of an event occurring within a unit of volume is small, the difference in results from the discrete and continuous models is negligible.

The problem is approached through classical hypothesis testing. The null hypothesis (H₀) is that the true event rate, r_t, is less than or equal to the benchmark event rate, r_b (which means that the system is conforming).

The alternative hypothesis (H₁) is that the true event rate, r_t, is greater than the benchmark event rate, r_b (which means that the system is non-conforming).

Assuming an event rate of r, the probability of observing n or less events for volume v is the value of the Poisson cumulative distribution function.

Let n_o be the number of events observed during testing and v_o be the volume produced during testing. The probability α of rejecting the null hypothesis when it is in fact true is limited to be less than 0.1. Thus, H₀ is rejected only if the probability of n_o or more events occurring given a (marginally) conforming system is less than 0.1. H₀ is rejected if 1−P(n_o−1,r_bv_o)<0.1, which is equivalent to P(n_o−1,r_bv_o)>0.9. This corresponds to the 90th percentile of the distribution of the number of events that would be expected to occur in a marginally conforming system.

If at the conclusion of the test campaign the null hypothesis is not rejected, this does not necessarily mean that conformity has been demonstrated. It merely means that there is insufficient evidence to demonstrate non-conformity with 90 % confidence.

Calculating what has been demonstrated with 90 % confidence, after the fact, is completely separate from the test described above, but the logic is similar. Suppose there are n_o observed events after volume v_o. Solving the equation P(n_o,r_dv_o)=0.1 for r_d finds the "demonstrated rate" r_d such that if the true rate r_t were greater than r_d, then the probability of having n_o or fewer events would be less than 0.1. The value of r_d could be greater or less than the benchmark event rate r_b mentioned above.

Please note that the length of testing is determined in advance by the approved test plan. To adjust the length of testing based on the observed performance of the system in the tests already executed would bias the results and is not permitted. A Probability Ratio Sequential Test (PRST) [Wald47][Epstein55][MIL96] as was specified in previous versions of these VVSG varies the length of testing without introducing bias, but practical difficulties result when the length of testing determined by the PRST disagrees with the length of testing that is otherwise required by the test plan.

5.3.2 Critical values

For a fixed probability p and a fixed value of n, the value of rv satisfying P(n,rv)=p is a constant. Part 3: Table 5-1 provides the values of rv for p=0.1 and p=0.9 for 0≤n≤750.

Given no observed events after volume vo, the demonstrated event rate rd is found by solving P(no,rdvo)=0.1 for rd. The pertinent factor is in the second column (p=0.1) in the row for n=no; dividing this factor by vo yields rd. For example, a volume of 600 with no events demonstrates an event rate of 2.302585/600, or 3.837642×10−3.

Since the condition for rejecting H0 is P(no−1,rbvo)>0.9, the critical value vc, which is the minimum volume at which H0 is not rejected for no observed events and event rate benchmark rb, is found by solving P(no−1,rbvc)=0.9 for vc. The pertinent factor is in the third column (p=0.9) in the row for n=no−1; dividing this factor by rb yields vc. For example, if a test with event rate benchmark rb=10−4 resulted in one observed event, then the system would be rejected unless the actual volume was at least 0.1053605/10−4, or 105.3605. Where the measurement of volume is discrete rather than continuous, one would round up to the next integer.

The values in Part 3: Table 5-1 were generated by the following script and Octave[2] version 2.1.73.

silent_functions=1

# Function for the root finder to zero.  fsolve won't pass extra
# parameters to the function being solved, so we must use globals.
# nGlobal is number of events; pGlobal is probability.
function rvRootFn = rvRoot (rv)
  global nGlobal pGlobal
  rvRootFn = poisson_cdf (nGlobal, rv) - pGlobal
endfunction

# Find rv given n and p.  To initialize the root finder, provide
# startingGuess that is greater than zero and approximates the
# answer.
function rvFn = rv (n, p, startingGuess)
  global nGlobal pGlobal
  nGlobal = n
  pGlobal = p
  startingGuess > 0 || error ("bad starting guess")
  [rvFn, info] = fsolve ("rvRoot", startingGuess)
  if (info != 1)
    perror ("fsolve", info)
  endif
endfunction

function table
  printf (" n      P=0.1        P=0.9\n")
  for n = 0:750
    rv01 = rv (n, 0.1, -4.9529e-05*n*n + 1.0715*n + 2.302585093)
    rv09 = rv (n, 0.9,  4.9522e-05*n*n + 0.9285*n + 0.105360516)
    printf ("%3u %.6e %.6e\n", n, rv01, rv09)
  endfor
endfunction

fsolve_options ("tolerance", 5e-12)
table