The EAC’s Managing Election Technology Series
"The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency."
The Election Official of today is an Information Technology (IT) Manager – whether they think they are, whether they want to be, or whether they were trained to be. IT Management requires a unique set of attitudes, knowledge and skills in order to plan, direct, and control contemporary election administration. This series of guides to managing election technology identify the primary areas in which the effective Election Official must recognize their role as an IT manager and provides ideas and best practices to assist in accommodating the demands of the modern election’s office.
Implementing Voting Systems with COTS Products
Over the course of the past several years, election jurisdictions seeking to purchase new election systems have begun a significant shift in focus away from the traditional approach of procuring predominantly proprietary election systems and toward procuring systems largely composed of commercial products. This shift follows what has become the practice over the past decade in state and U.S. Federal Government procurements, particularly those of the Department of Defense (DoD) and other large agencies. Many current Requests for Proposals (RFPs) being issued by government agencies now include a mandate concerning the amount of “COTS” (commercial off-the-shelf) products that must be included.
The considerations noted in this paper related to COTS products are cautionary: it is obvious to observers of election administration in this country that the growth in proprietary hardware/software costs are likely to continue, and that appropriate use of commercially- available products is one of the remedies that might enable election jurisdictions to acquire new and improved systems in a cost-effective manner. Additionally, COTS-solutions may provide needed flexibility in managing the service life of election systems as well as increasing their adaptability to changing requirements.
Like any solution to a problem, there are drawbacks as well as benefits to using COTS in election systems: while it may not be readily apparent to the election community as yet, many trade-offs exist when integrating a predominantly commercial product into an election system. Using COTS components in any given circumstance might prove cheaper and easier to implement, but this is certainly not guaranteed. Election officials, in their capacity as IT managers, must understand that the use of a COTS component may be a reasonable solution, but its use should be the product of careful analysis, reasoning, well-formed policy and practicable engineering decisions. COTS solutions present both opportunity and risks to jurisdictions who specify their inclusion in their election systems.
This paper discusses major considerations election officials should ponder before determining the extent to which their future election or voting system implements a predominantly COTS inclusive approach.
1. Use the term “COTS” precisely.
It is important to get a clear definition of the term “COTS.” The following definition is included in the EAC Voluntary Voting System Guidelines, Version 1.1 and is the definition agreed to by the panel convened at the 2011 EAC Roundtable on COTS:
COTS: “Software, firmware, device or component that is used in the United States by many different people or organizations for many different applications other than certified voting systems and that is incorporated into the voting system with no manufacturer- or application-specific modification.”
Note that some items that appear to be COTS are really more correctly modified COTS1. Any change made by an integrator or third party to a COTS product disqualifies it from being truly COTS. True COTS should be able to be purchased at a retail store like Best Buy or directly from the manufacturer (Dell or Apple, for example) and be able to be plugged into the election system with no modification of the COTS product and only trivial accommodation required from existing components (ex: installing a COTS printer may require the installation and customization of a printer driver)
Likewise, it is equally important to define the term “proprietary”:
Proprietary: “A technology or product that is owned and distributed exclusively by a single company and that is protected by patent, copyright or trademark.”
A common example of a proprietary product is Adobe Acrobat, whose Portable Document Format (PDF) files can only be read with the Acrobat Reader.
To further complicate best efforts to use these terms in a very precise manner, many products (and many, if not most, current voting system components) use numerous COTS or slightly modified COTS products) in what we generally term “proprietary” voting systems. This fact is best illustrated by the fact that several current voting system manufacturers use the same COTS scanning device as the basis for their proprietary precinct count optical scanning products.
2. Understand the impact of COTS products on the RFP and selection process
The use of COTS products has significant impact on both the RFP requirements and the evaluation of proposals, an impact that must be understood early in the election system acquisition process. When purchasing COTS based systems, an election official must have a full accounting of the various implementations of the systems as well as who is responsible for each portion of the process supported by the systems. The purchasing jurisdiction must view itself as an election system integrator instead of the more traditional view of the election official as an election system implementer. The election official may want to include an IT integration requirement to the RFP so there is a clear line of responsibility and accountability for bringing together the various COTS based systems with already existing election technology in the office. In the end, the election official will be held accountable for all the components and systems working together as intended.
Election officials will do well to work closely with their IT procurement officers to ensure that there is no misunderstanding about the intended role of the COTS products, their maintenance, and the potential impact of unplanned and unauthorized substitutions.
3. Understand COTS impact on system integration.
Integrating COTS components into your system is not necessarily simpler than integrating proprietary components, and may in fact be more difficult. The current thrust toward incorporating COTS products takes place in the context of “COTS-based voting/election systems.” That is, complex groupings of components, interacting in diverse ways, and in which introducing commercial components will simultaneously result in lower cost as well as providing the system “plug-and-play” characteristics
The phrase “plug-and-play” is significant, and is frequently the unspoken motivator for much of the current interest in COTS products. It conjures up a “software centric” environment wherein heterogeneous hardware components can be easily inserted or replaced, and in which components interconnect and interoperate without modification or effort on the part of the integrator2. It is based on the realities of the larger hardware world, where some degree of “plug and play” really does exist; boards, cables, printers, monitors, and keyboards can all be purchased, replaced, and upgraded independently and easily. Even with this degree of interoperability however, changes in hardware can have significant impact on a system. (As an example, substituting a short, shielded cable with a standard 3m USB cable might very well have a negative effect on the EMI characteristics of the system.)
The “plug and play” notion also rests on assumptions about data that different tools will share. For the past several years, the IEEE P1622 committee3 has been working to specify a standard or set of standards for a common data format for election systems partially in order to assist in the integration and testing of components from different manufacturers and vendors. This work shows much promise, but still has an incredibly long way to go before allowing for true “plug and play” capability in election systems.
4. Understand COTS impact on the testing process.
Testing and validation of COTS-based voting and election systems is a substantially different process than testing and validating proprietary systems. All three pre- election system test efforts and post-election audits will be impacted by the introduction of primarily COTS based systems:
Federal Certification Testing
COTS products are, by nature, tested very differently than traditional voting system products. On the COTS manufacturer side, testing is market driven, not statutory or rule drive like Federal and State voting system testing. Significant research will need to be done to determine exactly what levels of testing are appropriate for COTS based voting and election systems.
Much will depend upon the availability of testing and quality assurance information available from the COTS vendor. Certainly and at minimum, COTS products will need to be functionally tested with the full voting/election system to be certain that at least the versions of COTS submitted for testing work as intended in the system. New requirements (VVSG) must be written to focus more on the product/system interface level and focus on performance or functional characteristics (i.e., what should the product/system be capable of doing as opposed to how the product/system should do it). These functional characteristics should also be quantifiable and testable by the VSTLs or others.
Risk will also need to be factored into determinations on the level of test effort. What is the risk that the COTS product has defects? What percentage of allowable defects is acceptable to the COTS manufacturer(s)? We know that commercial products are never defect- free. Large commercial product vendors conduct a continuous balancing act between the quality of the system and time-to-market delivery, and often time-to market considerations win out over quality. In addition, while counterfeiting is an infrequent occurrence in COTS products, it does happen and when it occurs, it presents an additional obstacle to testing and assurance.
Federal certification testing also places significant importance on finding the root cause of failures encountered during the testing process. For some COTS-based system failures, finding the actual source of the failure, the root cause, or even re-creating the failure, will be more complex and time- consuming. This will be especially true when the COTS election system involves products from multiple COTS vendors.
State certification testing will have to address many of the same issues faced by the Federal certification program, but perhaps with fewer resources and less time to complete the testing.
States who engage in certification testing of voting systems will need to modify and validate their testing protocols to accommodate COTS components.
State/Local Acceptance Testing
The purpose of acceptance testing is to verify that the certified version of the election system is delivered to the jurisdiction and that the system can perform the functions required by the State and/or local election jurisdiction. Acceptance testing becomes a more complex process when integrating COTS components. Commercial COTS products typically have an accelerated development lifecycle and may change every 1-2 years or less. This will guarantee that units delivered to the jurisdiction will likely not be identical to either the certified version, or to versions delivered at other times.
Another factor to consider when acceptance testing COTS based systems is determining how to test to ensure that the COTS products work correctly with all other products in your dependent voting or broader election system. Do they negatively impact your election night reporting system? Your e- pollbook system?
Jurisdictions will need to implement strict version control protocols to ensure that all individual units are using the same software releases, or when not possible, to document the difference.
Logic & Accuracy (L&A) Testing
Pre and post-election logic and accuracy testing will be impacted as the system may actually introduce new COTS components at any time, thus making your system “new” again every single time it is used. To that end, election officials will likely want to revise L&A test scripts each time the system changes. These script changes may be major or minor, depending on the nature of the system change.
L&A Testing will continue to evolve so that not only is the ballot being tested, but that the system itself is being tested. L&A tests that utilize sampling methods will have to be expanded to include a comprehensive review of all equipment to be used in the upcoming election. This use model will need to include end-to-end testing methods, pushing past just vote capture and tabulation, to include election night reporting and post election activities that are dependent upon the correctness of the system.
Audits and Other Post-Election Testing
Any jurisdiction post-election audit procedures will need to be reviewed to identify controls that may be impacted by the revision of COTS components in the system. In fact, the COTS components themselves may become the subject of such audits.
5. Understand that a COTS approach makes a voting or election system dependent on the COTS vendors.
Vendor support for the commercial components in a COTS-based system is critical to the success of that system; many unforeseen problems can accompany a commercial system after deployment. The role of the COTS components’ vendors can be a decisive factor in successfully implementing and maintaining a COTS- based system. Several aspects are especially significant.
Does the COTS vendor supply adequate documentation for the component or components in question? Some COTS products offer extensive and useful documentation, but this is by no means universal. Is the documentation well- written and accurate? Can it be relied upon to be the integrator in order support election system testing, maintenance and future development? What kind of user support is available? Is the vendor responsive to user inquiries? In the current predominantly proprietary voting system world, problem calls will generally go to a help desk or to a field representative of your system manufacturer at their headquarters or in your own State or jurisdiction. COTS products often use outsourced help desk/call centers that are located in various locations around the world with varying levels of service.
Finally, if the COTS component is to be part of a election system that is expected to operate for several years, what are the probabilities that the company will exist for that time? Finally, even assuming that the company exists, how long will it support the COTS product in question? Vendors often phase out their support for any given product: would such an occurrence have an impact on the maintenance of your election system? By way of example, on September 30, 2015, Apple stopped signing4 iOS 8.4.1 and the first version of iOS 9 for the iPhone, iPad, and iPod touch, meaning users could no longer upgrade or downgrade to those versions of iOS . Apple is now signing iOS 9.0.1 and iOS 9.0.2 only.
6. Realize the cost of maintaining a COTS system.
The presence of COTS components does not necessarily mean low maintenance cost. COTS components can, in fact, cause complex problems during system upgrade and system maintenance. These problems could potentially exceed the maintenance cost of a proprietary system.
First, upgrading COTS-based system software means that as new releases of the commercial components are made by the various vendors, the system will incorporate them. In some instances, users can refuse upgrades and new releases, but it is safer to assume that the upgrades are inevitable.
Commercial vendors tend to support only a limited number of versions at any one time, and ignoring a vendor’s new releases is not an appropriate long term solution.
A further complicating factor is that different COTS components of the system will likely be upgraded at widely varying intervals; licenses will be invalidated and need to be revalidated for different parts of the system at random intervals. It should be kept in mind that COTS component upgrades can result in unforeseen problems including incompatible files and databases, different naming conventions, and software author and guarantee that the code has not been altered or corrupted since it was signed by use of a cryptographic hash.
introduction of new conflicts between COTS components. Depending on the number of COTS components and different COTS vendors, the effect of these multiple dependencies can vary from short-term user inconvenience to total system instability.
Finally, election officials will need to begin looking at voting and election system components more as disposable commodities rather than assets. “Maintenance” of these systems will most often entail replacing, upgrading and substituting components rather than the traditional model of sending a system back to the manufacturer, or having them come into your office or warehouse to conduct service or maintenance.
7. Implementing COTS is not an automatic cost-saver
Although it might initially appear that the availability of COTS components brings down election system costs, there are also offsetting costs to consider and manage over the course of the COTS system lifecycle.
One of the most overlooked, yet potentially significant costs associated with the use of COTS products is effective COTS-specific planning and budgeting. Without adequate planning, the jurisdiction may become reactive to the inevitable COTS-driven obsolescence situations. These situations limit management options (when expanding jurisdictional options is likely one of the primary reasons for moving to COTS in the first place…) and could force jurisdictions to adopt more costly solutions when COTS components go end-of-life.
Another cost related consequence of using rapidly changing COTS products within an election system is the likelihood, as mentioned elsewhere in this paper, that the system will include more than one configuration of the COTS component or components used in the system. This situation requires a rigorous application of configuration management (CM) processes to document and manage system baselines. Documenting product and system changes and instituting strong CM processes ensures the ability to determine the impact of product changes to all affected configurations of the election system.
Without the traditional voting system vendor model, the election administrator will likely become the system integrator and therefore will need to assume these roles and responsibilities within the election office. Without adequate in-house expertise in CM management, the election official will need to hire additional staff or contract out to a third party for such expertise.
3. Implementing COTS election systems must be part of a large-scale paradigm shift
A change in mindset for election officials is as important as any change in technology when using election systems depending solely or predominantly on COTS products.
The move to the next generation of COTS- based voting and election systems constitutes a significant paradigm shift not only for programmers, system developers and integrators, but also constitutes a significant paradigm shift for the election officials charged with the operation, testing, and day-to-day maintenance of these systems as well. Election officials need to assess their ability and their staff’s ability to work in the often technologically volatile world of COTS. Finding staff with the right attitude and right skills set may be difficult and expensive to maintain.
The move towards COTS-based voting and election systems is not simply a technological change. It affects everyone interacting with the system in profound ways. Election jurisdictions will experience changes in the activities they undertake, their structure, required training, IT policies, and the relationships between the election jurisdiction and their vendors. Election officials will need to leverage “lessons learned” by other government agencies and industry and imbed them in a practical manner into an overall management plan to effectively acquire and support COTS based election systems.
Conclusion: COTS Benefits and Mitigating Risks
Election officials must continue to find ways to drive down the operational costs of elections, while maintaining inventories of technologies that meet the growing expectations of their constituencies. COTS- based solutions represent an opportunity to do this – but only if properly planned and managed. Election officials, voting system vendors, and testing authorities must recognize the risks and opportunities present in COTS and find ways to mitigate the risks while reaping the benefits.
1 Modified COTS (or MOTS – Modified Off-the-Shelf) refers to an off-the-shelf product that is customized by a commercial vendor to respond to specific requirements of the election community.
2 An integrator (sometimes known as systems integrator) is a person or company that specializes in bringing together component subsystems into a whole and ensuring that those subsystems function together. As noted above, the election administrator may well become the integrator of the future.
3 This work is now integrating with the EAC/NIST/TGDC VVSG standards development process.