Colorado’s Implementation of Risk-Limiting Audits

Oct 30, 2017

By Dwight Shellman and Jennifer Morrell

Colorado is pioneering a new method of auditing elections known as a risk-limiting audit (RLA) that improves the integrity of the election process and sets the standard for other local and state election officials to follow. The promise of this effort can be attributed to a six-year collaboration between state and county election officials, voting system providers, subject matter experts, and election advocates.

Colorado will conduct the first statewide RLA after the November 7 Coordinated Election. Ahead of this effort, election officials worked to understand the concepts behind RLAs and the technology necessary to implement this type of audit. Along the way, new election rules were written, training and best practices developed, and a software tool was created to move RLAs from academic theory to Colorado’s new standard for auditing elections.

A risk-limiting audit is a post-election audit that provides strong statistical evidence that the reported outcome is correct, and a high probability of discovering and correcting a wrong outcome. The “risk limit” is the largest chance that the audit yields strong evidence of a correct outcome when the reported outcome is, in fact, wrong. RLAs conclude in one of two ways.  The audit either (a) stops when it finds strong evidence that the reported outcome is correct, or (b) fails to find strong evidence that the outcome is correct, and evolves into a full hand count of the ballots. 

RLA experts recognize two basic types of risk-limiting audits – comparison audits and ballot polling audits. A comparison audit requires a data file showing how the tabulation system interprets the voter markings on each uniquely identified ballot. This is known as a ballot-level cast vote record (CVR).

A ballot polling audit is less efficient. This method is recommended with older tabulation systems that do not capture or easily export ballot-level CVRs.  In a ballot polling audit, the auditors simply report voter markings in the audited contest from randomly selected ballots until the risk limit is met. A ballot polling audit requires an examination of more ballots than a comparison audit because there is no CVR to compare the auditor’s interpretations against. Fortunately, 54 of Colorado’s 64 counties will use a voting system this year that both tabulates the ballots cast in the election and simultaneously creates a CVR file for export with the proper content and format.

Based on recommendations from a RLA working group, a model for conducting the audits was developed by the Secretary of State’s Elections Division staff, which strikes a balance between specifying high-level objectives without micromanaging how each county achieves them. This is done by having the Secretary of State administer RLAs centrally, while bipartisan audit boards conduct the audit locally. This approach has been codified in Colorado’s election rules.

This centrally administered process also required the development of a tool to compile a large amount of data in a short amount of time (almost 3 million votes in 2016). The Secretary of State (SOS) entered into an agreement with Free & Fair of Portland, Oregon to develop a software application to be hosted in environments owned and operated by the Department of State. The software applications are collectively called the Colorado Risk-Limiting Audit (CORLA) Tool.

County Perspective of Risk Limiting Audits

With a clear timeline and roles and responsibilities defined, the audit process from a county perspective becomes very simple. The ease and success of any audit begins long before the actual audit takes place, and is tied directly to the level of detail in organizing the information beforehand. Risk-limiting audits are no different. The method or the process that local election officials use to organize, track, scan and store ballots is crucial to the audit’s success.

Election staff in Arapahoe County, for example, will focus on three key elements – organizing and tracking ballots, correctly identifying the ballots selected for audit, and creating an electronic ballot manifest for ease and accuracy in retrieving ballots.

The organization process begins by moving ballots from the disorganized state of a ballot box into organized batches of no more than 100 ballots, creating a traceable chain-of-custody all the way back to the moment the batches were created and ensuring any change to the batch count can be explained, i.e. empty envelopes or ballots removed for duplication.

Once mail ballots have been removed from their envelope and secrecy sleeve, and the in-person ballots are reconciled with the e-poll book all ballots get scanned in preparation for adjudication and tabulation. Each batch of scanned ballots is identified by the scanner ID number and sequential batch numbers assigned to that scanner. This will be used to identify ballots selected for auditing and help the audit board locate the ballot. For this reason, it is essential that once ballots go through the scanner, they remain in the same order that they were scanned. Election rules created by the SOS require the original ballot to be audited, so the same order and organization is critical for duplicated ballots.

Ballots selected for audit will be selected at random. This requires local election officials to be able to identify the exact location of a ballot for retrieval. Arapahoe County worked with Dominion Voting Systems to install a mechanical imprinter on every scanner which “stamps” the date and time that the ballot was scanned along with the scanner ID number, the batch number, and the ballot sequence number for that batch. This “stamp,” called a ballot tracking number, will be identified in the CVR and the ballot manifest.

The final element implemented at the county level was creating an electronic ballot manifest to aid in finding and retrieving ballots. The CORLA tool requires a list of every scanned ballot and returns the same list with only the ballots selected for audit.  The electronic manifest tool created by elections staff includes pre-printed labels with the scanner ID number printed at the top. Scanned ballots are placed in an envelope labeled with the same scanner and batch number and placed in a clear container that can hold up to 15 batches. Once the container is full, the pre-printed label is affixed to the front and the bar code is scanned from the label. The bar code triggers the electronic ballot manifest to pre-populate the scanner, batch numbers, and storage container number. This electronic manifest can then be formatted for upload into CORLA.

Once the list is returned with the ballots selected for audit, it can be sorted by storage container, batch and sequence number for easy retrieval and verification. All that remains is to input the interpretation of voter markings into the CORLA tool by the audit board for a comparison to the CVR.

Colorado voters and election advocates eagerly await the outcome of the risk-limiting audit. The EAC Commissioners will also observe part of the RLA in Colorado after the elections in November and will report back on what they observed during the process.

Dwight Shellman is the County Support Manager in the Elections Division of the Colorado Department of State.

At the time of this post, Jennifer Morrell was the Deputy of Elections & Recording in the Arapahoe County Clerk and Recorder’s Office. She is now a consultant leading Democracy Fund's Election Validation Project.

post election

Blog Authors

Content field - Reaction