Analyzing Post-Election Audits.

The document, Draft, VSRA Election Process and Voting Systems Models (Election Operations Assessment), Phase I, does not sufficiently cover post-election audits of election results, especially the significance of post-election audits in reducing risks. In the document, neither the description of the "Election Closeout (e.g., results certified, post‐certification system audit, data archived, voting system returned to storage)", nor the flow chart in section 3.5 on p.74. mentions random post election audits.

The League of Women Voters' Report on Election Auditing observed:

Properly performed audits will guard against both deliberate manipulation of the election and software, hardware or programming problems, since any of these factors could alter the election outcome.

The LWV report also noted that:

About half of all states have laws or regulations and procedures relating to recounts of contested elections, and about one third of the states currently require election audits.

Clearly, post-election audits can play a major role in detecting hardware or software problems, as well as guarding against deliberate manipulation of elections. Furthermore, post-election audits are becoming increasingly commonplace. Therefore, it is imperative that the second stage of the risk assessment analysis closely examine the role of post-election audits in reducing election-related risks. In addition, the study should analyze risks to the post-election audits themselves.

Let's not jump to conclusions.

Need to clarify that the resulting recommendations and standards apply to future voting systems, not voting systems currently in use.

I realize that a number of election officials, almost all of whom are strapped for funds, are worried that recommendations resulting from the risk assessment study might apply to voting systems that they currently use.  They are apprehensive that proposals for hardware, software, or procedural changes could be costly and perhaps difficult to implement.  These are all valid concerns.

However, it is possible that there will be critical recommendations that are relatively easy and inexpensive to implement and that will assist election officials with their very important job of making our elections as secure, accurate, and reliable as possible.  Obviously, when funds are limited, people will have to make trade-offs.  But a blanket insistence that no recommendations or standards be applied to currently deployed voting systems is putting the cart before the horse.  Let's first see what recommendations are forthcoming.

We should not muzzle the study before we even know what the outcome will be.
 

Overview (pg. i):

• The general explanation of the voting system risk assessment in the overview needs to more clearly identify that this is a "looking forward" assessment.  The purpose of this risk assessment as outlined in the RFP Statement of Work is to facilitate the EAC is making decisions relative to the next iteration of the Voluntary Voting System Guidelines (VVSG).  The voting systems currently in use were not tested against and are not subject to the next version of the VVSG, and therefore it should be clearly stated that the recommendations coming out of this assessment do not apply to the voting systems in use today.

• The voting system risk assessment needs to consider that voting systems are part of a process in election administration.  The voting systems should not be considered in an isolated manner.  In order to identify an acceptable level of risk, the entire election process (i.e., storage and maintenance, voting system setup) must be taken into consideration because any part of the process could potentially impact the functionality of the voting systems.  Looking at the voting systems in this manner mirrors the reality of what occurs in the State election jurisdictions.  The assessment should be a tool used by the EAC and other stakeholders to understand the acceptable level of risk to voting systems in the election process and determining how the process needs to mature moving forward to mitigate the risk.      

What is the criteria for determining whether a term is defined in the glossary or in the text?  For example, should the terms and definitions on page 8 be included in the glossary instead of in the text?

Overview (page i)

General comment:  Since primary purpose of project is the assist with the development of the Voluntary Voting System Guidelines (VVSG) and testing protocols by NIST, this should be clearly stated in the Overview.

• Need to clarify that the resulting recommendations and standards apply to future voting systems, not voting systems currently in use

Specific comment:  The EAC’s name is the Election Assistance Commission, not Elections Assistance Commission.

Modeling (page i-ii):  It would be helpful if there was a plain English explanation of bulleted items under the models, diagrams, and flow charts.

Project Scope Discussion (page iii-iv)

• General Comment:  Recognize that administrative procedures vary from state to state and jurisdiction to jurisdiction.  The document should explain that whether the administrative procedures are representative of all jurisdictions or whether they represent a few jurisdictions.  If the latter, the report should describe from which jurisdictions these procedures came. 
• Specific Comments:
  • The statement "While a deficiency in any of these activities may have an impact on how well the voting system functions, it is a fairly low impact" is not necessarily true.  For example, failing to maintain or improperly maintaining a voting system could have significant impact on the functionality of the voting system.
  • Recommend that voting system maintenance and programming electronic voting devices should be included in the model as they have the potential to impact significantly the functionality of the voting system.
  • Recommend that verifying voter identification and eligibility be excluded from the model as it has no impact on "how well the voting system functions."  Voter identification and eligibility has an impact on how well the election process works but is irrelevant to how the voting system functions.  As a result, the following diagrams can be removed from the model: (1) 3-6 Voting Data Flow Diagram; (2) 3-7 Voting (Remote) Data Flow Diagram; (3) 3-9 through 3-12; (4) 3-30 through 3-33; (5) 3-37 Physical Handoff Activity Diagram; (6) 3-38 through 3-39; (7) 3-41 Spoil Ballot (Remote Activity Diagram); (8) 3-44 Ballot Box Accounting Flow Chart; and (9) 3-54 State Accumulation Flow Chart.

Domain Models

• 1-1 Election Domain Model & 1-2 Precinct Domain Model:  Explain what the thick black line dividing the chart means.
• 1-2 Precinct Domain Model: (1) What is the significance of "Rural_Voter" in the "Precinct" box.  (2) What does "PWID" stand for in the "Poll Worker" box?  The acronym is not defined in this chapter or in the glossary.
• 1-3 Ballot Domain Model: (1) Should other types of voting systems (i.e., Internet voting, hand counted paper ballots) also be included with Ballot Marking Devices, Optical Scanner, and DRE? (2) Should "rejected" ballot be included, since spoiled ballot (a ballot that is not counted) is included?  (3) Since provisional ballot is included, should absentee ballots be included as well?  If these three issues are not included, can you please explain why they are not?
• 1-4 Voting Machine Domain Model: (1) "Election Counter" should be added in "Optical Scanner" box.  (2) Should other types of voting systems (i.e., Internet voting, vote by phone) be included?
• 1-5 Person Domain Model: (1) Why is SSN included in the "Citizen" box?  Non-citizens have social security numbers.  (2) Should "Party Affiliation" be included in the "Citizen" box?

State Transition Diagrams

• 2-1 Ballot State Transition Diagram: (1) What does "PWCommit" mean?  I assume that it means that the pollworker casts the abandoned ballot but it might be helpful to define or describe this term. (2) What about rejected ballots?  If this is a model just for polling place voting (which means that rejected ballots would not be included), specify that this diagram is a polling place diagram.
• 2.2 Ballot Venn Diagram: Should rejected ballot be included in the diagram?
• Pages 10-11:  (1) Should these terms and definitions be included in the glossary instead of in the text? (2)
• 2-4 Voting Machine State Transition Diagram: (1) What does "AwtVoter" mean? (2) What do the "e" in the arrowed lines mean? (3) Should other voting methods (i.e., Internet voting, vote by phone) be included?  (4) There is no differentiation between "Pre-Election Mode" and "Election Mode."
• 2-6 Collect Ballots by Touchscreen State Transition Diagram: What does "PageOp" and "NewPage" mean?

Voting System

• 3-3 Ballot Preparation Activity Diagram: Should the process of translating ballots into other languages be included?
• 3.1 Ballot Preparation:  There is a bracketed 1 ([1]) under AllVotableBallotsReady?.  Where is the reference?  (2) Since AllVotableBallotsReady? is listed, should it be somewhere in one of the diagrams?
• 3-5 Voting Activity Diagram: (1) what does the information in the box above the diagram mean?  (2) There can also be lines before check-in.  Should "VoterQueue" be added as well?
• 3-6 Voting Data Flow Diagram and 3-7 Voting (Remote) Data Flow Diagram:  Should rejected ballots be included in both diagrams?  If limited to polling place voting, diagram should specific that limitation.
• 3-8 Ballot Possession Sequence by Voting System: Include a note explaining that VBM(a), VBM(b), and VBM(c) and RE(a) and RE(b) represent different scenarios of vote by mail and remote electronic voting, respectively.
• 3-10 Voter Checkin Data Flow Diagram & 3-12 Authenticate Voter Activity Diagram: Use of SeniorPW seems jurisdiction specific.  A more jurisdiction neutral term would be PW.
• 3-12 Authenticate Voter Activity Diagram: The process if the voter is in the correct location or not will vary greatly by jurisdiction.
• 3-14 One Voter Activity Diagram:  Should the "override" process be incorporated since the voter has the opportunity to override the error message and cast the ballot with an incorrectly marked selection.
• 3-15 One Voter Data Flow Diagram: (1) Should rejected ballots be included in both diagrams?  If limited to polling place voting, diagram should specific that limitation.  (2) Contains jurisdiction specific steps (i.e., pollworker submits ballots, pollworker duplicates ballot, etc.) and voting system specific steps (i.e., with a DRE voting system, the pollworker does not duplicate the ballot). 
• Page 33:  In the 2^nd bullet under "MarkSpoiled," the statement (For e-ballots (DRE) the spoiled ballot (cancelled ballot) is electronically marked (recorded in a log entry) is not the case in all jurisdictions.  If a ballot is cancelled, the selections are not electronically saved.
• Page 35: In the 1^st bullet under "CommitAttempt," define what "PBHC" stands for.  In the 2^nd bullet, the sentence reads that there are overvotes with DRE voting systems; DRE voting systems prevent overvotes.
• Page 36: Under "CommitAttempt," should vote by mail or Internet voting also be included?  Other types of voting systems and methods are included.  Under "CommitBallot," should other types of voting systems and methods be included (i.e., vote by Internet, vote by mail, vote by phone)
• 3-20 Commit Ballot Activity Diagram: What does "HCICommit" stand for? An explanation is on page 40 – three pages after the term is used.  Should the explanation be in the glossary?
• 3-21 Commit Ballot Data Flow Diagram: What does "HCISelection" mean?
• Page 45: Add a period at the end of the sentence under "VoterWishesToOverride." 
• 3-28 Keypad or Voice Selection Activity Diagram: The diagram and explanations of terms used in the term indicate that it only describes vote by phone (e.g., disconnect).  There are other voting systems that use keypads (i.e., DRE with keypad and headsets available for voters with visual impairments).  If this diagram is limited to vote by phone, it should say that.  Should there be a comparable diagram for other voting systems with keypad or voice selections?
• 3-35 One Voter (Remote) Activity Diagram: (1) The heading of the diagram should be OneVoter(R), not OneVote(R). (2) With remote voting, is there no way to correct the ballot (rather than spoil the ballot)?
• Page 55: The reference to Maryland’s use of an outer envelope does not accurately describe Maryland’s absentee envelope.  It is correct that we use only one envelope but the oath that the absentee voter signs is not removable; it is printed on the back of the only envelope and covered by a removable flap (nothing is printed on the removable flap).
• 3-40 Seal Envelope Activity Diagram: This diagram is the same as 3-39 Seal Envelope Activity Diagram.  Should it be included twice?
• Page 58: Under "SpoilBallot(R)," the sentence starting with "Historically . . ." and including the bullets is not a complete sentence.  Also, is this sentence necessary with the paragraph at the top of page 59.
• Page 75 and 3-55 Recount Flow Chart: The definition of a recount includes only the retabulation of votes of accepted ballots on the contest being recount, but the flow chart includes a contest audit as part of the recount.  Generally, a recount does not include a contest audit; it is, as the definition states, a retabulation of accepted votes.   The contest audit should be removed from the flow chart.
• Page 77: The definition of "PostCertificationAudit" does not reflect the primary purpose of a post-election audit.  A post-certification audit primarily enables election officials to assess whether election-related procedures were followed; assessing the effectiveness and efficiency of the procedures might be the consequence of the audit but it is not the primary purpose.

As a follow-up and expansion to Linda Lamone’s comment on Voting System, 3-28 (Keypad or Voice Selection Activity Diagram) there appears to be confusion and inconsistency in the analysis of Vote-By-Phone systems.  A phone voting system is not a separate approach to voting.  It provides the same audio/tactile voter interface that is provided by DRE’s and ballot marking devices.  It can produce a variety of different vote records:  an electronic vote record (like a DRE); a marked optical scan ballot (like a ballot marking device); and/or another form of paper vote record that can be counted by hand or by bar code.  The paper record can be produced and counted at the polling place (like PCOS) or at a central site (like CCOS).  In some cases, Vote-By-Phone systems produce a paper ballot that must be "translated" by an election official into a "machine countable" form which greatly increases the chance for error and can create privacy issues.  The current draft does not address all of these variations and does not acknowledge the overlap and relationship of Vote-By-Phone systems with other types of voting systems.  A number of the current diagrams (such as 3-20 - Commit Ballot Activity and 3-21 -Commit Ballot Data Flow) need to be revised to address these issues.  Since Vote-By-Phone systems are used in some jurisdictions as the accessible voting system, it is critical for the system to be accurately analyzed to support development of appropriate standards that ensure delivery of a secure, independent and secret ballot for voters with disabilities. 

Project Scope Discussion (pgs. iii-iv):

• There are risks associated with voting systems and then there are also risks in the election process and procedures.  These two categories of risks must be analyzed together to determine an overall acceptable level of risk.  As such, the scope of this project should be expanded to include election administration functions such as voting system storage and maintenance, and voting system set up and validation, particularly programming the voting devices and logic and accuracy testing.  In addition, the procedures at the polling places should be included in the scope as well as consideration of the training of poll workers.  All of these elements are germane to discussion of voting system risks and any one of these elements could have an impact on the functionality of the voting systems.  The project scope indicates that vote tabulation is included.  This is important so that movement of the ballot through the entire process is considered.               

Modeling (pg. i-ii):

• As we understand it, the target audience for the voting system risk assessment is the EAC and the entire election community.  As such, this document should provide some easier to understand explanations and aides in deciphering the Unified Modeling Language (UML).  For example, it may be helpful to include a fairly basic chart noting the common multiplicities used (e.g., 0..1 means no instances, or one instance, etc.).  You may also consider expanding upon the UML explanation in Section 5, but doing so in much more simple terms.  Examples would probably be most effective in helping the overall target audience understand the concepts in UML.             

I agree with the statement that the document needs to be more transparent in regard to the "looking forward" concept.  Those states that followed the HAVA mandate to replace older voting systems with new technologies and accessible systems, should not be subject to new procedures or technologies not available at the time the original purchase/implantation decisions were made.  I also believe that all processes concerning the voting systems should be included in the risk assessment, such as storage, maintenance, setup, post and/or pre election audits or public tests, etc.).  Further recognition that this document serves as guidance and not a "one size fits all" approach or mandate to voting systems standards for all jurisdictions is necessary.

In regard to provisional ballots, there should be a process flow indicted that these ballots can be rejected.  In Indiana, provisional voting is generally a last resort for voters who do not appear on the precinct poll list and cannot be located in any other precinct within the jurisdiction (county).  Poll workers have been trained to first determine where the voter should be before casting a provisional ballot [the entire ballot is rejected if the voter is not eligible, or votes provisionally in the wrong precinct]. 

To begin I'd like to the thank the VSS Committee and Commission personnel for their hard work in preparation for this meeting. 

Keith Cunningham showed great leadership in keeping this on track.  The committee members:  Bill Cowles, Jim Dickson, Linda Lamone, Doug Lewis, Wendy Noren, David Orr, Helen Purcell, Todd Rokita and Barbara Simons spent many, many hours reviewing materials and participating in teleconferences.  Thank you.

Commissioner Hillman, Maisha Leek and Matt Masterson, thank you very much for providing the resources and organization to get everyone to this meeting. 

The committee recommendations are sound and on point.  I especially agree with the limitation of the scope of this project.  The day may come when EAC or some other organization will need to provide risk assessment protocols for election officials and others, but in this realm we are not there yet.  The priority is to apply this material to the processes that define risk in the voting system certification process.  Once it is clear what the guidelines accept as risks then there may be a necessity to test that at the state and local levels. 

I still wonder what environment all of this will work within.  State laws and procedures along with local procedures are the environment in which risk should be assessed.  Again, I'm a little baffled at how this will work out.  This being an early phase of the project, I know that this will be addressed as the project continues.

Again, thanks to all who participated in preparation for this meeting and in the meeting itself.  It may be that further discussion of this will be necessary in June.

The US Election Assistance Commission cannot emphasize enough the need for sufficient security without adding inefficiencies, difficulties in use or high costs.  The large majority of election officials are from very small jurisdictions and have neither additional staff or money to implement  complex or restrictive requirements.  Additionally, precinct style equipment is implemented by election workers, who are usually not attorneys or computer technicians.  This is mentioned in the overview but must be addressed in the project presentations. 

Additionally, the project group does need to understand that not all security is electronic in nature.  How equipment is used, tested and maintained is sometimes the most secure with the oversite of election workers / the public, and can be verified with something as simple as a numbered Poll Book.  The project awardees need to demonstrate a clear understanding of how election equipment is used.

With that being said, security is the top priority in maintaining the integrity of the election.   I am a little concerned that Polling Place Administration, Voting System Deployment and Voting System Set Up and Validation are not included.  I do not know how you can separate these subjects from the "procedures and equipment that move the ballot through the electoral process" in precinct tabulation equipment or direct recording equipment, both of which is implemented at the voting place.  Voting System Storage and maintenance is also not included, yet a secure lock-down system is needed in any system when it is not in use.  Meanwhile, Election Definition is included, yet surely the security of a system is just as important whether it is for a federal or a township election. 

Finally, it is always good to review voting system security so that election officials and law makers can respond to vulnerabilities for the protection of the integrity of the election.

Many of the comments posted by Linda Lamone in terms of identifying some potential missing aspects of the diagrams or asking for clarification as to why certain concepts were excluded are right on point.  We had many of the same questions as Linda.  However, for the sake of not repeating comments already posted, we will not ask the questions again, but only ask that you please give careful consideration to Linda's well thought out comments and/or questions.  We will mention though that it is important to remember that the entire election community should be able to make sense of this document and in some instances there are acronyms/information that could be interpreted in different ways (e.g., "PWCommit"), or lack definition at all (e.g., "PWID").          

I would like to thank the VSS Committee and the EAC for their work on this project.  I look forward to further discussion on this matter at the Board's meeting in June. 

The comment made by Barbara regarding post election audits goes to the heart of what we have been concerned about with this study.  The information and models presented are not clear about the scope of this project and because of that will generate these kinds of criticisms of what is in and what is out.  It's not that post election audits are a problem - I like them and have been doing them for 20 years - it's that they are an administrative issue not an equipment issue. 

As several have stated, it was our initial understanding that the study was requested by NIST for developing risk assessment models for equipment not administration.  I don't see how testing scripts for equipment can be developed if this assessment includes administrative/legal issues that vary considerably from state to state.   Barbara's comments are not unique nor are they off base under an administrative risk assessment - there are numerous areas in the models that point to administrative issues - see ballot where issues such as layout, location of instructions etc are listed.

Never having seen the precursors to risk assessments - which is what we are evaluating - then I'm not sure if the items included are necessary for the development of the equipment component or are leading to an overall election management risk assessment tool - which was not the scope.

On to details:

1.3 Ballot

Why is "Receipt" designated on here.  I'm assuming you don't mean a receipt for the ballot.  Can you be clearer on this.

You designate provisional ballots.  Are you intending to look at other kinds of ballots (FWAB or other forms of absentees)

 What is the purpose of "Rural Voter" designation under Precinct? 

Under contest you have vote for n of n.  What about IRV voting as some jurisdictions are now going that route

Should any consideration be given to issues, retention races etc - restricted ballot to contests

You list election counter and life cycle counter of DRE but not on optical scan - any reason?

Also ballot image is attached to DRE but some Optical Scan systems are based on the collection of the image.

I've asked before - what is the meaning of the numbers on the chart (1 1.0 11 etc) - also the *

Should some consideration be given to multiple page paper ballots - particularly as they relate to straight party voting and the party candidates run to second page of ballot

Why do some things have + by them and others have -

Where would you place over voted ballots or mismarked ballots that are not spoiled.

I have a tornado going on will save this in pieces so I don't lose it.

After reviewing Phase 1 of the Voting System Risk Assessment, I concur with the four recommendations of the Voting System Standards Committee. The report is an appropriate starting point for revisions to the Voluntary Voting System Guidelines. Including instructions and explanations of the various graphs and models, and utilizing uniform election terminology will allow for a broader dissemination of the report’s eventual conclusions.

Comments from the Open Source Digital Voting Foundation.

1. We believe it would be useful and productive if the Project formerly known as "VSRA" were able to produce a comparison of challenges, issues, and even risks of different voting methods (e.g. PBHC, all DRE, all PCOS mixes) rather than specific voting system products.
2. These comparisons suggested in item 1 (above) will be especially helpful if the they are based on estimates from people who are experienced in elections operations.
3. As the term is used herein, "estimates" in item 2 (above) refers to educated guesses at how likely a particular issue, challenge, or risk is, or the relative magnitude of downside consequences.
4. Further to this point, we believe it would be of considerable value if the comparisons were the result of a documented methodology that other Researchers could use to make alternative comparisons based on different estimates of the use of the same voting methods.
5. We emphasize "Researchers" in this context to indicate that we do not assume that election officials, themselves, would perform this work.
6. We emphasize "methodology" in this context to indicate those processes (such as CCOS for VBM, combined with DRE-VVPAT for precinct voting), but without referring to any particular vendor's products (CCOS devices, DREs, etc.)
7. We believe these results would [a] be valuable as guidance, going forward to those who are developing voting system technology, and [b] could benefit their product development and feature/function sets from some specific explanation of operational risk factors.
8. Finally, we believe this approach could greatly enhance the current anecdotal statements about existing products creating or exacerbating risks, which are not "actionable" for voting technology developers who would otherwise seek to mitigate or remove such risks.

1.4

See comments for 1.3 on counters, votable ballot only containing contests not issues and other areas that are on this chart and on 1.3

You have removable media - what about modem/wireless transfers

Also on removable media - what about lifespan and battery integrety

The polling place stuff is a perfect example as to why there is so much confusion about this document - what is its purpose on this chart particularly issues such as parking and hours open.  Why is this necessary for developing equipment standards and testing scripts.  Including something like this is why Barbara is legitimately asking about administrative procedures.

Under voting machine you list "known issues" - not sure how that relates to testing scripts or standards. 

1.5

I have to admit I don't understand this one but here are some questions/issues

I think its been brought up before that citizen is not always a requirement.

Not all voters have to show voter id

What is SSN on the chart for

You may want to add credentials to poll watchers.  Not all poll watchers are citizens we have international observers

If you have poll watchers why not media

2.1 Ballot Transition

This has some explantion on it that helps - do the same thing other charts

You state that Provisionals go through same states as regular but what about those areas that only count parts of the Provisional (i.e. voter has moved and only part of the ballot is valid).  In addition the provisional voter doesn't have the second chance voting opportunitiy for mistakes on ballot in those jurisdictions that put ballots in affidavit for later processing.

Do you want to address rejected write-ins - some states have restrictions on write-ins (i.e. certified write-ins)

2.2 Voter

I think we've previously discussed the problems with your definitions of legal voter eligible voter and qualified voted.  They are backwards from the definitions in my state i.e. we go from eligilbe voter to legal voter to qualified to vote in an eleciton.

Can't see how this will relate to equipment security testing.

Signed in vs not signed in - our provisional voters don't "sign in".  Also we have some who "sign in" then we discover they've moved so they have to void the sign in and send them to correct new polling place.

2.3 Voting Machine

I'm having a little trouble following this arrows - seems there should be a couple of  power offs and and power ons before it is open for voting (i.e. testing ESP).  Even without that I don't see a power up after adding the ESP and before open for voting.

Also on second page - scanning.  Most of the OS systems do not create an image from the scan - they allocate votes from reading marked oval or arrow but do not collect an image in the traditional sense.  At least one system does collect the image so you may need to differentiate.  Why does one of the error detected go directly to close out.  On most systems the count is still updated (i.e. ballots counted also counted as an overvote or undervote)

On DRE - not sure the write to memory is in correct spot.  In addition there are multiple writes to multiple memory locations of most systems

Also there is generally a review page and warnings issued for non voted races.

3.1 Voting system

You have post certification audit - some have pre certification audit.  Also - provisionals are processed after precinct closeout.  Also - do you need to fit absentee accumulation in here someplace.

3.2 - you use the term artifacts please define what you include in this (couldn't find in glossary)

You list precinct closeout.  What about early voting sites.  Also central counts and absentees, provisionals come between precinct closeout and canvass

3.2 Voting you say there are only 2 possible steps 1) the voter is authenticated and receives a a ballot  2) The voter marks and commits the ballot

 Other possible steps:

  1)The voter is not authenticated and receives a provisional ballot

 2)  The voter marks the provisional ballot but is many cases that ballot is not commited until the voter can later be authenticated

Also:

Some voters do not commit the ballot (they don't understand when the error message rejects an overvote) that ballot may later be committed by the pollworker.  Same with some fleeing voters on DRE.

also in some cases there is additional authentication for absentee voters (i.e. notary signature verifcation etc).  The voter was authenticated and provided a ballot, voter marked the ballot and delivered it but ballot was not committed or accepted for lack of signature/notarization.

(this also applies to chart on 3-7)

3.6 Voting Dataflow diagram - not all systems have ballot tokens.

3.12 Voter Authentication Diagram

When is a provisional not appropriate if the vote is not allowed to vote? Whether or not the voter is eligible, legal, etc.  I think federal law requires that we give them a provisional if the voter asserts they are eligible.

Also, where is primary ballot selection accounted for.  In some states it occurs at voter authentication but others the voter gets to choose which ballot - maybe need a subunit of select votable ballot. 

3.14 One voter activity

In some cases when ballot is not correctly marked voter chooses to commit ballot anyway rather than spoil ballot.

3.15

Ballot marked with error.  You have pollworker making corrections by duplicating ballot.  In our case those are handled centrally to insure compliance with state regs on what constitutes a vote.

3.16

You have page backward after review - some system they just select the race not have to page back

Many of these things are addressed in later diagrams.  Example - I mentioned the option of absetee not counting because of notary as one of possible scenario when you said there were only 2 possible steps to voting - later you have that addressed in the absentee section.

In most cases the later diagrams are pretty complete.  This initial diagrams are confusing as the don't appear to have much rhyme or reason for what is on or off.

I have to concurr with the detailed notes that Linda Lemone submitted on the diagrams.

I may have some minor additions to those I will submit informally.

I also agree with Elizabeth Ensley's comments on security in storage/movement of equipment. If you've got other polling place issues then why leave this out.

A Post-Election Audit is a critical tool for threat mitigation.

Wendy has expressed the concern that post-election audits are administrative, as opposed to equipment-related.  She then says:
 

I don't see how testing scripts for equipment can be developed if this assessment includes administrative/legal issues that vary considerably from state to state.

In fact, developing testing scripts is not the primary goal of the study.  As listed in the Statement of Work, some of the key goals are to identify risks and to perform "risk assessment of the potential harms and possible mitigations for these threats".   I'm sure these are goals with which we all agree.

When dealing with computerized voting systems, the best tool for the mitigation of threats coming from software bugs or malicious code is a post-election audit of voter-verified paper ballots (or ballot images).  In particular, if there are good chain of custody records and a good post election audit protocol, we can essentially guarantee that the election outcomes are correct in spite of the possibility of arbitrary bugs and (most kinds of) malicious code.

It is precisely because we are using equipment that relies on computers for key elements of election tabulation, and sometimes even casting ballots, that post-election audits cannot be separated from any reasonable risk analysis of voting equipment.  Post-election audits are far and away the best tool at our disposal for detecting problems with or verifying the accuracy of machine generated tabulations.