1 1 U.S. ELECTION ASSISTANCE COMMISSION 2 * * * 3 PUBLIC HEARING AGENDA 4 5 6 U.S. Environmental 7 Protection Agency HQ 8 1200 Pennsylvania Ave., NW 9 Washington, DC 10 11 Wednesday, May 5, 2004 12 9:01 a.m. 13 14 15 Meeting of the U.S. Election Asistance 16 Commission, was held on Wednesday, May 5, 2004, at 17 1200 Pennsylvania Ave., NW, commencing at 1:30 p.m., 18 Chairman Soaries, presiding. 19 20 21 22 2 1 P R O C E E D I N G S 2 (9:01 a.m.) 3 CHAIRMAN SOARIES: Good morning everybody. 4 (No response.) 5 CHAIRMAN SOARIES: I'm Baptist. Good 6 morning, everybody. 7 AUDIENCE: Good morning. 8 CHAIRMAN SOARIES: Welcome all of you to 9 this public hearing sponsored by the United States 10 Election Assistance Commission. My name is DeForest 11 Soaries, Jr., and I am the Chairman of this new 12 Federal Agency for this year. 13 I would like to thank the Environmental 14 Protection Agency for hosting us and providing for us 15 accommodations and water and all of the hospitality 16 that they've provided. Their staff has been very 17 helpful, and we are grateful to them for this 18 facility. 19 I also would like to thank in advance all 20 of those who have come to serve on panels today. We 21 have great minds and outstanding talent coming to 22 help us understand more about this issue of 3 1 electronic voting, and people have come from near and 2 far and we are the beneficiaries of having been 3 rejected by no one. 4 We have the unenviable task of telling 5 quite a few people that we just did not have space 6 for all of those who sought to testify orally, but we 7 have received written testimony from scores of people 8 which will be a part of our focus as we issue our 9 report and a part of our consideration as we 10 deliberate this important subject. 11 I would also like to thank the very small 12 EAC staff for the work that they did. Many of them 13 have not slept recently, and we would like to 14 recognize them for their work. 15 Four-and-a-half months ago the four 16 Americans that you see seated before you embarked 17 upon this mission called the Election Assistance 18 Commission. 19 We knew that there were challenges that we 20 faced. There were realities that we inherited due to 21 the timing of our appointment and the nature of our 22 work, but greater than the challenges were the 4 1 opportunities to pursue this national consensus that 2 every voter matters, and that every vote counts. 3 So for four-and-a-half months we have been 4 working hard on administrative startup activities. 5 We've been working hard to facilitate the 6 distribution of federal funds that were made possible 7 by the Help America Vote Act to the States. 8 We've been busy visiting primary 9 elections. We've worked hard to meet with various 10 groups whose perspective is important as we do our 11 work and shape our own internal organizational 12 structure that we might be effective in maximizing 13 the use of the resources that we have. 14 We started out work by the publishing of 15 the States Plans required under HAVA for the release 16 of $2.3 billion that has yet to be released in 17 Requirements' Payments. Shortly those funds will be 18 released, which created some sense of urgency as we 19 began hearing from States. 20 It was clear that the use of those funds 21 would be subject to the kind of guidelines that we 22 issued. Much of our work is subject to a process 5 1 that is rather long-term within the scope of what we 2 know is a challenge for this November. But much of 3 what we do can have an impact on this November's 4 election, and that brings us here today. 5 We know, all of us know that voting in 6 America has evolved since the founding of this 7 Democracy. Not only has the Constitution been 8 amended to expand the persons who have the right to 9 vote, but also the manner in which we have voted has 10 changed over the last 200 years. 11 Early in the founding of our Democracy a 12 few men would gather downtown and would verbalize the 13 candidate of their choice, and that was an election. 14 There were times when each political party 15 printed the ballots, and you knew which party you 16 were voting for by the color of the ballot. And 17 there were other times when people would just write 18 their names on a book. 19 The concept of privacy in voting, the 20 secret ballot, emerged quickly as the standard for 21 this country. And the way we vote is what brings us 22 here today. Our commitment to universal suffrage is 6 1 juxtaposed to the technology that we now use to case 2 our private ballot. 3 And so the Election Assistance Commission 4 was formed in large measure in response to the issues 5 that really became front and center in November of 6 2000. 7 This Commission has responsibilities that 8 are well articulated in the Help America Vote Act, 9 but what is not in the Help America Vote Act is what 10 I'd like to describe so that you will understand more 11 about who we are. 12 We came together after having been 13 appointed by the President December 13th, and 14 immediately we made commitments that we hope are 15 transparent and self-evident in what we do today and 16 in the follow years. 17 The first commitment we've made is to be a 18 bipartisan commission in function and not just in 19 name. We are two Democrats and two Republicans, but 20 we believe in our hearts that the issue for which we 21 are responsible is so important to the country that 22 it is incumbent upon us to leave our partisan 7 1 identities and personal philosophies at the door. 2 From day one we have maintained a 3 bipartisan spirit. As one of my colleagues will say, 4 if you look in on our discussions it would be 5 difficult if not impossible to determine who was a 6 Republican and who was a Democrat. 7 There is a time when partisan differences 8 are healthy for the country, but there is also a time 9 when certain issues are urgent to the extent that 10 partisan differences should not stop us from making 11 progress. That is a commitment of this Commission, 12 and we are going to urge not only the panelists but 13 the people with whom we work after today to attempt 14 to rise to level of bipartisan spirit. 15 The second commitment we've made is to 16 move by consensus. We try not to lobby each other 17 and broker deals, but rather form consensus through a 18 deliberative process. In that process we attempt to 19 be civil in our tone. We attempt to be conciliatory 20 in our outcomes. We refrain from any personal 21 attacks. And we try to stay open-minded. 22 I described that hoping that those who are 8 1 on panels today will respect the fact that that's the 2 way we operate, and that is the kind of hearing we 3 would like to manage. 4 As Chair, I will try my best today to keep 5 us not only on time but to keep us within the 6 framework of civility. 7 The final commitment is to results. This 8 hearing is not an academic exercise, although we have 9 some great academicians. Rather, this hearing is 10 strategically called today six months prior to the 11 November election aimed at concrete actions that we 12 can take to not only, as I said, fulfill our long- 13 term mandates but also to help America vote in 14 November of 2004. 15 And so we are honored today to have great 16 minds, experienced professionals, and committed 17 advocates. There will be a tension between certain 18 views, but sometimes tension can produce healthy 19 outcomes if that tension is managed and articulated 20 in a positive way. 21 And so we are thrilled that you have come. 22 We have urgent business to do, and we are going to 9 1 ask each of you to either participate or observe with 2 a certain set of ground rules. 3 In the first instance I would like to ask 4 that everyone turn off their electronic devices. 5 There should be no computers in the room, but if 6 there are any other electronic devices: beepers, 7 watches, Blackberry, Blueberry-- 8 (Laughter.) 9 CHAIRMAN SOARIES: --because it will be 10 easier to proceed without that. 11 We are asking each panelist to make a 12 short opening statement. I will remind them that the 13 panelists will give an opening statement for seven 14 minutes, and then that leaves time for the four 15 Commissioners to ask questions. 16 I will try to keep us on track. Each 17 Commissioner will lead the questioning for a panel, 18 and then after that lead questioning then each 19 Commissioner will have a chance to ask a few 20 questions after that lead Commissioner. 21 I want to ask the audience not to make any 22 demonstrations of support--this is not a pep rally; 10 1 or against--this is not a protest. We would like the 2 audience to be careful to conduct itself in a manner 3 that coheres with the civility that we are attempting 4 to portray. 5 I would like to thank the media for taking 6 this issue seriously because our experience is that 7 the country cares about this matter of voting, and I 8 hope that you will find today as interesting as we 9 intend to find it because as a result of what we 10 learn we will craft our work to take actions that 11 will support all Americans as we prepare for a new 12 Presidential Election. 13 I would like now to invite my colleagues 14 to give an opening statement, after which I will 15 introduce our first presenter. 16 Vice Chair Hillman. 17 VICE CHAIR HILLMAN: Good morning. I join 18 my colleagues in thanking you for finding the 19 commitment and time to be with us. We can only 20 benefit by the input of such a diverse group of 21 people. 22 We welcome your input, and your presence 11 1 here motivates and energizes us. So on the one hand 2 I apologize to the people who are not able to find 3 seating. It is so hard to know when you hold a 4 hearing like this, especially for the first time, how 5 many people will really be interested enough to be 6 here. 7 On the other hand, it is a terrific 8 showing for us. And as I said, it energizes and 9 motivates us. 10 We, as the Chairman said, are very pleased 11 to have been able to come this far in the short 12 period of time that we've been assembled with the 13 many things that we have had to do, but our 14 commitment is to make certain that we move as quickly 15 as possible to meet the mandates of the law, to 16 fulfill our responsibilities and to move our mission 17 forward. 18 And so I again thank you for being here 19 and look forward to your input not only today but in 20 the months and years to follow. 21 Thank you. 22 CHAIRMAN SOARIES: Commissioner 12 1 DeGregorio. 2 COMMISSIONER DeGREGORIO: Thank you, Mr. 3 Chairman. It is indeed an honor to be here and to be 4 a part of this very important hearing. 5 I served for eight years as an election 6 official in St. Louis County, Missouri, and I bring 7 that perspective to this Commission. Back in 1990 I 8 had the closest Congressional rate in the country. 9 51 votes separated the winner or loser and it was a 10 punch card system. So I have been through a lot of 11 experiences because of that. 12 For eight years after I served as Director 13 of Elections I worked overseas in 15 countries and 14 advised Russians, and Indonesians, and others on how 15 to conduct their elections. 16 I was asked to serve on this Commission, 17 and as I saw the past few months the discussion 18 evolve in the country about the security of systems, 19 the use of systems, it became very apparent to me 20 that this Commission needed to have a hearing such as 21 this, and I was very pleased when my fellow 22 Commissioners and the Chairman, suggested that we do 13 1 something just like this. 2 It is important in any democracy that 3 there be freedom of speech and honest debate and 4 informative debate over issues such as this. I hope 5 that this hearing encourages a civilized debate. 6 When I was first appointed to this 7 Commission, there was a web site someone put out that 8 I had worked for the CIA overseas, which was not 9 true. But then I find in the past few weeks and 10 months that this same person is on CBS News, on CNN, 11 and quoted by The New York Times, and that does 12 concern me because I hope that in the discussion of 13 this very important issue that people stick to the 14 proven facts; that they give us informed opinions; 15 and that we stay away from rancor and personal 16 attacks and partisanship. Because the Nation is not 17 served by division, it is served by an honest 18 discussion. 19 I so hope the debate is civilized for a 20 very important reason, because we want to encourage 21 people to participate in our election process. We 22 want to have the largest turnout in American history 14 1 in November, and I hope the discussion of these 2 important issues is at a level that encourages people 3 to participate and doesn't discourage people from 4 participating because our Democracy will not be 5 served if people don't come to the polls. 6 Thank you, Mr. Chairman. 7 CHAIRMAN SOARIES: Commissioner Martinez. 8 COMMISSIONER MARTINEZ: Thank you, Mr. 9 Chairman. 10 Through the passage of the Help America 11 Vote Act of 2002 and our subsequent confirmation and 12 appointment, the U.S. Election Assistance Commission 13 was created to assist in the administration of 14 Federal Elections, and to otherwise provide 15 assistance for certain Federal Election laws and 16 programs. 17 Moreover, the EAC is to establish minimum 18 standards for election equipment, and to act as a 19 national clearinghouse with regard to Federal 20 Election administration. 21 It is this function of national 22 clearinghouse, Mr. Chairman, that I want to focus on 15 1 for just one minute this morning. 2 I think the big picture intent of what we 3 are trying to accomplish with this hearing today is 4 worth reiterating. While roughly 29 percent of 5 registered voters will be voting in November 2004 6 with electronic voting machines--and we will hear 7 some very compelling statistics from Mr. Brace, our 8 first panelist--and some 30 percent of registered 9 voters will be using punch card and lever machines, 10 and another 32 percent will be using optical scan 11 machines, one of the primary purposes of this hearing 12 is to begin gathering important information on the 13 use, the security, and the reliability of all voting 14 systems to be used in this coming November's 15 election. 16 There are no doubt unique challenges with 17 regard to each voting system, and unique challenges 18 we will hear today with regard to the use of DREs, 19 but it is important I think for our audience to 20 understand--our audience here today and the broader 21 audience through the media--to understand that this 22 Commission takes its role as a national clearinghouse 16 1 very seriously. 2 A significant step in that direction is to 3 produce timely Best Practices' guidance to states and 4 local governments regarding the use, the security, 5 and the reliability of all voting systems, including 6 optical scan, punch card, lever, DREs, and paper 7 ballots. 8 Releasing this Best Practices' guidance 9 before November in a timely fashion will in my view, 10 Mr. Chairman, contribute positively toward the most 11 fundamental task we have as a Commission. That is, 12 to ensure that the American public has full 13 confidence in the administration of our Federal 14 Elections. 15 With that, I look forward to the 16 discussion. Thank you. 17 CHAIRMAN SOARIES: Thank you, 18 Commissioners. Much of what we know about who votes 19 on what kind of voting device in this country we know 20 because of the work of our next speaker. We are 21 pleased to have as our opening presenter who will 22 give us an overview on electronic voting the 17 1 President of Election Data Services, our friend Mr. 2 Kim Brace. 3 PRESENTATION OF KIM BRACE, PRESIDENT 4 ELECTION DATA SERVICES 5 MR. BRACE: Thank you, Mr. Chairman. 6 Commissioners, it's a pleasure to be here this 7 morning, and I certainly don't envy your task ahead 8 of you in-- 9 AUDIENCE MEMBERS: Can't hear. 10 MR. BRACE: Yes. My name is Kim Brace, B- 11 R-A-C-E. I'm President of Election Data Services. 12 We are a provider of elections' information around 13 the country, and we have compiled information for the 14 past 30 years in terms of what types of voting 15 systems have been used around the Nation. 16 I have a full statement that I would like 17 to have entered into the record, and I will summarize 18 a couple of key points for your benefit and try to 19 keep us on track of the timetable. 20 In terms of the history of voting systems, 21 this country has had a long history, starting off 22 with paper ballots, as the Chairman mentioned. As 18 1 the country grew and became more urbanized, the task 2 of counting ballots, paper ballots, took longer. 3 With the industrial revolution, a mechanical way was 4 found to produce almost instantaneous election 5 results, the lever machine. 6 Lever machines were invented in 1890, and 7 their use in the elections grew over the next 70 8 years. It is interesting to note, in light of the 9 current controversy that we have over electronic 10 voting systems, that for those 70 years voters were 11 not receiving nor were election officials counting 12 physical ballots on lever machines. 13 Now precincts tended to be smaller in size 14 at that point in time because of the high cost of 15 lever machines, but by the middle of the 20th Century 16 the main source of polling place judges, housewives, 17 had begun moving into the workforce. As a result, 18 this loss in manpower, or womanpower, election 19 officials looked to cutting the overall number of 20 precincts and increasing the size of the remaining 21 polling places. 22 Punch-card voting systems, first 19 1 introduced in 1964, were a popular solution to this 2 problem. These were mainly used in urban and 3 suburban communities around this country, but in the 4 rural parts of this country they looked towards 5 continuing to use paper ballots but find an easier 6 way of tallying those paper ballots. 7 This led to the development of the optical 8 scan systems in the 1970s. With the advent of 9 computers and the need to replace the aging lever 10 machines, the 1970s also found the introduction of 11 the electronic voting systems. Early electronic 12 voting systems looked much like lever machines with 13 pushbuttons replacing levers on a large panel. 14 Newer DREs resembling ATM machines had 15 touch screen panels and key pads for entering write- 16 in votes. Voter preferences went directly into the 17 electronic storage usually without a paper record of 18 the voter's intent. 19 As I indicated, we have kept track of what 20 kind of voting system is used around the country. We 21 started in 1980. In 1980 we found just two 22 electronic voting systems in use at that time, the 20 1 Video Voter and Votronics in use in just seven 2 counties of this Nation. They accounted for just one 3 percent of the registered voters nationwide. 4 Our most recent survey that we have done 5 for the 2004 projected election shows that the number 6 of counties using electronic voting systems has grown 7 to 675 counties in this country. 8 These counties, located in more than half 9 of the states, amount for almost 50 million 10 registered voters, or 30 percent of overall 11 registered voters. 12 Our surveys look at a lot of different 13 voting systems, and when one looks at those and 14 analyzes the comparison of the percent of registered 15 voters, or the percent of counties, one finds 16 differences that are significant in terms of the size 17 of jurisdiction. 18 Right now we're looking at slightly more 19 than 48 million registered voters who are expected to 20 cast ballots this fall on an electronic system, 21 compared to 53 million that will use optical scan 22 systems, and 22 million that would still use some 21 1 form of punch cards. 2 About the same number of voters, 22 3 million, will use lever machines, while about 1 4 million voters will still use paper ballots. Voters 5 using paper ballots represent only two-thirds of one 6 percent of all registered voters in the country. 7 In 1980, over 1200 counties or 41 percent 8 of the counties used paper ballots. We have produced 9 in the statement, of which we have copies on the back 10 table, maps of both what the country looked like in 11 1980 as well as what the country looks like today in 12 2004. 13 Our information and our research indicates 14 that for 2004, while there have been a lot of changes 15 going on in the last four years, upwards of 74 16 percent of the voters in this country will use the 17 same type of voting system that was used in November 18 of 2000. 19 Now besides being the sole repository of 20 historical information on voting systems around the 21 country, we provide information for many of the 22 academic surveys and studies that you heard about 22 1 over the last three years. We also compile and 2 collect voting statistics which are allowed to be 3 pared to the voting information and allow people to 4 come up with what people have referred to as "error 5 rates" in different types of voting systems. 6 I believe the use of the term "error 7 rates" is a misnomer. Because many people have 8 assumed that when people go to the polls they will 9 vote for all offices on the ballot, or at least the 10 offices at the top of the ballot. 11 Empirical evidence, however, shows that 12 neither of these assumptions are correct. 13 14 15 16 17 18 19 20 21 22 23 1 Academic studies have shown that people 2 experience ballot fatigue as they move down the 3 ballot and don't vote for everyone. This phenomenon 4 is called many things, but I refer to is as "drop 5 off". Drop off is roughly equivalent to what others 6 say is the residual vote measures that you may have 7 heard about in recent studies. 8 As I indicated, we've looked at and 9 compiled and we did a study for the Congressional 10 Research Service going back to 1948 that looks at 11 drop off across the country in every county of the 12 Nation. And there is a summary table in the back of 13 my statement for you, but overall what one finds in 14 looking at election statistics is that drop off tends 15 to account for 1.5 to 2.5 percent in Presidential 16 Elections when the data is available, and ranges up 17 to 4.5 percent in non-presidential elections. 18 The problem is that not all states report 19 the actual number of persons that went to the polls 20 on election day. While the availability of this data 21 has improved over time, just 17 states reported that 22 number in 1948. There are still ten states that do 24 1 not compile this information and did not for the 2002 2 election. Those states are Alabama, Arkansas, Maine, 3 Mississippi, Missouri, Oklahoma, Pennsylvania, 4 Tennessee, Texas and Wisconsin. 5 CHAIRMAN SOARIES: Anybody from those 6 states see us after okay. 7 (Laughter.) 8 MR. BRACE: Yes. Now, drop off is a 9 combination of both what is called "over votes" and 10 "under votes". Over votes occur when electors cast 11 more votes than they're allowed for a particular 12 office. For example, they may have voted for two 13 candidates as opposed to one being allowed. 14 Our experience has shown that voters are 15 more likely to cast over vote situations when you 16 have a multiple vote for office; a vote for four, or 17 a vote for five. People don't keep track of how many 18 candidates they vote for and so they over vote. 19 Commissioners DeGregorio and I were just 20 out in Illinois earlier this year and observed a 21 large number of over votes in the Office for 22 Delegates to the National Convention, in a vote for 25 1 seven. In most instances we find that over votes 2 result from improper ballot design which is an 3 important point for election administrators. 4 On the other hand, under votes is more 5 likely to be intentional than over voting. Often if 6 voters don't have enough information about the 7 candidates, they may skip the contest and not vote in 8 that contest. 9 Offices where candidates are unopposed or 10 where candidates have just minor opposition, find a 11 large amount of under voting. 12 What we find and if the data is available 13 a normal election will produce a drop off rate that 14 is generally composed of about 90 percent under votes 15 and just 10 percent over votes. That's in a normal 16 election. Unfortunately elections tend to not be 17 normal in many instances. 18 Unfortunately, in many instances and in 19 many election jurisdictions around the country, over 20 votes and under votes are not reported. In fact, 21 I've been in offices on election day and overheard 22 vendors specifically discourage officials from 26 1 producing reports on over votes and under votes. 2 It's a shame for both the American public as well as 3 the election official. 4 If an election official does not study the 5 results of the election, that official is no better 6 than an ostrich with its head in the sand. Looking 7 for abnormal voting patterns or unusual over vote or 8 under vote relationships are important steps to 9 everyone's research effort, including the use of 10 potentially mapping that information out. 11 Therefore, members of the Commission, as a result of 12 our experience, I would recommend that the Commission 13 undertake the following steps. One of the greatest 14 problems with evaluating different types of voting 15 systems is the lack of data. Therefore, my 16 recommendations would be certainly that the 17 Commission collect more data specifically, more 18 detailed information on voting equipment in use 19 around the nation. 20 Secondly, actual number of persons that 21 voted in each election, the voter turnout and 22 certainly encourage those ten states to finally come 27 1 into the fray and collect those. 2 You should collect precinct by precinct 3 election analysis and election results including over 4 votes and under votes to enable a detailed analysis 5 of the returns for all precincts of the country. 6 You should also collect sample ballots so 7 that one can look at how those ballots appear to the 8 voters and how they might allude to why you see 9 abnormal data in the data that you're collecting. 10 Also I would encourage that the election 11 vendors that are producing software for doing 12 tallying of ballots facilitate this process by 13 putting out data files, not just print files. So 14 that the analysis of this kind of information can be 15 done by both the election administration in that 16 jurisdiction as well as other people. 17 I congratulate the Commission for 18 undertaking this important hearing on voting systems 19 and I would be happy to answer any questions you 20 have. 21 CHAIRMAN SOARIES: Mr. Brace, we want to 22 thank you not only for your presentation today, but 28 1 for the assistance you have given us since we started 2 our work and for the work that you do around the 3 country with election administrators. 4 MR. BRACE: Thank you. 5 CHAIRMAN SOARIES: Is there a question 6 that any Commissioner has for Mr. Brace? 7 VICE CHAIRMAN HILLMAN: I do have just one 8 point for clarification. When you were addressing 9 the issue of over votes and under votes and talking 10 about under votes not being reported, do you mean 11 that they're not counted in some instances? 12 MR. BRACE: In a number of voting systems 13 and tallying systems they do have capabilities of 14 reporting the number of under votes and the number of 15 over votes for each office. 16 What we find is that those kind of reports 17 tend to not be produced election day or post-election 18 day. In fact, if you go and look for and try to 19 compile that information, one finds that election 20 administrators have to go back and rerun the 21 information to generate those kind of reports. 22 It's information that is there. Certainly 29 1 the ballots as they are counted and cast will show 2 you whether or not there's an under vote for that 3 office or an over vote, and so it's something that 4 should be reported. 5 VICE CHAIRMAN HILLMAN: Let me see if I 6 can ask it a little differently because I'm still not 7 8 MR. BRACE: Okay. 9 VICE CHAIRMAN HILLMAN: If you can choose 10 four candidates out of seven 11 MR. BRACE: Okay. 12 VICE CHAIRMAN HILLMAN: -- and you only 13 choose two, and I vote for Soaries and DeGregorio, 14 does my vote in those instances that you just 15 described, do they count for those two candidates or 16 not? If I have chosen not to go to the maximum four, 17 is my vote counting for these two candidates or not? 18 MR. BRACE: Yes. In almost all instances 19 they are counted. Different election laws may be 20 different, but generally, yes, those two votes would 21 be counted. The two additional votes that you did 22 not partake in would be what I would categorize as 30 1 the under votes. 2 And, as I said, they may or may not be 3 counted from the system. 4 CHAIRMAN SOARIES: Commission Martinez has 5 a question that he says is a quick question. 6 MR. BRACE: Okay. 7 CHAIRMAN SOARIES: He's an attorney and so 8 we have to understand the 9 COMMISSIONER MARTINEZ: The question is 10 quick, I don't know what the answer will be. 11 CHAIRMAN SOARIES: And we are really out 12 of time. 13 COMMISSIONER MARTINEZ: We are out of 14 time. 15 CHAIRMAN SOARIES: And I want to make a 16 comment. 17 COMMISSIONER MARTINEZ: Right. Mr. Brace, 18 the idea of not reporting this information, I know 19 you've worked with and for many state and local 20 jurisdictions around the country. We'll have 21 obviously election administrators to talk to as the 22 day progresses. What's the general response as to 31 1 why these ten states are not reporting? They're 2 obviously it sounds like they're collecting it, why 3 are they not reporting it? 4 MR. BRACE: What you end up finding in a 5 number of those jurisdictions and those states is 6 that the data tends to be there. It's down at the 7 county level. The state itself is not collecting it 8 up to present data that can be readily available. 9 CHAIRMAN SOARIES: I want to say, Mr. 10 Brace, that since you gave us introductory 11 information early on this year and we began talking 12 about the error rate and the perceptions about what 13 the does and does not mean, what I've discovered is 14 that there is also confusion between what "error 15 rate" means in terms of over vote and under vote and 16 what "failure rate" means in terms of the malfunction 17 of equipment. And I think as we move forward, we 18 have to dissect that issue because if we talk to 19 people outside of the elections industry, error rate 20 is often synonymous with failure rate and not all 21 errors are due to failure. 22 MR. BRACE: You're quite correct, 32 1 Commissioner, yes. 2 CHAIRMAN SOARIES: Thank you so much for 3 your presentation. 4 MR. BRACE: Indeed. 5 CHAIRMAN SOARIES: We look forward to 6 hearing from the next panel. 7 Our next panel consists of some of the 8 academic luminaries of our time and scientific 9 experts of our country. We are honored to have them. 10 I would like them to come forward now so that I can 11 introduce them individually. 12 I think you can create a little more space 13 for yourself because there's an empty chair. 14 On behalf of this Commission let me thank 15 you gentlemen for being here. We were if my sons 16 were in the presence of MBA stars they would probably 17 have the feeling that is analogous to the feeling we 18 have being in your presence. You are the superstars 19 of your field and you have made contributions 20 already. And our desire is to take your expertise, 21 both the summaries you offer today and the written 22 testimony you've given us and to use this information 33 1 to guide us as we form a national consensus on the 2 issues of electronic voting. 3 We've asked you to help us consider the 4 four critical areas, the accessibility, the 5 usability, the reliability, and security issues that 6 relate to electronic voting. Each of you in your own 7 right as an expert could give a day-long 8 presentation. However, we've asked you to summarize 9 your thoughts if possible in seven minutes. We will 10 then ask you questions and then certainly reserve the 11 right to contact you in the future so that you can be 12 you can be heard through our work. 13 We have from the state of Georgia, Kennesaw 14 University, Dr. Brit Williams. 15 From Johns Hopkins University, Dr. Avi 16 Rubin. 17 From the Institute of Electrical and 18 Electronics Engineers, my neighbor, Stephen Berger. 19 And from the Massachusetts Institute of 20 Technology, Dr. Ted Selker. Welcome gentlemen and if 21 you would speak in the order that you appear on the 22 program, I would appreciate your cooperation. 34 1 Dr. Rubin. 2 PRESENTATION OF DR. AVI RUBIN, JOHNS HOPKINS 3 UNIVERSITY, INFORMATION SECURITY INSTITUTE 4 DR. RUBIN: Thank you, Mr. Chairman. Good 5 morning. 6 My name is Avi Rubin and I'm a professor 7 of computer science at Johns Hopkins University. My 8 area of specialization is computer security and 9 applied cryptography. 10 I've been studying electronic voting since 11 1997 and recently last year served on the 12 security peer review group for the SERV project for 13 overseas Internet voting. 14 Last year I also participated in the 15 analysis of the Diebold acuvote TSX and we since 16 published a paper about the security issues with that 17 machine and the top peer reviewed security conference 18 which is the IEEE security and privacy symposium. 19 By further way of introduction, I this 20 past March served as an election judge in the primary 21 in Baltimore County. I think we'll all agree that 22 security is very important in elections. There are 35 1 many other important things in elections as well; 2 accessibility for blind people; for people whose 3 primary language is not English, and ease of use of 4 the machines are all very, very important. But my 5 expertise is in security. And there are other people 6 that will speak to those issues and I will speak 7 about security. 8 Today's DREs increase accessibility which 9 is great. I like that, but they are insecure, which 10 I don't like. 11 I don't think that security and 12 accessibility are competing goals and I don't think 13 they inherently need to be. But I think with today's 14 deployed DREs, we're in a position that they are. 15 We must demand both accessibility and 16 security from our election machinery and I think that 17 that can be achieved. Let me outline my primary 18 concerns with today's DREs for you from a security 19 perspective. 20 The first and foremost is that there is no 21 way for a voter to verify that their vote was 22 recorded correctly. Machines have the votes inside 36 1 their internal processor inside the memory and even a 2 sophisticated computer security expert cannot look at 3 a machine and tell you what's going on inside of that 4 machine. Only the people who wrote the software know 5 what's going on inside the machine and even they 6 don't really know because it's impossible to develop 7 large software packages without introducing bugs and 8 flaws into them. 9 Another problem that I see with the DREs 10 is that there is no way to publicly count the votes. 11 There's no way for the votes to be counted in a way 12 that's publicly observable because, again the 13 counting is going on inside of a computer. 14 In the case of a controversial election 15 and many elections are controversial. We always have 16 losers in our elections and there are always 17 extenuating circumstances. And when an election is 18 controversial, there are laws in some states that 19 require the ability to do a recount. A meaningful 20 recount means that you are going to believe you have 21 more confidence in the recount than you had in the 22 original vote. And with fully-automated, 37 1 computerized voting equipment, there is no way to do 2 any kind of a meaningful recount. You can just 3 reprint the results and get exactly the same result 4 again. 5 We must trust these machines for several 6 things. We must trust them not to fail. We must 7 trust that they haven't been programmed maliciously 8 and we must trust that they have not been tampered 9 with. And that's a tall order. 10 One of the fundamental concepts in 11 computer security that I teach in my courses and 12 uncovered in all the text books is the concept of a 13 trusted computing base. 14 In a system we try to keep the trusted 15 computing base as small as possible so it has the 16 least amount of code and the least chance that 17 something can go wrong. 18 In today's DREs the trusted computing base 19 is approximately 50,000 lines of computer code 20 sitting on top tens of millions of lines of Windows 21 CE which is more computer code. Not all of the 22 vendors use Windows CE, but the Diebold machines that 38 1 we looked at do. And it is impossible to secure such 2 a large trusted computing base. 3 Future systems should involve the security 4 community that have a lot of experience, there's a 5 lot in the government and a lot in academia, a lot of 6 experience built up on how to design a security 7 system to have as small as possible a trusted 8 computing base. 9 We have techniques for building secure 10 systems, but they are currently not being utilized. 11 When we looked at the Diebold machines, we found 12 gross, gross security and programming errors. We 13 pointed these out in our papers and presented them to 14 our peer community which has widely agreed with this 15 opinion. 16 The worst thing that I see is that when 17 I'm constantly asked, well how bad are the other 18 vendors, or how good are the other vendors, how do 19 they compare to Diebold, and I to say, I don't know, 20 because I can't get access to their code. 21 If people who have security expertise want 22 to analyze and tell the public how secure these 39 1 systems are prohibited from getting access to them, 2 then the public is left wondering what is being 3 hidden inside of there. I'm a strong proponent of 4 opening up these systems for scrutiny. 5 I don't think that we can achieve perfect 6 security. I know better. I know that we cannot 7 achieve perfect security in any useful system. But I 8 believe that there's a spectrum of really, really 9 terrible to very, very good. And my opinion after 10 looking at DREs and looking at the Acuvote TSX from 11 Diebold is that right now we're sitting very, very 12 close to terrible. And I think we can do a lot 13 better. 14 I am not against electronics in voting. I 15 think that we can have computers help us with the 16 voting process, but they need to be designed with 17 input from security experts, and I feel that security 18 experts in general have been shut out from a lot of 19 the decisions about the designs of these machines 20 when approached at all. 21 I do not speak in a vacuum. There have 22 been three other studies, one by SAIC, Robbin 40 1 Technologies, former NSA members and the State of 2 Ohio. 3 Every single study has cited serious 4 security concerns with the DREs. And many election 5 officials I hear and many vendors come out and say, 6 "our systems are secure" and they just repeat that, 7 but they don't show any evidence to back it up. I 8 haven't seen any studies showing what the security 9 measures are. 10 And I think that what we need is to 11 involve the security community the same way we're 12 involving the accessibility community and all the 13 others, it's all part of the puzzle that needs to go 14 together. 15 I will wrap up in a minute. I just want 16 to say that I think you will hear a lot of rhetoric 17 today from my experience. You are going to hear that 18 the procedures in place make the process secure. But 19 I don't think that there are any procedures that can 20 prevent say a malicious program inside of the 50,000 21 lines of code on top of the tens of millions of lines 22 of code that changes votes from one candidate to the 41 1 other. 2 The other problem I have with the claims 3 of the procedures solving all the security problems 4 with the machines is that it is very difficult to 5 design contingency plans. What happens if at the end 6 of the day the machines say, you know, 144,000 people 7 voted and we catch that with our procedure, but there 8 are only 19,000 voters registered. And that actually 9 happened in Fairfax County in the last election. 10 What do we do? Do we throw our hands up 11 if this happens on a national scale and say, well, 12 you know, we messed up? I think that if we built the 13 systems a little more carefully, we could avoid 14 having to rely on procedures that are our contingency 15 plans. 16 I've run out of time so I will be happy 17 during the question and answer to talk about the 18 problems that I see with the logic and accuracy 19 testing versus security testing which are completely 20 different things. And I also don't buy the argument 21 that these machines have worked right in the past so 22 we need so we believe they're perfectly secure. 42 1 If we know that the machines have worked 2 well in the past, then we know they've worked well in 3 the past. But we don't know that they're going to 4 work well in the future and I don't think we should 5 sit on our hands and not enhance them with security 6 to prevent a problem from happening in the future. 7 In conclusion, accessibility and security 8 are not mutually exclusive. We need to develop 9 systems that do not require completely trusting a 10 vendor with the outcome of the election. 11 We need to develop systems that are 12 auditable, including the ability to perform 13 meaningful recounts. And we need to develop systems 14 where votes know that their completed ballot is 15 recorded correctly. We also need transparency in the 16 process and no hidden code. Today's DREs have none 17 of that. 18 Thank you. 19 CHAIRMAN SOARIES: Thank you, Dr. Rubin. 20 We are going to hear from all of the 21 panelist before we do questions and answers. 22 So, Mr. Berger. 43 1 STATEMENT OF STEPHEN BERGER, INSTITUTE OF 2 ELECTRICAL AND ELECTRONICS ENGINEERS 3 MR. BERGER: Thank you very much, Mr. 4 Chairman. I appreciate this opportunity to address 5 the Commission. 6 I got involved in this process in 2001. I 7 have a professional background in telecommunications 8 development of standards particularly for regulatory 9 purposes and then qualification of products to ensure 10 that they meet the requirements. 11 12 13 14 15 16 17 18 19 20 21 22 44 1 From that background I've been involved 2 for some time in the IEEE Standards Association. The 3 IEEE is the largest standards' body in the U.S., I 4 believe, if not the largest one of them. We operate 5 under American National Standard processes to develop 6 consensus documents that represent the center of 7 technical thinking on any given subject. 8 After the 2000 election, some of our 9 members approached the Standards Association 10 essentially saying they felt the engineering 11 community needed to contribute what it could to the 12 improvement of the system. 13 We certainly agreed and started a 14 standards project at that point which continues to 15 this day. There are four things I would like to 16 primarily say to the Commission this morning. 17 The first is, as I've been involved in the 18 system, one of the very pleasant experiences is to 19 realize the tremendous contribution that's been made 20 to the system that we've inherited. 21 A number of people, deeply committed to 22 our Democracy, have worked tirelessly to deliver the 45 1 system that we have today. There is a lot of value 2 there. To be sure, it can be improved but there's a 3 lot to be appreciated and protected. 4 So I would commend to the Commission to be 5 very careful to retain the value that's been 6 delivered to us by those who have worked in days 7 before. 8 Secondly, and almost as a corollary to 9 that, I would observe that probably all the easy 10 problems have been solved. What remained are complex 11 compromises against often-competing requirements. We 12 certainly, as Dr. Rubin has said, want systems that 13 are secure, but also are accessible to people with 14 disabilities that have reliability but could be 15 actually afforded in budgets of jurisdictions all 16 across this country. 17 We received today compromises in those 18 competing requirements. We look for better 19 improvements. Innovation could allow us to more 20 satisfactorily address competing requirements. 21 I believe the best approach to achieving 22 that, as you have already identified, is consensus 46 1 processes where we bring together expertise from 2 various fields and allow all the stakeholders to 3 input to the process. 4 Let me say a few words on where we are in 5 standards in this area. As you well know, in 1990 6 the FEC established the first National Standard for 7 Voting Equipment. It was a tremendous contribution. 8 For the first time there were recognized requirements 9 across the Nation for our voting equipment. 10 Those standards didn't do everything to be 11 sure, but they made an important and large first step 12 in the process of unifying requirements. 13 Standards themselves don't do everything. 14 They have to be addressed into a quality system that 15 implements and monitors their effect and sees that 16 the desired outcome is achieved. And so we have 17 today the ITA system supervised by NASA that 18 implements the standards. 19 Of course in 1998 the FEC staff revised 20 the standards for the 2002 version, which is in force 21 today. We met with the staff shortly after that 22 document was revised and all agreed there was further 47 1 work to be done. 2 There were areas that could use yet 3 further development, particularly in the areas of 4 security, useability, disability access, and others. 5 And those are the focus of the IEEE effort today. 6 Two other comments I would make is that, 7 as we consider the voting system and the quality 8 system, if you will, it is important to recognize 9 that there are four levels that need to be addressed. 10 Certainly there are national requirements 11 such as we have today in the 2002 FEC Standard and 12 the ITA testing to that standard. 13 Then in every state there is a second 14 level of inspection as the states individually 15 evaluate the equipment for use in their own 16 particular use and style. 17 Following that, there is a third level of 18 acceptance testing to ensure that the equipment 19 delivered was represented in that that was evaluated 20 at the state and national level. 21 Finally, there is the Logic and Accuracy 22 Testing to ensure that the equipment on election day 48 1 is functioning properly and accurately. Standards 2 are needed at all those levels, and I would encourage 3 the Commission to pay careful attention to deal with 4 all four of those levels. Some of them have had a 5 great deal more attention than others. 6 It is also important in this area to 7 encourage innovation, but as in all important areas 8 of technology to have a carefully considered way for 9 introducing innovation. 10 We need very much a way of introducing 11 innovation that puts it through careful evaluation, 12 trial, development of specifications to safeguard 13 against possible vulnerability, and in phased 14 deployment so that we guard the system against 15 unintended consequences. 16 That does not exist in a unified way 17 today, and is very much needed. 18 So I will close with that introduction to 19 my comments. There is more detail in the written 20 version, but I thank the Commission for this 21 opportunity to address you. 22 CHAIRMAN SOARIES: Thank you very much, 49 1 Mr. Berger. 2 Dr. Selker. 3 STATEMENT OF DR. TED SELKER 4 MASSACHUSETTS INSTITUTE OF TECHNOLOGY 5 DR. SELKER: I am Ted Selker, and I am a 6 Professor at MIT at the Media Lab. I have been 7 involved with making many products at IBM, including 8 an accessibility package for the OS2 Operating 9 System, which has tens of millions of lines of code. 10 David Baltimore from Cal Tech and Charles 11 Best got together after the election in 2000 and 12 said, you know, maybe the technologists can help. 13 And in creating this forum for political scientists 14 and computer scientists and other technologists to 15 get together, we all learned from each other. 16 The most exciting thing we learned was 17 that in fact the electronic technology that is most 18 useful right now for understanding this stuff is the 19 Internet. We found that lots and lots of the data, 20 the forensics is public data and it is available on 21 the net, and we have done lots of studies to learn 22 such things as that the registration data base 50 1 problem is the largest problem in how we lost our 2 votes in 2000. 3 Probably between 1.5 and 3 million votes 4 were lost because registration data bases are in 5 error. 6 We don't have any way of checking how many 7 New Yorkers are registered in Florida. In fact, it 8 is not illegal. We don't know how we are choosing 9 who we are going to check the registration data base 10 and eliminate possible people that are not supposed 11 to be voting. 12 I don't know any changes that have been 13 made systemically, or even best practices, as a 14 result of the well-reported problems of Florida in 15 2000. 16 As we go through and understanding that a 17 lot of what's been going on is we've been starting 18 with assumptions. Many people have been spouting off 19 about technology and problems with technology and 20 other things in elections. That's not new. But we 21 have to replace that with testing. 22 What is exciting about the more data that 51 1 we have today is that testing is more feasible. We 2 really want to make these standards performance- 3 based. We want them to be better than they were 4 before, as a criteria. 5 If we look at the goal of protecting, 6 detecting, and correcting problems we have ways of 7 detecting the kinds of fraud that I'll be talking 8 about. 9 Parallel testing is the mechanism by which 10 you run elections, phantom precincts on the day of 11 elections using actual machines and show that the 12 input equals the output. 13 In many case, voting machines don't have 14 clocks in them. I just checked over a machine from 15 Ireland that did not have a clock in the machine. 16 That simplifies various aspects of the testing. 17 It does not mean that somebody couldn't 18 get a foundry, build a chip, put a battery inside 19 with what looks like an E-prong and put that into the 20 ballot module so that it could have a clock and know 21 to expose its Ester DG (?) on the day of election. 22 However, there are many other ways of 52 1 defrauding elections that might be easier. So that 2 is not the most expected approach for causing that 3 kind of mischief. 4 The real center of my comments probably 5 has to do with how do we vet the qualifications of 6 the people that we need to help us through this 7 process. We have to develop experts, experts that 8 can be trusted, experts that can help the EAC, I 9 hope, figure out what is good and what is wrong, what 10 are the critical things that have to be improved, 11 experts that can help the people that are making 12 decisions about what equipment to buy. 13 These local election officials today have 14 all sorts of problems. If you take a look at the 15 useability problem--and I'm an expert in 16 useability--if you take a look at the 13,000 ballots 17 that were thrown out for over-votes in Palm Beach 18 County in 1996, the Democrats and the Republicans 19 signed off on that butterfly ballot. 20 In 2000, again the Republicans and the 21 Democrats signed off on it. There were only 22 19,000--it was 300 or so ballots that had chad 53 1 problems. There were 19,000 over-votes because of 2 the design of the ballot. 3 Probably one percent of our electorate was 4 lost because of bad ballot design in this country, 5 and I don't know of anybody that is saying: How do 6 you run a simple test to see if this ballot is good? 7 Polling place practices were equally 8 flawed. I have watched polling place practices where 9 people teach their officials by telling them, or 10 teaching them concepts. Others by procedures. We 11 know that we have simple procedural understanding and 12 simple things to go on. You can make better choices. 13 One million votes were lost that way. 14 But I shouldn't dwell on these non- 15 technical matters. Let me just say that I believe 16 that the elections over the last few decades have 17 reduced the errors and the failures gigantically over 18 what it was before. 19 We don't have enough data to do more than 20 state it. We can show some examples. But in fact we 21 have to figure out how we move forward. As we look 22 at the machines that we are testing today, we are 54 1 thinking: Well, can we rely on parallel testing? 2 The doomsday scenario that people are 3 terrified of is what if we had to run another 4 election? Well people have had to run other 5 elections when they've had troubles in the past, but 6 if we refuse to take that we can go for verification. 7 Verification is an important idea. The 8 question is: Can people improve the election through 9 verification? 10 Now I know of no study--in fact, the most 11 recent one that I've been involved with, we had 3 12 people out of 1000 making mistakes when there was 1 13 person doing the task, a second person watching over 14 their shoulder and signing each time they did the 15 task that they had done it right, and the third 16 person doing the same thing. 17 Still, there was a .3 percent error. This 18 is an unacceptable level of error for testing for 19 fraud or for testing voting kinds of equipment. 20 The question is: If we had a perceptual 21 task--I'm in favor of having a task such as redundant 22 information. It uses the already available 55 1 electronics inside of the DREs of today. You can 2 produce an audio. That audio can be heard while 3 you're making the decision--a perceptual task that 4 happens while you're making the decision is one that 5 people universally can do. Cognitive and memory 6 tasks, you act after you vote by looking at another 7 action, another piece of paper, are not so easy. 8 In Wilton, Connecticut, where they tried 9 it, they had terrible problems. Twice as many ballot 10 workers. Twice as long for the voters. People, the 11 exit polls did not show confidence in the system. 12 So I am very excited about using a tape 13 recorder with a separate record and playback head. 14 You play back something that's already been recorded 15 onto it. If somebody tries to erase that, that tape, 16 you have integrity. We'll all remember that 19 17 minutes of erased tape for a long time. 18 In Wilton, Connecticut, there were 19 actually slots at the bottom of the ballot box that 20 the ballots could fall out. We're talking about the 21 first time in a very visible place where voting 22 verified paper trails were tried. 56 1 So I believe that audio verification is 2 available today. It is available with equipment that 3 we own, and it can do a better job in helping people 4 verify and validate that they have voted the way they 5 want. 6 The best thing about it, as well, is that 7 it can be read by a computer and by a person. This 8 is not true of most of the technologies that people 9 are considering today. 10 We don't know how to count receipts at the 11 100,000 level that we've tried to specify for 12 election equipment. I'm not sure that I should go on 13 very much longer. I just want to thank you all for 14 being here and I would welcome any questions. 15 CHAIRMAN SOARIES: Thank you so much. 16 Dr. Williams. 17 STATEMENT OF DR. BRIT WILLIAMS 18 KENNESAW UNIVERSITY, GEORGIA 19 DR. WILLIAMS: Well I'd like to thank you 20 for that glowing introduction. I wish my president 21 had been here to hear it. 22 (Laughter.) 57 1 CHAIRMAN SOARIES: You do fine with your 2 president. 3 (Laughter.) 4 DR. WILLIAMS: I think the one thing that 5 we all agree on is that there is ample room to 6 improve our existing voting systems, and that is the 7 goal that all of us have got before us. 8 But we have to keep in mind in doing that 9 that there are a lot of aspects to a voting system 10 other than just accuracy and security. We have got 11 to look at availability. We've got to look at 12 reliability, maintainability, useability, and even 13 affordability. 14 We could build the quintessential voting 15 system, but if nobody can afford to buy it it is a 16 futile exercise. So any change to a voting system 17 has to be evaluated on the basis of its impact on the 18 entire system, and I think that is the whole purpose 19 of the formation of this Commission. 20 What we need to guard against I think is 21 the tendency to go out and do something quick and 22 dirty that is a rapid, poorly formulated addition, 58 1 such as a paper receipt for instance, to an existing 2 voting system could have an adverse effect that far 3 offset any of its advantages. 4 And furthermore, actions like this are 5 unnecessary because we're not in any eminent danger. 6 To do the kinds of things we're talking about here is 7 not going to be fast. We're not going to implement 8 Dr. Rubin's recommendations in the short term. 9 In the short term--and by "short term," 10 I'm really talking probably four to six years--we're 11 going to have to dance with them what brought us. 12 And so we really need to look at what we can do with 13 our existing voting systems to compensate for these 14 vulnerabilities that we know are there. 15 I agree with Dr. Rubin that you can't 16 compensate for them 100 percent, but nobody 17 guaranteed me that that airplane I'm flying home on 18 is 100 percent safe, either. 19 So in that spirit, one of the hardest 20 things I have had to do--I submitted this long 21 discourse to you--and one of the hardest things I've 22 had to do is to say, now what am I going to use this 59 1 little precious seven minutes to talk about? 2 So what I've decided is to look at some 3 recommendations, some things that I think we can do 4 based on our experiences in Georgia that maybe we can 5 carry nationwide that would shore up some of the 6 immediate problems that we've got to deal with in 7 order to run elections in 2004 and 2006. 8 The number one recommendation I have is to 9 implement a nationwide secure voting system software 10 library. NIST currently has a secure law enforcement 11 software library. They use that, or the way that 12 library works is that if you have law enforcement 13 software, you submit it to NIST. NIST puts it in the 14 secure library. They compute a hash signature on 15 that, and then that signature can be used in a court 16 case or in a challenge to verify that software that's 17 in use in the field is in fact unaltered from the 18 software that's in that software library. 19 I think we could very quickly extend this, 20 since that technology is already in place, we could 21 very quickly extend this to voting system software. 22 The way it work would be that when the ITA completes 60 1 their qualification of a voting system, they submit 2 the software, not the vendor, but the ITA submits to 3 NIST for the secure software library the exact system 4 that they've just finished qualifying. 5 Then from there on, NIST handles it the 6 way they handle the law enforcement software. If 7 there's a challenge to that software, or if any 8 jurisdiction has any concerns about the validity of 9 their software, they could get that signature from 10 NIST, run the same signature against their own 11 software, and verify that there's been no 12 modification to the software they have. 13 We do that in Georgia. When we bring a 14 system into Georgia, we give it a software from the 15 ITA, not from the vendor, and we compute a hash 16 signature that I believe is the same identical 17 signature that NIST uses. It's in the paper I 18 submitted to you. 19 Then on a period and on a random basis 20 when we have people out in the field, we run 21 signatures against the installed software to verify 22 that it has not been altered from the software it is 61 1 served by. So this is something that the mechanics 2 and the mechanisms are in place. 3 Now there's a lot of software out there, 4 so I'm not suggesting that we go try to round it all 5 up. What I'm suggesting is that we start with the 6 new systems coming out. And then as jurisdictions 7 request to add new systems to the systems, to that 8 library, so that if a jurisdiction is running say a 9 version of ES NIST software and they want to include 10 it in the library, then they give NIST--they identify 11 it uniquely to NIST using either the qualification 12 number or the vendor version specific numbers. 13 NIST obtains that from the ITA's archives 14 and implements it into the secure library. The 15 second recommendation I have is probably as equally 16 important, but a little part of it is not going to be 17 as easy to do. 18 If you go and look at anomalies that have 19 occurred in recent elections, you will find almost 20 without exception that those could have been maybe 21 avoided, and at least minimized, by well trained poll 22 workers or well trained election officials. 62 1 Poor ballot design leads to all kinds of 2 problems. Poorly trained poll workers, where things 3 that could have been a simple problem escalate 4 because the poll worker didn't know how to handle it 5 quickly on the spot. 6 So to that end, again in Georgia--and 7 Kathy Rogers in her presentation is going to go into 8 some more detail on this program--we've developed a 9 64-hour program of training, and we have a State law 10 now that says by I believe it's 2005 that every 11 county office has to have a State-certified person in 12 that county office. That is, someone who has 13 successfully completed our 64-hour training program. 14 Now all states probably can't do that, but 15 all states have universities that have departments of 16 continuing education, and all states have technical 17 institutes. Maybe this Commission could give block 18 grants to those institutions to develop specific 19 programs for those local jurisdictions--not 20 generalized, here's generally how you run an 21 election, but here's how you run an election in this 22 county under these State laws with this equipment, 63 1 similar to the program we have in Georgia. 2 Now that's going to require some 3 additional documentation. Already the Office of 4 Election Administration out of your office has done a 5 lot of work in developing generalized election 6 management type documents. 7 What we need now is some way to take the 8 vendor documents and customize those into specific 9 documents that can be used by localities. Mostly 10 it's a cut-and-paste kind of thing, because the 11 vendor document has got every feature of the system 12 in there and nobody implements every feature of the 13 system. 14 So what you need is to pare those things 15 down, and then turn them into specific documents. 16 Here's the document for the person who is going to 17 build the ballots. Here's the document for the 18 person who's going to train poll workers. Here's the 19 document for the precinct manager. 20 I will stop at that. I very much 21 appreciate the opportunity to talk to you today, and 22 I look forward to working with you. 64 1 CHAIRMAN SOARIES: Thank you, so much. 2 Let me just share how much I appreciate your 3 discipline. We know that you have so much to say and 4 to offer, but you have given us time to ask you 5 questions and you have left room for the other panel 6 and I really appreciate that. But you have said so 7 much that I hope you know that we will be calling 8 you. 9 Our questions will be led by Commissioner 10 Martinez. 11 COMMISSIONER MARTINEZ: Thank you, Mr. 12 Chairman. 13 Let me add my thanks to all of you. I 14 appreciate your time and your commitment to be here. 15 Your verbal and written testimonies I think are very 16 much on the mark of what we were looking for in this 17 first public hearing. 18 Let me--what I will do is I will just ask 19 questions in the order that you all spoke. To the 20 extent that you can keep your answers to a relatively 21 short response so that I can leave time for my fellow 22 Commissioners to also ask you questions, but I do 65 1 have specific questions as I've had a chance to take 2 a look at your submitted testimony, et cetera. 3 So I will start, Dr. Rubin, if I could 4 with you. Thanks again for being here. 5 Ever since I was approached about serving 6 on this Commission, and perhaps even before then, but 7 certainly since around March of last year, I have 8 followed very intensely the debate that mostly rages 9 in the media between computer scientists and election 10 administrators. 11 It seems that even up to this very day 12 that computer scientists are talking at and over 13 election administrators and the same is coming back 14 at you. 15 I am interested, Dr. Rubin, my first 16 question is just to get--you served as a poll worker 17 and wrote I think a very interesting and compelling 18 account. It sounds like you did it at 5:00 o'clock 19 in the morning, so I applaud you for doing that, but 20 I think I read it actually at 5:00 o'clock in the 21 morning. But give me just your general impressions. 22 I know what your conclusion was, and I 66 1 read through your essay about that particular 2 experience. I mean you have entered--you know, what 3 you did is you took off the hat of computer scientist 4 and entered the world of essentially election 5 administration for a full day. I think you served 6 the entire day as a poll worker. 7 Talk a little bit about your general 8 impressions. What did that experience impart to you? 9 What has changed in your view in terms of the 10 vulnerabilities of DREs, and what has reinforced your 11 view of those vulnerabilities? 12 DR. RUBIN: Okay, one of the big 13 criticisms that I received from a lot of people after 14 our report came out was that I didn't know that much 15 about elections, that I was a computer scientist and 16 I needed to learn about elections, and I thought that 17 that would be a very good way to do it. So I 18 volunteered and served as an election judge. 19 It was interesting to me that the machines 20 in the site where I was were the very machines that I 21 had analyzed the code for. It was a very unusual day 22 for me because I saw voters coming in and universally 67 1 liking the machines. They really liked them, which 2 told me that there is something good about the design 3 here, something good about the human factors here and 4 that we need to preserve that, and I think these 5 comments were made earlier about preserving what is 6 good. 7 At the same time, I felt a little nervous 8 and almost hypocritical supervising machines that I 9 knew were not secure and that I was concerned would 10 not operate properly. 11 In the statement that I wrote up that you 12 referred to, I did mention that the experience 13 focused my opinion both on things that I had thought 14 were problems before that I thought were less of a 15 problem in practice, and things that had not occurred 16 to me that I viewed as being more serious problems in 17 the experience. 18 So what it did was, it was an excellent 19 thing for me to do because it focused me on what was 20 a realistic evaluation. I think ever since that 21 experience I've been able to speak with a lot more 22 authority about the security issues in these 68 1 machines. 2 One of the issues that we brought up in 3 our report was the fact that, when looking at the 4 code in the computers--and those of you who are 5 familiar with these computers know that you take a 6 smart card which has a ballot on it, and you put it 7 in the machine and it's designed to prevent you from 8 voting more than once. 9 Given that a smart card has a chip on it 10 and some protected storage, there are ways--and we 11 know in my community how to do that--and they didn't 12 do it right. It was actually as bad as you could 13 possibly imagine. No cryptology. No authentication 14 whatsoever. They could have been using matched 15 stripes for all they did with that. So we wrote 16 about that. 17 When I served as a poll worker, I was in a 18 precinct that had nine election judges and five 19 machines. In the entire day, 16 hours, we received 20 199 votes. 21 So when somebody went up to a machine and 22 voted, the card was knocked out and there was a loud 69 1 clicking sound, and we were already heading towards 2 them to take it away and thank them for voting and 3 give them a sticker, et cetera. 4 The attack we designed in our paper was 5 one where you could manufacture your own smart cards, 6 walk up to a machine and vote 20 times. Now in my 7 precinct that would not have worked, and so I pointed 8 that out in my statement that I wrote up. 9 However, one of the things I also noticed 10 was: At the end of the day the memory cards in each 11 computer were collected that had the tallies on them, 12 were taken out of all the machines after the totals 13 were printed up, and then put into one machine and 14 they were accumulated there together. 15 As a computer security person, I always 16 look for the point of highest vulnerability, and I 17 thought that was it because that was the point where 18 we had all the votes on one machine, and then they 19 were supposed to get modemed back to the back end 20 servers at the Board of Elections, or wherever they 21 go. 22 Now that was another part of the code that 70 1 we had analyzed, and they did the cryptography on 2 protecting that communication incorrectly. They used 3 a broken site in a mode that's insecure, so even had 4 it not been broken it would have been bad, and they 5 used one key that was hard-wired into all of the 6 machines, which is a no-no in computer security. 7 And so I became concerned thinking that, 8 you know, here we have something completely 9 ephemeral, these bits that are representing all of 10 the votes and, as a security person, that made me 11 very nervous. 12 I actually at the symposium that NIST put 13 on, it was when one of the secretaries of state that 14 was there came up and told me that I really should 15 serve as an election judge, I'm very, very grateful 16 for that advice. So I think, you know, that that 17 summarizes the experience. It's really helped me 18 focus a lot. 19 COMMISSIONER MARTINEZ: So is it possible 20 for election administrators to be a computer 21 scientist for a day? 22 DR. RUBIN: That would be harder. 71 1 (Laughter.) 2 COMMISSIONER MARTINEZ: That's harder to 3 do. 4 DR. WILLIAMS: Could I speak to that just 5 a minute? 6 COMMISSIONER MARTINEZ: Yes. 7 DR. WILLIAMS: That perception is not 8 quite accurate. Those votes that are accumulated on 9 that accumulator are for press release purposes only. 10 The official tally is done from the individual cards, 11 from the individual machines. They're taken back to 12 the central location, not transmitted by modem. That 13 accumulation in that modem transmission on election 14 night is purely for the benefit of the press and so 15 forth. The official tally is conducted from the 16 individual voter cards in the county office the next 17 day. 18 COMMISSIONER MARTINEZ: Thank you, Dr. 19 Williams. And I am going to ask you to follow up, 20 Dr. Rubin, so if you want to respond to that you can 21 do so. 22 DR. RUBIN: I appreciate that opportunity. 72 1 It is very interesting. You asked what were my 2 impressions and my feelings. 3 COMMISSIONER MARTINEZ: Yes. 4 DR. RUBIN: I've dealt with PCMA memory 5 cards, the ones we're talking about, very often and 6 the thought that from when the voters came in until 7 those cards were removed from those machines, there 8 was no physical record of those votes is what made me 9 very uncomfortable that day. 10 COMMISSIONER MARTINEZ: I see. Dr. Rubin, 11 generally speaking what are the types of--and I don't 12 know if you can do this without speaking the computer 13 scientist language which would go over my head, 14 unfortunately, but what are some of the general types 15 of security threats, the risks that you've identified 16 in the machines that you've looked at? 17 And if you can, what's the likelihood of 18 such a risk occurring? 19 DR. RUBIN: Okay, there are two different 20 levels to answer this on. One is specifics of the 21 Diebold Acuvote TSX, which I think are less 22 interesting because that's one machine that's 73 1 received a lot of scrutiny and I think there is the 2 issue of security of DREs in general. 3 COMMISSIONER MARTINEZ: Sure. 4 DR. RUBIN: And I'd rather address the 5 second one. If you want me to address the first one, 6 I-- 7 COMMISSIONER MARTINEZ: The second is much 8 more appropriate. 9 DR. RUBIN: My biggest concern is that in 10 a very large trusted computing base the threat that 11 somebody with access to the development environment 12 of the code base--typically the vendor--basically is 13 in a position to make the outcome of the election 14 come out however they like. And they can be 15 infinitely clever about how they do this, and it's 16 virtually undetectable. 17 So let me give you an example that comes 18 to my mind. Say that I am malicious and I am hired 19 by a vendor to build a voting machine and I'm one of 20 the programmers on it. I embed malicious code in 21 there that actually does nothing until something 22 happens. The thing that has to happen is a voter has 74 1 to walk in and touch the touch screen in a very 2 unusual fashion, say put four fingers on the screen 3 three times in a row. Call it the knock. And when 4 that happens, the machine changes its behavior and 5 takes the internal votes and shifts five percent of 6 them from one candidate to another. In addition to 7 doing that, then removes itself, removes the 8 malicious code from the machine. 9 To try to figure out how realistic and 10 difficult that was, I teach a graduate course in 11 computer security at Johns Hopkins and this past 12 semester I had 40 mostly Ph.D. graduate students 13 build mock voting systems and embed back doors in 14 them with a secret knock. 15 They did that for half of the semester, 16 and the other half they received each other's--they 17 received several machines from other classmates not 18 knowing if we had given them one that had a back door 19 on it or not. 20 I was astounded to see the cleverness and 21 the ease with which the malicious code was hidden, 22 and how difficult it was to find. 75 1 The last part of your question is: What 2 is the probability that something like this would 3 happen? I believe that we have to look at the 4 incentives out there to tamper with the election. 5 You've got billion dollar contracts 6 dependent on the outcome of elections, and so I think 7 we've got very well funded and bad intentioned 8 adversaries to worry about. 9 DR. SELKER: Could I respond to that? 10 COMMISSIONER MARTINEZ: Dr. Selker, sure. 11 DR. SELKER: That particular idea of 12 having a funny user interface that somebody could 13 walk into is an extremely labor-intensive way to 14 change votes. That means that somebody will have to 15 go into a balloting booth in many, many places to 16 make a change. Unless, you know, maybe for a water 17 district it might be worthwhile, but for other things 18 it isn't. 19 So the leverage of the attack is really 20 one of the things that Avi and many of us have 21 focused on. So the thrust that I'm most concerned 22 about are ones that are systematic that will be part 76 1 of the whole system and will affect large elections. 2 And those ones can be tested for by parallel testing 3 and even before elections, and as well for the code 4 that persists after elections. And some of the 5 threats can be detected with other means as well. 6 COMMISSIONER MARTINEZ: Very quickly, Dr. 7 Rubin, back to you, and again as I ask you one more 8 question and then you can respond if you want to to 9 what Dr. Selker said. 10 And I do have other questions, and I am 11 running out of time unfortunately, but Dr. Rubin in 12 the continuum that you've described from one being 13 terrible to ten being very, very good, if in the 14 interest of our Democracy you and Diebold decided to 15 go into business together, what could we do to move 16 up that spectrum? 17 If you were advising Diebold, and I guess 18 you have suggested some things already, but just for 19 the record what are some things that--and I don't 20 mean Diebold specifically, I mean to stick with the 21 general DREs--what are some things that can happen? 22 I guess I'm trying to get to: From your perspective, 77 1 and I know there are some who believe this, but from 2 your perspective is a voter-verifiable paper ballot 3 the only way to fully secure--again understanding 4 that we could never have a 100 percent fully secure 5 system--but is that the only answer, is what I'm 6 trying to get to, from your perspective. 7 DR. RUBIN: I believe there's a short-term 8 answer to that and a long-term answer. 9 I think in the short term, meaning 10 November 2004, that a voter verifiable paper ballot 11 is necessary because it's the only way to get around, 12 it's sort of an end-run around all of the security 13 problems in the machines. 14 If the voters see their paper, and if it 15 is implemented correctly, and that is the ballot the 16 way they meant to vote it, and that is kept, then we 17 can have recounts. We get around the problem of not 18 being able to audit with recounts. 19 Then the voters have some confidence that 20 they're leaving the poll place with something behind, 21 which is their vote exactly the way they voted it. I 22 do believe that in the long, long term we should 78 1 explore other cryptographic options and combinations 2 of techniques. 3 I happen to think that the most bang for 4 the buck you can get is by adding paper, voter 5 verifiable paper, into the process because it avoids 6 so many pitfalls. Then the challenges are to design 7 the system so that it works so that, you know, you're 8 not dealing with paper jams. 9 I think I am much more worried about a 10 poll worker dealing with a very bad software bug on 11 election day than a jamming printer. 12 CHAIRMAN SOARIES: Commissioner Martinez, 13 if the other Commissioners are going to ask this 14 panel questions they've got to start now. 15 COMMISSIONER MARTINEZ: Okay. Thank you. 16 CHAIRMAN SOARIES: Commissioner 17 DeGregorio. 18 COMMISSIONER DeGREGORIO: Thank you, Mr. 19 Chairman. I know that because of limitations in time 20 I won't be able to ask each panelist a question, but 21 let me ask Mr. Berger who has been involved in the 22 Standards' process for many years, as he described, 79 1 and is a representative of IEEE and will be on the 2 Technical Guidelines Development Committee that will 3 be set up very soon to look at the standards that 4 this Commission will adopt eventually. 5 I am concerned because it is my 6 understanding that the 2002 Standards that were 7 developed by the FEC that you had a hand in--they 8 were updated--that there are very few systems out 9 there that meet those 2002 Standards right now. 10 What can you tell me that would encourage 11 me that these vendors of this equipment will be 12 tested and will meet these 2002 Standards for the 13 2004 election? 14 MR. BERGER: Well as you very well point 15 out, there is a process. You have to have 16 specifications. The vendor has to have time to 17 respond to them. And then their offerings have to be 18 evaluated, be certified, and then acquired and 19 deployed. 20 That takes time. It is something that 21 every field has. In this particular case, I think 22 one of the best features to put in the system is 80 1 fully engaging the vendors in the development of 2 those specifications. 3 We certainly don't want to turn over the 4 system to the vendors but they know what they can 5 implement quickly and what they can't. They have 6 insights as an important stakeholder to the process, 7 but perhaps most importantly if you use a consensus 8 process and the vendors see the handwriting on the 9 wall, if you will, the experience in many areas is as 10 the standard works through its final approval process 11 and implementation the vendors are very busy in their 12 product development having products ready for market. 13 That very often stands in contrast to 14 processes where you somewhat hold the development of 15 specifications behind closed doors, and then you 16 serialize that process. 17 I'd like to add a comment if I may, 18 quickly, to the previous discussion. It would simply 19 be this: We need to look around for other fields 20 that have something to offer in the issues we were 21 just discussing. 22 In an election audit, we are essentially 81 1 involved in an historical research. We are trying to 2 determine what the voter did at a point in time. 3 Recent history, to be sure. 4 It is well established in historical 5 research that you have the highest confidence that 6 you understand what occurred by multiple independent 7 witnesses and accounts that have been kept separate 8 so that they don't influence one another. That is a 9 principle that I think we need to think carefully 10 about in this field; that as quickly as possible, and 11 as independently as possible, we have independent 12 records of what the voter does so that audits can 13 compare separate accounts. That's a well established 14 principle, and I think it avoids the kind of 15 bottlenecks that Dr. Rubin pointed to. 16 CHAIRMAN SOARIES: Commissioner Hillman-- 17 COMMISSIONER DeGREGORIO: One last 18 comment, Mr. Chairman, while I have the floor--I'm 19 going to steal the floor-- 20 (Laughter.) 21 COMMISSIONER DeGREGORIO: --but I just 22 want to compliment Dr. Rubin for working at the polls 82 1 and joining the ranks of the million Americans or so 2 out there who work at the polls. 3 I read your commentary the next day, too, 4 and it wasn't five in the morning, but it may have 5 been eight o'clock in the morning, because I know 6 that when I was an election director we had 7 difficulty recruiting good workers. 8 I encourage everyone in this room to work 9 at the polls, if you can, and the media out there to 10 encourage people to become poll workers. 11 I know Ted Selker and I spent 15 hours at 12 the polls in Los Angeles last October, and so I think 13 it is important for people in the academic, 14 scientific, media, to get it from the inside and work 15 at the polls. 16 I do have one concern, though, when I see 17 emails that go out to encourage people to be poll 18 workers to not be real poll workers but to subvert 19 the system. I'm not suggesting that at all about you 20 or anyone else here, but I have seen some of that go 21 on in the past few weeks and it does concern me that 22 people are out there to pretend to be poll workers 83 1 but really want to subvert our electoral system and 2 process. 3 Thank you, Mr. Chairman. 4 CHAIRMAN SOARIES: Vice Chair Hillman. 5 VICE CHAIR HILLMAN: Thank you. 6 I have two questions, but one I would like 7 to ask each of the panelists to submit your response 8 in writing. That is, on the issue of a way for the 9 voter to verify that their votes were recorded 10 correctly. 11 I would just like to see from your 12 perspectives the difference between--I obviously know 13 how you can do it with a paper ballot--but with the 14 lever machine, the Opti Scan, and the DREs, the voter 15 verification question. Because I'm not seeing in my 16 mind the difference between the lever voting, which 17 has been used for decades, and the DRE once you hit 18 that lever and push that button it's been gone. So 19 for 90 years the issue wasn't discussed, and now it 20 is. So that will help me. 21 DR. SELKER: Could I speak to that for a 22 moment? 84 1 VICE CHAIR HILLMAN: Well I do have 2 another question, but if you could just submit your 3 responses to me, just a one-pager would be fine, I 4 would really appreciate that. 5 My other question: Dr. Williams, if you 6 could just briefly share your observation and your 7 thoughts about the role that the independent test 8 agency plays in this whole discussion about the 9 certification of the machines, and that as a useful 10 tool and any suggestions or thoughts that you would 11 have to the Commission about the work of that agency. 12 DR. WILLIAMS: Well of course one thing 13 they do is give us a uniform starting point. 14 CHAIRMAN SOARIES: Hold on, Doc. If you 15 could just pause so that the microphone can go up, 16 that way people can hear the first part of your 17 statement. 18 DR. WILLIAMS: Is it on now? 19 VICE CHAIR HILLMAN: It is. 20 CHAIRMAN SOARIES: There's a little time 21 delay here.I 22 DR. WILLIAMS: It gives us a uniform 85 1 starting point. I've got on my desk the 1990 2 Standard and the 2002 Standard, and when the ITA 3 tells me that they have evaluated a system with 4 respect to one of those two Standards, then I know 5 what that means. I know what they've done to it. I 6 know what the system had to do to come through that. 7 It tells me, for instance, that the system 8 is reliable; that the system is maintainable, that 9 the components in its are quality components; that 10 the engineering that went into it is quality 11 engineering; and that the functionality of it is a 12 voting system. 13 It also tells me that it has been at least 14 looked at from a cursory basis from the point of view 15 of security and fraudulent code and those kinds of 16 things. 17 Now, you know, every time you say that 18 everybody goes (fluttering hands) ohhhhhhh, you know, 19 you can't do that. Well, no, you can't. There's no 20 such thing as a 100 percent secure system of any 21 kind. But the more it is looked at by the ITAs and 22 so forth, it raise your confidence level. 86 1 So their evaluation brings your confidence 2 level in the system up to a point. 3 Then the next step in the Standards, and 4 we think, when we talk about the standards we tend to 5 talk about them as if they were federal-level 6 standards; they're actually standards at three 7 levels. That second level, then, is state 8 certification. 9 The next thing a state should do is bring 10 that qualified system into the state and do a review 11 on it at the state level, number one, to see if there 12 are any peculiarities in the state law, the state 13 code, the state regulations that need to be examined 14 that the ITA didn't examine. 15 Pennsylvania, for instance, has a very 16 unique way of voting, changing your vote in a multi- 17 member straight-party election called "The 18 Pennsylvania Method." No other state does it that 19 way. 20 21 22 87 1 DR. WILLIAMS: And then you should always 2 look at the system from the point of view of 3 usability and affordability at the state level 4 because the ITA's do not consider these two hardly at 5 all. 6 And certainly not affordability. They 7 don't even know what that costs. That's strictly a 8 local concern. 9 VICE CHAIR HILLMAN: Is that a transparent 10 process? I mean, would most people who would want to 11 know how the ITA is doing this process, is it 12 transparent to us or to elections administrators? 13 DR. WILLIAMS: Yes, I think it is. I 14 mean, certainly this is not any kind of secret 15 proprietary process. The standards are yours. They 16 are EAC standards. And the ITA's are intermediaries 17 for an asset. 18 Now, the problem you run into is how do 19 you fund this thing? See, we have no money. 20 So the way it's funded is the -- contracts 21 with the ITA for the evaluation. So up to the point 22 where that evaluation report is released, that's a 88 1 propriety relationship. 2 Now, once that report is through, it 3 becomes pretty much a public document although 4 officially it belongs to the vendor to pay for it. A 5 vendor would have to be out of their mind to refuse 6 to give it to you. 7 I mean, so those are very available. And 8 you can. Yes, it's a very open process. 9 VICE CHAIR HILLMAN: Thank you. 10 CHAIRMAN SOARIES: Thank you. I've got 11 two quick questions. 12 We did inherit FEC standards. And we are 13 working hard to position ourselves to enhance the 14 standards pursuant to many of the principles that you 15 made clear today. 16 I think we have to acknowledge as often as 17 we need to today that all of this costs money. And 18 I've been pressed by the media particularly to find 19 out what happens next. And just to give you just a 20 preview of what we'll say after this over, we've got 21 to raise money. 22 All of this costs money. And I know if 89 1 there's one thing we'll all agree on from every 2 perspective today -- that we need money to invest in 3 this process. So we do value the work that's been 4 done because much of it has been done by volunteers. 5 I don't know how many of you were involved 6 in the development of the '90 standards and then the 7 update to '02. But I'm just curious to know -- maybe 8 you, Mr. Berger, would know -- in the '90s there was 9 reference to a standard for paper verification. And 10 the '02 standard there's no such thing. 11 I'm just wondering if it was an oversight. 12 Was there a conscious decision made to make no 13 reference to paper verification in '02? What -- 14 DR. WILLIAMS: I don't recall that as a 15 conscious decision. That's something we could talk 16 with Penelope about. You know, Penelope was the 17 editor-in-chief of that. 18 But I don't remember any discussions in 19 any of the meetings I was in where a conscious 20 decision was made to leave that out. It sounds like 21 an oversight. CHAIRMAN SOARES: Okay. 22 MR. BERGER: I'll just say that I got 90 1 involved late in the process as the -- became 2 engaged. There was a mature draft at that point. We 3 took