MIN(D)ING THE VOTER FILE AT THE RRCC

What is the Registrar Recorder/County Clerk and what do we do?

• Medium size LA County Department (roughly 1000 permanent employees)
• Largest voting jurisdiction in the US (little over 4.8 million registered voters)
• Hatch 'em, match 'em, and dispatch 'em
• Want to buy a house?
• Want to vote? Where do I vote? Want to be a poll worker?

What kinds of problems are our Election Services Operations facing?

• Duplicate records in our voter file
• Deceased voters remaining on the active rolls
• Dealt with an internal County Audit

What can be done?

• Ongoing file maintenance
• Enhance this work via data analytics
• Instead of staff chasing after needles in a hay stack… find the needles first then work the records

County Audit

• County Auditor Controller reported that they had identified a potential 52K potential duplicates (over 105K voter records) using a match of First Name, Last Name, and Home Address
• After random sampling and further analysis, the Department determined that only 8200 (16%) were real duplicates
• How do we separate the true positive matches from the false positives without manually inspecting 105K records?
• Significant costs for false positives… voter disenfranchisement

Fuzzy Matching (Levenshtein Distance)

• 84% false positive matches were due to generational differences between parents and offspring who shared the same name and lived together
• True positives showed DOB's that were similar but had slight variations (likely due to typo error)
• Agrep() would return both records using a cost threshold of 2:

John Smith 06-23-1967

John Smith 06-23-1976

Results

• Agrep() returned 8700 duplicate matches of the 52000
• 8600 were true duplicates, manually inspected and confirmed by staff (99% accuracy)
• 6 additional ongoing queries using a variety of matching keys and mechanisms to continue to identify duplicates for operations staff to address
• In the last year we have cancelled over 37000 duplicate records

What About Deceased Voters?

• Lists received from the CA Secretary of State did not contain home address or SSN
• Manual inspection by staff was cumbersome and labor intensive
• Staff would compare printed death certifcate to voter file
• Matches on First, Last, DOB, and Home address would result in record being cancelled
• How do we speed this up?

Better Data and Automation

• Requested data sets include additional data points like Home Address and SSN
• Matching is now automated and results go directly to IT for cancellation
• On going data sharing between RRCC and Department of Public Health
• Monthly deceased voter maintenance results in 1200-2000 voters canceled
• Nearly 30000 voters cancelled in the last year

Poll Worker (PW) Recruitment

• What happens when we can't recruit enough PW's
• Or cancel last minute
• Or even worse… no show on election day

We may end up with…

CORE PROBLEM: Finding the right PW's

• Intuitions by our recruiters are good, but can only do so much
• Intuition varies from recruiter to recruiter
• For the Novemmber 2014 general election, roughly 24000 PW's worked the polls
• Unfortunately we had to call 36500

PRIMARY VERSUS GENERAL (2014)

     Outcome     Election Count
      WORKED 2014 PRIMARY 19157
      CANCEL 2014 PRIMARY  2201
     NO-SHOW 2014 PRIMARY  2813
 UNAVAILABLE 2014 PRIMARY   194
      WORKED 2014 GENERAL 24152
      CANCEL 2014 GENERAL  4389
     NO-SHOW 2014 GENERAL  5515
 UNAVAILABLE 2014 GENERAL  2443

ROOM FOR IMPROVEMENT

• Given success rates of 67% for our November 2014 election and 78% for our June 2014 election, clearly room for improvement (less so for our primary elections)
• Can begin better differentiating between commited versus uncommitted PW's
• How do we model the very intuitions that our recruiters are attempting to access
• And find additional patterns that are derived mathematically

GOAL OF THIS PROJECT

• Minimize negative outcomes (PW said no thanks, cancelled last minute, no showed on election day) while still identifying sufficient numbers of PW candidates to actually work
• Costs of incorrectly identifying a negative outcome as a positive outcome: potential late openings of certain polls, last minute shuffling of resources, many more phone calls required

Challenges

• What data do we have, what can we use?
• Messy data sets, inaccuracies, duplicate voter records, missing data from different fields
• Valuable demographic data not collected by the Department (income level, education attainment, race/ethinicity, etc)

What do we know about our pollworkers

• Some demographic data (gender, age)
• Voting history= civic participation?
• Previous pollworker history predicts future pollworker participation?
• Distance travelled from home to poll site matters?
• Specific outcomes (response)
• PW's can be split into 6 distinct clusters using unsupervised K means clustering

UNSUPERVISED CLUSTERING (Primary Election PW's)

• Cluster 1: pretty good voting record, regular PW, 57 yo, low income, predominantly female
• Cluster 2: likely to be first election worked, very new voter, youngest average age
• Cluster 3: likely to be first election worked, voted in previous election but low voting rate, entirely female
• Cluster 4: excellent PW record, outstanding voter record, pretty good income, 50/50 gender split, oldest age group

CLUSTERING cont.

• Cluster 5: likely new PW, so-so voting record, decent income, entirely male
• Cluster 6: likely worked the previous time called, so-so PW history, good voting history,lowest income group, predominantly female

So what's in a Model?

Modeling Challenges

• For both Primary and General PW data, there exists a significant class imbalance between successful outcomes and unsuccessful outcomes
• Not to the level of fraud detection ML problems, but the imbalance is causing significant skewing at the point of model training
• Some ML algorithms used simply predicted the majority class without actually returning class probabilities
• How do you address this problem?

Class Imbalance

• Rebalance the training data using different sub-sampling techniques: Upsampling the minority classes, downsampling the majority classes, SMOTE'ing or ROSE'ing the data set
• Introduce a customized loss function or cost matrix into the ML algorithm being trained (some ML algorithms have built in weighting functions for this very issue that can be estimated via cross validation)
• Thresholding: Using the ROC curve to identify a different probability threshold in which the majority class will be assigned (don't use default >.50)

What is the prediction algorithm doing?

• For each record being predicted on, assigning a class probability (between 0 and 1) to the outcome variable
• Under normal circumstances, anything with a probability >.5 will be predicted as a successful outcome (“A”), anything less will be considered an unsuccessful outcome (“U”)
• Setting the threshold higher makes the algorithm more discriminating in its predictions, allowing us to shed our unsuccessful outcomes
• Ended up using a mix of GBM and ADA Boost algorithms

What is being provided to our Operations staff?

• Lists of pw candidates from each of the 6 clusters who exceed the class thresholds
• Can always loosen the thresholds and send new lists if they run out of candidates
• This list will assist them in prioritizing who they call to recruit

How good are the predictions?

• Used the 2014 Primary and General PW data as the hold out test set
• For the 2014 primary election, the algorithm selected 8720 candidates that met the individual cluster thresholds, of which 7616 were correctly identified as “A”, giving us an accuracy rate of 87% (accuracy rate of 88% on the 2014 General election test set)
• The obvious problem is that we are short about 11000 poll workers.

How good are the predictions (cont.)?

• We can loosen the thresholds to pull in more “A” candidates that didn't make the higher thresholds, but end up sacrificing accuracy since we will also be pulling in more “U” candidates
• Or, we can turn to our untapped pool of 109K poll workers that were not contacted by recruiters, and use the prediction algorithm to find more candidates.
• As it turns out, using the same higher probability thresholds for this untapped pool, the algorithm finds another 17573 candidates to suplement the 7616 already correctly identified from the test set.

Doing 2014 over

• Were we to have had this prediction algorithm for the Primary and General Elections in 2014 we would see the following results…
• Primary: Recruited 19157 resulting in a successful outcome, recruited 2862 resulting in a negative outcome (reduction of negative outcomes by 2346, or half)
• General: Recruited 24152 resulting in a successful outcome, recruited 3273 resulting in a negative outcome (reduction of negative outcomes by 9074, or three quarters)

Contact Info

Benjamin Uminsky
Executive Assistant: Data Analytics
Registrar-Recorder/County Clerk
12400 Imperial Hwy., Executive Office Norwalk, CA 90650 (562) 462-2731
[email protected]