Assignment 4: Processing files and data

Due

by 10:30pm on Tuesday, 20 February 2018.

Summary

For this assignment, you will put your data science skills to work by processing data and files.

Collaboration

You must work with your assigned partner(s) on this assignment. You may discuss this assignment with anyone, provided you credit such discussions when you submit the assignment.

Submitting

Email your answers to csc151-01-grader@grinnell.edu. The subject of your email should be [CSC151 01] Assignment 4 and should contain your answers to all parts of the assignment. Scheme code should be in the body of the message, not in an attachment.

Warning

So that this assignment is a learning experience for everyone, we may spend class time publicly critiquing your work.

Problem 1: Iowa voter registration

Topics: Files, Heterogeneous lists and tables, Strings

Many states now provide a wealth of information online. For example, the state of Iowa has a collection of data sets available at https://data.iowa.gov. One such data set is county-by-county voter registrations for nearly the past two decades. That data set is available at https://data.iowa.gov/Communities-People/State-of-Iowa-Monthly-Voter-Registration-Totals-by/cp55-uurs and as a CSV file that we’ve downloaded for you.

Here are a few lines from that file.

02/01/2018 12:00:00 AM,19001,Adair,929,1901,12,2055,5,4902,69,74,3,166,0,312,5214,41.3307464,-94.4709413,"(41.3307464, -94.4709413)"
02/01/2018 12:00:00 AM,19003,Adams,550,935,6,1158,1,2650,49,64,1,145,0,259,2909,41.0289839,-94.6991849,"(41.0289839, -94.6991849)"
02/01/2018 12:00:00 AM,19005,Allamakee,1994,4135,19,2990,7,9145,173,182,4,390,0,749,9894,43.2842838,-91.3780923,"(43.2842838, -91.3780923)"

Of course, those lines are meaningless without a “code book” that explains what each piece is. You can find the code book on the data.iowa.gov Web site. We’ve also provided a simple summary.

Because the voter registration roles are so large, we’ve created two subsets of the data.

• iowa-voter-registration-2018-02.csv contains just the information for February 2018. The associated code book is available as iowa-voter-registration-2018-02.txt.
• poweshiek-voter-registration.csv contains just the information for Poweshiek county. The associated code book is available as poweshiek-voter-registration.txt.

Make copies of those files (right-click and “Save As”) and then do the following.

a. Document and write a procedure, (most-republicans fname), that finds and returns a list of the names of the five counties with the most active Republican registrations. You will run this procedure on iowa-voter-registration-2018-02.csv.

b. Document and write a procedure, (most-democratic fname), that finds the five counties with the highest ratio of active Democratic registrations to total active registrations. Your procedure should return a list in which each entry is a list containing the date, the county, and the ratio. You will run this procedure on iowa-voter-registration-2018-02.csv.

c. Document and write a procedure, (greatest-democrat-change fname), that reads a file of registration that is in date order and returns the five months that saw the greatest absolute change in the number of Democrats. Note that you will need to figure out a way to compare each month to the prior month. You will run this procedure on poweshiek-voter-registration.csv.

d. Document and write a procedure, (greatest-democrat-change-alt fname), that reads a file of registration that is in date order and returns the five months that saw the greatest relative change in the number of Democrats, where the relative change is computed as the absolute change divided by the previous number. You will run this procedure on poweshiek-voter-registration.csv.

Problem 2: Splitting dates

Topics: strings

As you may have noted, many people choose to represent date-time entries in a computationally inconvenient form, such as “03/01/2015 12:00:00 AM”. We tend to refer to that as “MM/DD/YYYY HH:MM:SS xM” form. Note that this form is ambiguous, not only because many countries use the form “DD/MM/YYYY” for dates but also because it does not specify where in the world that date is computed.

If we wanted to sort a list of data by date, we would therefore need to find a way to extract the date in a computable form.

Document and write a procedure, (split-date date-time), that takes a string in the aforementioned format and turns it into a list of the form (YYYY MM DD HH xM MM SS) where everything other than xM is an integer.

> (split-date "03/01/2015 12:00:00 AM")
'(2015 3 1 12 "AM" 0 0)

Problem 3: Testing the split-date procedure

Topics: Testing

Write a RackUnit test suite for split-date. You may assume that the input to split-date has the proper type and is formatted correctly, but be sure to consider a variety of date-time entries.

We are likely to run your tests on some non-working versions of split-date. Your tests should catch any reasonable errors.

Problem 4: Iowa voter registration, continued

Topics: Files, Heterogeneous lists and tables, Strings, Filtering

You’ve recently explored some interesting computation with Iowa voter registration information using two files.

• iowa-voter-registration-2018-02.csv contains just the information for February 2018. The associated code book is available as iowa-voter-registration-2018-02.txt.
• poweshiek-voter-registration.csv contains just the information for Poweshiek county. The associated code book is available as poweshiek-voter-registration.txt.

As you know from experience, some unexpected data can cause problems for data analysis. We often use filtering to address that issue.

a. Earlier, you wrote a procedure, (most-democratic fname), that extracted the five most Democratic counties according to some metric. Document and write a similar procedure, (most-libertarian fname) taht finds the five most Libertarian counties using the metric of “ratio of active Libertarian registrations to total active registrations”. Your procedure should return a list in which each entry is a list containing the date, the county, and the ratio. You will run this procedure on iowa-voter-registration-2018-02.csv.

b. Document and write a procedure, (most-active-other fname), that finds the five counties that have the highest ratio of active “Other” registrations to inactive “Other” registrations

Problem 5: Searching for words

Topics: Files, Lists, Strings

Write a procedure (file-find str filename) that finds all the occurrences of the string str in the file filename. The return value of file-find should be a table whose entries correspond to lines that contain the given string. Each entry of the table should contain the line number in the file and the actual string of the corresponding line. For example,

> (define file "/home/username/Desktop/us-zip-codes.txt")
> (file-find "Zip" file)
'((1 "US Zip Codes ")
  (4 "Zip codes and other information of cities in the US.")
  (8 "* 0: Zip code"))
> (file-find "Cleanup" file)
'((17 "Cleanup by Samuel A. Rebelsky on 2017-09-11: Removed quotation marks from")
  (20 "Cleanup by Samuel A. Rebelsky on 2017-09-12: \"Federated States of Micro\""))
> (file-find "curmudgeon" file)
'()

Hint: There is a procedure built-in to Scheme called string-contains? that may be helpful for this problem.

Evaluation

We will primarily evaluate your work on correctness (does your code compute what it’s supposed to and are your procedure descriptions accurate); clarity (is it easy to tell what your code does and how it achieves its results; is your writing clear and free of jargon); and concision (have you kept your work short and clean, rather than long and rambly).