Python, DictReader, DictWriter

Because I can never, ever, remember exactly how to code these. Example of both, basic.

data_list = []

with open(input_file, 'rU') as f:
  data_file = csv.DictReader(f)
  for line in data_file:
    data_list.append(line) # gets you a list of dicts

the_header = ['h1', 'h2', 'etc'] # column headers, a list of text strings

with open(output_file, 'w') as f:
  file_writer = csv.DictWriter(f, fieldnames=the_header)
  file_writer.writeheader()
  for line in data_list:
     file_writer.writerow(line)

Here I am going to experiment linking it as a script-tagged element from Gist via GitHub:

	# Because I always forget exactly how to do these.
	# Note that in function calls the exact code will be a little different (may need additional code).
	# Reading
	data_list = [] # Will be output, list of dicts.
	with open(input_file, 'rU') as f:
	data_file = csv.DictReader(f)
	for line in data_file:
	data_list.append(line) # Gets you a list of dicts.
	# Writing
	the_header = ['h1', 'h2', 'etc'] # Your column headers, a list of text strings. These are placeholders.
	with open(output_file, 'w') as f: # Don't forget your output file has to have a location/name.
	file_writer = csv.DictWriter(f, fieldnames=the_header)
	file_writer.writeheader()
	for line in data_list: # data_list is the list of dicts you are writing.
	file_writer.writerow(line)
	# Done!

view raw csvDict_read_write.py hosted with ❤ by GitHub

16th Century Maps for 21st Century Data Science

Maps bother me. I love them, and I'm not a geospatial GIS coding specialist, but I do visualizations, and we keep using the wrong maps. Greenland is a lot smaller than all of Africa, ok?

This is the map in my office kitchen:

It's the typical Mercator projection (projection, since you have to "project" a sphere onto a flat surface, which doesn't work well). Mercator came up with this map view in 1569, according to Wikipedia. Yet we still use it for 21st century data science! Granted just because something is old doesn't mean it's not useful, but in this case the Mercator projection was created primarily for navigation, that is, sailing the seven seas. When you present geospatial data the only thing your viewers are navigating is your data. As such this is totally the wrong mapping projection to use. Totally. Don't do it. Data visualizations are about accuracy, and using the Mercator projection starts you off with a completely inaccurate mapping. Greenland and Africa? "Africa's area is 14 times greater" than Greenland according to that Wikipedia article! Fourteen!

So what to do instead?

Wikipedia has a page of many different projections, I'd vote for one of the equal-area ones, and am a fan of the Gall-Peters projection (which was the centerpiece of a great segment on The West Wing), but you'll need to decide what's best for your use.

So, I'm a little upset about the giant Mercator map in my office, but with good reason.

Code for All Summit

Had a great time at the Code for All summit, held here at Civic Hall. Global meets local with a variety of civic tech people and a few government and NGO people thrown in. Code for All is the global offshoot of Code for America.

One nice thing to see was that yes, sometimes the best solution is SMS and not a fancy app.

That's me in the front row second from the left.