Friday, April 17, 2015
Omidyar Network hosted a great bunch of ignite talks (the crazy auto-slide-advancing ones) at Civic Hall on April 13th, including a few people I know which was exciting!
- Laurenellen McCann Mo Open Data
- Tony Schloss Red Hook WIFI- The Realest Community Tech
- Miriam Altman The uphill battle to graduation day
- Chris Whong In Search of Hess' Triangle
- Rose Broome What is the Basic Income?
- Lane Becker Good Enough for Government Work?
- David Riordan The NYC Space/Time Directory: How The New York Public Library is unlocking NYC's past
- Gavin Weale SA Elections 2014: a youth odyssey
- Paul Lenz The Resilience of the Past
- Kathryn Peters Disrupting government
- Joel Mahoney Civic Technology and the Calculus of the Common Good
- Kate Krontiris Understanding America's Interested Bystander
- Nick Doiron The Civic Deep Web
- Jessie Braden Open Data Perils: Question Everything
- Daniel X. O'Neil Changing Civic Tech Culture from Projects to Products
- Daniel Latorre What I Learned About CivicTech from Eastern Europe
- Noel Hidalgo Clear eyes. Full heart. Can't Loose! NYC's civic tech in 2022.
Great topics, some great ideas, and some great slides.
Monday, April 13, 2015
Friday, April 10, 2015
I don't have a background in SQL, so getting the syntax correct for SQLite in Python was a little tricky, especially since it reads like it's straight out of 1983. So, here is some working syntax for a search/select and replace, and also a search/select and an iteration through the results.
Search for one result (on a unique variable) and change some of that entry's data:
cursor.execute("UPDATE outfits SET size=?, members=?, scraped=? WHERE id=?", (how_many_members, char_id_list, 1, int(outfit_id)))
This assumes you know a bit about SQL.
cursor is your cursor object. This snippet searches the db for any entries (lines, rows, whatever) for where the
id variable matches the value of
outfit_id. In this case, that variable will have all unique entries since I declared it that way when I made the db (which is some other SQLite code that is in multiple other places on the net). So, this line finds the one I want and then changes those three variables, then you
commit it which actually write it. That seems really weird to me, either do it or don't do it. I assume this made some sense back in 1983 when people wrote code in capital letters.
outfits here is the name of your table in the db. Well it's the name of
Iterator on search results:
cursor.execute("SELECT * FROM outfits WHERE scraped=?", (0,)) # this selects them but doesn't return them for use. NOTE TUPLE!!!
not_scraped_outfits = cursor.fetchall() # aha!
for an_outfit_row in not_scraped_outfits:
# do your stuff here
That seems weird to me, but I guess I don't understand the cursor idea. You
SELECT in caps, but then you have to
fetchall(). That seems like two steps where you only need one. So, you
SELECT everything (the asterisk, I think) from your table that matches the
WHERE call, here where
0, since it's a Boolean. That returns possibly none, one, or more. Usually for me in this particular code it will return several, and then you have to iterate through the results, and I think there are a few ways to code the iterator call, but the code I have here works so there you go.
SELECT which is a search (the
fetchall the results (even though you already selected them), then you can iterate through them.
NB: Tuple! When you do the funky security thing, which I can't explain and don't care about since I am only running local code, the argument has to be a tuple, so if you are just passing one argument you need a trailing comma:
cursor.execute("SELECT * FROM outfits WHERE scraped=?", (0)) # fail
cursor.execute("SELECT * FROM outfits WHERE scraped=?", (0,)) # success, due to the last comma there
Also, one of the many pages I was poking around at suggested SQLite Manager, a plugin for Firefox. There may be other similar things, I have no idea, but I really like it, it's free, and if you don't have anything that allows you to view the innards of your SQL db easily, I strongly recommend it. If you don't use Firefox, heck it's just another app (I tend to think I don't need three browsers on my machines, but hey).
More also, it is apparently a good idea to store really long ID numbers as text, not numeric. (Because something, somewhere, decided to round them all off so they were all wrong.)
Thursday, April 9, 2015
A great evening at Kickstarter HQ here in Greenpoint! Three fantastic talks about processes they use there:
- Kumquat: Rendering Graphs and Data from R into Rails, by Fred Benenson, Head of Data.
- Rack::Attack: Protect your app with this one weird gem! by Aaron Suggs, Engineering Lead.
- Testing Is Fun Again, by Rebecca Sliter, Engineer.
Friday, April 3, 2015
If you try to assign a list to a column in an R data frame, it won't quite work, you need unlist. (That's the short version for the search engine snippet, although it does not make for a great narrative intro, it's more the concise summary.)
A few days ago, I was working in R, and was generating a new data frame from another one. It was a little more complex than I was used to, for instance I had to bin one variable by the values of another variable, and make some new percentage/frequencies in the new, smaller, data frame, so I couldn't just use the non-looping approach that is common to R (and which is a lot nicer and fast). The data frame was relatively small, so one loop level was not a problem, even on my 4.5 year old MacBook Air.
In one section, I generated a list of values (numeric, nothing fancy), and then assigned that list to a column in my new data frame. When I called the data frame to look at it, it looked fine, but if I did str(my_df) or summary(my_df) something was horribly wrong -- the column wasn't a numeric column, it was some odd list format and wasn't working for my ggplot.
I tried assigning the generated values directly to the column in the data frame, with something like this inside the loop, where I also incremented i:
my_df[i, 'the_variable'] = one_generated_variable
(Note I can't use R syntax there with the greater than sign, Google barfs on the code even though it's text, so I have to use the equals sign which is older R style.)
one_generated_variable was just a numeric value. Should have been fine, I thought! But no, it still came out as a list. I have no idea why, honestly it seems impossible since the values were generated one at a time and assigned then and there -- they were not bundled into a list first. But, unlist fixed the problem.
my_df$the_variable = unlist(my_df$the_variable)
That did it. I still don't understand the details, since I don't see why it was a list in the first place (aren't columns vectors anyways?). I have never run into that problem before, although mostly I've been working in Python lately.
Also, a friend put me onto data.table instead of data.frame for bigger data.