Meta tags:
description=1 post published by GR during January 2016;

Headings (most frequently used words):

py, culture, code, and, archive, for, january, 2016, tw2csv, infoquery, datacollection, the, geeks, archives, recent, comments, blogroll, references, resources, scraping, twitter, with, python,

Text of the page (most frequently used words):
the (76), user_timeline (71), you (37), and (36), for (27), this (24), #twitter (22), user (22), users (19), that (16), code (15), from (15), twython (15), import (15), with (14), tw2csv (14), query (13), 2010 (12), 2011 (12), python (11), time (11), line (11), one (11), keys (11), but (11), data (10), 2009 (10), 2012 (10), entities (10), are (9), list (9), some (9), january (9), 2013 (9), object (9), tweets (9), cycles (9), api (9), retweeted_status (9), del (9), script (9), use (9), have (8), stata (8), april (8), march (8), july (8), november (8), file (8), quoted_status (8), can (8), handles (8), 100 (8), note (8), text (7), june (7), october (7), 2016 (7), all (7), screen_name (7), allows (7), not (7), dictionary (7), will (7), scripts (7), pprint (7), like (6), culture (6), may (6), september (6), february (6), module (6), handle (6), csv (6), app_key (6), access_token (6), 200 (6), tweet (6), only (6), per (6), coordinates (6), need (6), objects (6), run (6), get (5), other (5), what (5), things (5), december (5), 2017 (5), parentpath (5), command (5), argument (5), today (5), authentication (5), out (5), there (5), infoquery (5), these (5), com (4), imdb (4), migrants (4), recent (4), august (4), datacollection (4), sys (4), ceil (4), alphanumeric (4), characters (4), app_secret (4), get_user_timeline (4), count (4), most (4), 3200 (4), through (4), incremental (4), calls (4), val (4), enumerate (4), place (4), python3 (4), sociologicalsci (4), call (4), don (4), about (4), hard (4), queries (4), wordpress (3), blog (3), research (3), mac (3), science (3), strange (3), afoot (3), gabrielrossman (3), 2020 (3), 2021 (3), sociology (3), target (3), adapted (3), doing (3), lis (3), sleep (3), second (3), rest (3), minutes (3), followers_count (3), created_at (3), urls (3), str (3), user_mentions (3), hashtags (3), bash (3), application (3), change (3), made (3), two (3), see (3), was (3), any (3), writing (3), them (3), rate (3), open (3), osx (3), shows (3), twdata (3), set (3), dict_writer (3), fields (3), dictionaries (3), escaped (3), commas (3), type (3), sub (3), site (2), log (2), sign (2), subscribed (2), subscribe (2), perl (2), org (2), theory (2), posts (2), notes (2), simulation (2), lseltzer (2), networks (2), comments (2), 2014 (2), 2015 (2), 2018 (2), scraping (2), rss (2), math (2), custom (2), rossman (2), documents (2), twittertrucks (2), argv (2), takes (2), screenname (2), strftime (2), csvfilepath (2), csvfilepath_clean (2), authenticate (2), oauth_version (2), simple (2), obtain_access_token (2), deep (2), necessary (2), tweetsum (2), than (2), range (2), iterate (2), max (2), extend (2), between (2), conservative (2), clean (2), save (2), qt_id (2), name (2), urlj (2), mentionj (2), hashtagj (2), nonetype (2), dict (2), extended_entities (2), geo (2), above (2), register (2), key (2), secret (2), reason (2), retweet (2), very (2), profile (2), structure (2), interactively (2), given (2), instance (2), has (2), few (2), would (2), over (2), same (2), last (2), tricky (2), thing (2), cycle (2), build (2), targetlist (2), csvfilepath_info (2), lookup_user (2), len (2), here (2), actual (2), generate (2), location (2), coded (2), default (2), getting (2), information (2), created (2), assumes (2), your (2), commented (2), which (2), likewise (2), csvfile_out (2), functools (2), allkey (2), union (2), output_file (2), another (2), missing (2), within (2), current (2), field (2), reasons (2), append (2), problems (2), utf (2), fleas (2), also (2), level (2), first (2), excel (2), long (2), lot (2), want (2), though (2), called (2), pretty (2), disk (2), limits (2), fifteen (2), pip (2), install (2), workflow (2), been (2), started, design, collapse, bar, manage, subscriptions, view, reader, report, content, privacy, already, account, now, join, 128, subscribers, create, free, website, unix, tutorial, davis, ucla, academic, technology, services, safari, books, gated, internet, movie, database, raw, ninja, cpanda, california, center, population, references, resources, jenn, lena, gabi, huiber, tidbit, week, michael, mitchell, daily, sociological, imagination, social, statistics, harvard, soc2econ, rense, corten, religion, curiosities, permutations, eric, booth, econometrics, disgruntled, sociologist, drip, ssrc, blogroll, reading, cogiddo, jdgalt, soc, edudammerung, universe, kenny, online, teaching, 2019, 2022, 2025, archives, typesetting, editor, superstar, socm176, organizations, shell, satire, resampling, regular, expressions, random, variables, power, analysis, philosophy, phenomenology, netlogo, macros, lyx, loops, history, graphs, genetics, financial, crisis, ethnomethodology, epistemology, economic, economics, diffusion, culure, cleaning, causality, categorization, bayesian, asa, search, pkremp, geeks, simplejson, feeds_py, _clean, http, www, craigaddyman, mining, less, just, statuses_count, depth, point, trying, deeper, include_retweets, true, max_id, user_screen_name, user_followers_count, user_id, user_created_at, rt_count, retweet_count, rt_created, rt_user_screenname, rt_user_id, rt_user_followers, qt_created, qt_text, qt_user_screenname, qt_user_id, qt_user_followers, url_, expanded_url, mention_, hashtag_, none, coord_long, coord_lat, supply, versions, pulls, subdictionaries, deletes, care, interested, retweeting, background, color, ahead, modify, tips, exploring, choose, scrape, 1192, six, times, weeks, 1200, least, seven, makes, faster, ultimately, gets, collects, specified, moreover, delay, limited, tweaks, infoquery_py, listing, feeds, full, path, info_, rstrip, read, cmd, alternately, initialize, unlike, cap, total, taken, yourself, txt, input, output, important, operating, systems, calling, basic, date, description, follower, counts, basically, stuff, page, simultaneously, stored, lets, easier, designed, much, simpler, interactive, querying, def, reduce, lambda, dictwriter, writeheader, writerows, anyway, variable, number, entries, cases, biggest, problem, reusing, stack, overflow, required, adapting, give, used, entry, those, found, uniquely, past, future, sorts, order, randomly, overwrite, timestamp, tweak, into, lining, piece, delimited, foo, delimiter, comma, bindquote, strict, varnames, asdouble, encoding, clear, values, themselves, real, upon, kind, deal, pull, delete, others, version, defaults, internal, delimiters, cells, really, while, automatically, parses, correctly, unless, each, element, bunch, brackets, make, human, readable, whitespace, prettyprinter, indent, status, zoom, daughter, case, itself, discussing, even, directly, because, natural, discuss, somewhat, complicated, exporting, lists, good, feel, look, using, sample, explore, three, write, particular, slow, execute, deliberate, otherwise, end, hitting, passing, doesn, require, intervention, sure, knows, where, find, usually, keeping, working, directory, adjust, paths, when, accessing, instead, phone, usernames, passwords, secrets, registering, tip, installation, tends, syntax, let, follow, standard, instructions, installing, before, showing, person, comes, readers, remember, collecting, library, unfortunately, proven, buggy, mostly, having, such, decided, learn, migrate, project, overall, impressed, language, haven, had, dependency, works, smoothly, hand, requires, more, manual, coding, around, does, big, part, gabriel, archive, hello, world, home,

Text of the page (random words):
n python and migrate my project to the twython module overall i ve been very impressed by the language and the module i haven t had any dependency problems and authentication works pretty smoothly on the other hand it requires a lot more manual coding to get around rate limits than does twitter and this is a big part of what my scripts are doing i ll let you follow the standard instructions for installing python 3 and the twython module before showing you my workflow note that all of my code was run on python 3 5 1 and osx 10 9 you want to use python 3 not python 2 as tweets are utf 8 if you re a mac person osx comes with 2 7 but you will need to install python3 for the same reason use stata 14 for tweets one tip on installation pip tends to default to 2 7 so use this syntax in bash python3 m pip install twython i use three py scripts one to write twython queries to disk one to query information about a set of twitter users and one to query tweets from a particular user note that the query scripts can be slow to execute which is deliberate as otherwise you end up hitting rate limits twitter s api allows fifteen queries per fifteen minutes i call the two query scripts from bash with argument passing the disk writing script is called by the query scripts and doesn t require user intervention though you do need to be sure python knows where to find it usually by keeping it in the current working directory note that you will need to adjust things like file paths and authentication keys when accessing twitter through scripts instead of your phone you don t use usernames and passwords but keys and secrets you can generate the keys by registering an application tw2csv py i am discussing this script first even though it is not directly called by the user because it is the most natural place to discuss twython s somewhat complicated data structure a twython data object is a list of dictionaries i adapted this script for exporting lists of dictionaries you can get a pretty good feel for what these objects look like by using type and the pprint module in this sample code i explore a data object created by infoquery py type users shows that users is a list type users 0 shows that each element of users is a dictionary the objects are a bunch of brackets and commas use pprint to make a dictionary sub object human readable with whitespace import pprint pp pprint prettyprinter indent 4 pp pprint users 0 pp pprint users 0 status you can also zoom in on daughter objects in this case the user s most recent tweet object note that this tweet is a sub object within the user object but may itself have sub objects as you can see if you use the pprint command some of the dictionary values are themselves dictionaries it s a real fleas upon fleas kind of deal in the datacollection py script i pull some of these objects out and delete others for the clean version of the data also note that tw2csv defaults to writing these second level fields as one first level field with escaped internal delimiters so if you open a file in excel some of the cells will be really long and have a lot of commas in them while excel automatically parses the escaped commas correctly stata assumes you don t want them escaped unless you use this command import delimited foo csv delimiter comma bindquote strict varnames 1 asdouble encoding utf 8 clear another tricky thing about twython data is there can be variable number of dictionary entries ie some fields are missing from some cases for instance if a tweet is not a retweet it will be missing the retweeted_status dictionary within a dictionary this was the biggest problem with reusing the stack overflow code and required adapting another piece of code for getting the union set of dictionary keys note this will give you all the keys used in any entry from the current query but not those found uniquely in past or future queries likewise python sorts field order randomly for these two reasons i hard coded tw2csv as overwrite not append and build in a timestamp to the query scripts if you tweak the code to append you will run into problems with the fields not lining up anyway here s the actual tw2csv code tw2csv py def tw2csv twdata csvfile_out import csv import functools allkey functools reduce lambda x y x union y keys twdata set with open csvfile_out wt as output_file dict_writer csv dictwriter output_file allkey dict_writer writeheader dict_writer writerows twdata infoquery py one of the queries i like to run is getting basic information like date created description and follower counts basically all the stuff that shows up on a user s profile page the twitter api allows you to do this for 100 users simultaneously and i do this with the infoquery py script it assumes that your list of target users is stored in a text file but there s a commented out line that lets you hard code the users which may be easier if you re doing it interactively likewise it s designed to only query 100 users at a time but there s a commented out line that s much simpler in interactive use if you re only querying a few users you can call it from the command line and it takes as an argument the location of the input file i hard coded the location of the output note the 3 in the command line call is important as operating systems like osx default to calling python 2 7 python3 infoquery py list txt and here s the actual script note that i ve taken out my key and secret you ll have to register as an application and generate these yourself infoquery py from twython import twython import sys import time from math import ceil import tw2csv custom module parentpath users rossman documents twittertrucks infoquery_py targetlist sys argv 1 text file listing feeds to query one per line full path ok today time strftime y m d csvfilepath_info parentpath info_ today csv authenticate app_key 25 alphanumeric characters app_secret 50 alphanumeric characters twitter twython app_key app_secret oauth_version 2 simple authentication object access_token twitter obtain_access_token twitter twython app_key access_token access_token handles line rstrip for line in open targetlist read from text file given as cmd line argument handles gabrielrossman sociologicalsci twitter alternately hard code the list of handles api allows 100 users per query cycle through 100 at a time users twitter lookup_user screen_name handles this one line is all you need if len handles 100 users initialize data object hl len handles cycles ceil hl 100 unlike a get_user_timeline query there is no need to cap total cycles for i in range 0 cycles iterate through all tweets up to max of 3200 h handles 0 100 del handles 0 100 incremental twitter lookup_user screen_name h users extend incremental time sleep 90 90 second rest between api calls the api allows 15 calls per 15 minutes so this is conservative tw2csv tw2csv users csvfilepath_info datacollection py this last script collects tweets for a specified user the tricky thing about this code is that the twitter api allows you to query the last 3200 tweets per user but only 200 at a time so you have to cycle over them moreover you have to build in a delay so you don t get rate limited i adapted the script from this code but made some tweaks one change i made was to only scrape as deep as necessary for any given user for instance as of this writing sociologicalsci has 1192 tweets so it cycles six times but if you run it in a few weeks sociologicalsci would have over 1200 and so it would run at least seven cycles this change makes the script run faster but ultimately gets you to the same place the other change i made is that i save two versions of the file one as is and the other that pulls out some objects from the subdictionaries and deletes the rest if for some reason you don t care about retweet count but are very interested in retweeting user s profile background color go ahead and modify the code see above for tips on exploring the data structure interactively so you can see what there is to choose from as above you ll need to register as an application and supply a key and secret you call it from bash with the target screenname as an argument python3 datacollection py sociologicalsci datacollection py from twython import twython import sys import time import simplejson from math import ceil import tw2csv custom module parentpath users rossman documents twittertrucks feeds_py handle sys argv 1 takes target twitter screenname as command line argument today time strftime y m d csvfilepath parentpath handle _ today csv csvfilepath_clean parentpath handle _ today _clean csv authenticate app_key 25 alphanumeric characters app_secret 50 alphanumeric characters twitter twython app_key app_secret oauth_version 2 simple authentication object access_token twitter obtain_access_token twitter twython app_key access_token access_token adapted from http www craigaddyman com mining all tweets with python user_timeline twitter get_user_timeline screen_name handle count 200 if doing 200 or less just do this one line user_timeline twitter get_user_timeline screen_name handle count 1 get most recent tweet lis user_timeline 0 id 1 tweet id for most recent tweet only query as deep as necessary tweetsum user_timeline 0 user statuses_count cycles ceil tweetsum 200 if cycles 16 cycles 16 api only allows depth of 3200 so no point trying deeper than 200 16 time sleep 60 for i in range 0 cycles iterate through all tweets up to max of 3200 incremental twitter get_user_timeline screen_name handle count 200 include_retweets true max_id lis user_timeline extend incremental lis user_timeline 1 id 1 time sleep 90 90 second rest between api calls the api allows 15 calls per 15 minutes so this is conservative tw2csv tw2csv user_timeline csvfilepath clean the file and save it for i val in enumerate user_timeline user_timeline i user_screen_name user_timeline i user screen_name user_timeline i user_followers_count user_timeline i user followers_count user_timeline i user_id user_timeline i user id user_timeline i user_created_at user_timeline i user created_at if retweeted_status in user_timeline i keys user_timeline i rt_count user_timeline i retweeted_status retweet_count user_timeline i qt_id user_timeline i retweeted_status id user_timeline i rt_created user_timeline i retweeted_status created_at user_timeline i rt_user_screenname user_timeline i retweeted_status user name user_timeline i rt_user_id user_timeline i retweeted_status user id user_timeline i rt_user_followers user_timeline i retweeted_status user followers_count del user_timeline i retweeted_status if quoted_status in user_timeline i keys user_timeline i qt_created user_timeline i quoted_status created_at user_timeline i qt_id user_timeline i quoted_status id user_timeline i qt_text user_timeline i quoted_status text user_timeline i qt_user_screenname user_timeline i quoted_status user name user_timeline i qt_user_id user_timeline i quoted_status user id user_timeline i qt_user_followers user_timeline i quoted_status user followers_count del user_timeline i quoted_status if user_timeline i entities urls list for j val in enumerate user_timeline i entities urls urlj url_ str j user_timeline i urlj user_timeline i entities urls j expanded_url if user_timeline i entities user_mentions list for j val in enumerate user_timeline i entities user_mentions mentionj mention_ str j user_timeline i mentionj user_timeline i entities user_mentions j screen_name if user_timeline i entities hashtags list for j val in enumerate user_timeline i entities hashtags hashtagj hashtag_ str j user_timeline i hashtagj user_timeline i entities hashtags j text if user_timeline i coordinates is not none nonetype or dict user_timeline i coord_long user_timeline i coordinates coordinates 0 user_timeline i coord_lat user_timeline i coordinates coordinates 1 del user_timeline i coordinates del user_timeline i user del user_timeline i entities if place in user_timeline i keys nonetype or dict del user_timeline i place if extended_entities in user_timeline i keys del user_timeline i extended_entities if geo in user_timeline i keys del user_timeline i geo tw2csv tw2csv user_timeline csvfilepath_clean january 19 2016 at 8 10 am gr rss posts rss comments the culture geeks gr pkremp search for asa bayesian categorization causality cleaning culture culure diffusion economics economic sociology epistemology ethnomethodology financial crisis genetics graphs history imdb loops lyx macros netlogo networks perl phenomenology philosophy of science power analysis python r random variables regular expressions resampling satire scraping shell simulation sociology of organizations sociology of science socm176 st stata superstar text editor typesetting archives october 2025 july 2022 december 2021 november 2021 january 2021 april 2020 march 2020 april 2019 june 2018 february 2018 november 2017 september 2017 july 2017 march 2017 january 2017 june 2016 march 2016 february 2016 january 2016 november 2015 april 2015 october 2014 july 2014 november 2013 october 2013 september 2013 july 2013 june 2013 may 2013 april 2013 march 2013 february 2013 december 2012 november 2012 october 2012 september 2012 august 2012 july 2012 june 2012 march 2012 february 2012 january 2012 december 2011 november 2011 october 2011 september 2011 august 2011 july 2011 june 2011 may 2011 april 2011 march 2011 february 2011 january 2011 december 2010 november 2010 october 2010 september 2010 august 2010 july 2010 june 2010 may 2010 april 2010 march 2010 february 2010 january 2010 december 2009 november 2009 october 2009 september 2009 august 2009 july 2009 june 2009 may 2009 april 2009 march 2009 recent comments kenny on some notes on online teaching april 11 2020 data universe on edudammerung soc on eu migrants fr on eu migrants gabrielrossman on eu migrants jdgalt on eu migrants cogiddo on networks reading list lseltzer on strange things are afoot at the imdb gabrielrossman on strange things are afoot at the imdb lseltzer on strange things are afoot at the imdb blogroll data drip ssrc disgruntled sociologist econometrics by simulation eric booth s research notes mac research org theory org theory my posts permutations religion and other curiosities rense corten soc2econ social science statistics blog harvard sociological imagination stata daily stata on a mac stata tidbit of the week michael mitchell the stata blog the stata things gabi huiber what is the what jenn lena references resources california center for population research cpanda data ninja internet movie database raw text safari books gated ucla academic technology services unix and perl tutorial uc davis create a free website or blog at wordpress com subscribe subscribed code and culture join 128 other subscribers sign me up already have a wordpress com account log in now privacy code and culture subscribe subscribed sign up log in report this content view site in reader manage subscriptions collapse this bar design a site ...