Meta tags:
author=Jake VanderPlas;
description=;

Headings (most frequently used words):

python, in, the, packages, from, jupyter, waiting, time, paradox, or, why, is, my, bus, always, late, simulating, chutes, ladders, optimization, of, scientific, code, with, cython, ising, model, installing, notebook, exploring, line, lengths, exposing, private, dict, version, practical, guide, to, lomb, scargle, periodogram, group, by, scratch, triple, pendulum, chaos, reproducible, data, analysis,

Text of the page (most frequently used words):
the (108), and (61), this (40), that (28), you (25), #python (24), #jupyter (15), for (14), more (13), with (13), some (13), from (12), data (11), 2017 (11), read (10), can (10), time (10), version (10), are (9), how (9), but (9), notebook (8), post (8), ladders (8), minutes (7), into (7), lomb (7), scargle (7), all (7), not (7), when (7), limit (7), character (7), chutes (7), game (7), analysis (6), simulation (6), one (6), number (6), dictionary (6), what (6), distribution (6), like (6), bus (6), based (5), march (5), tweet (5), often (5), periodogram (5), blog (5), cython (5), waiting (5), pandas (4), code (4), out (4), twitter (4), which (4), group (4), users (4), where (4), see (4), every (4), optimization (4), pycon (4), packages (4), lengths (4), pip (4), conda (4), will (4), other (4), package (4), about (4), might (4), issue (4), paradox (4), your (4), videos (3), find (3), useful (3), exploration (3), reproducible (3), animation (3), whether (3), could (3), week (3), idea (3), operation (3), then (3), look (3), many (3), known (3), figure (3), found (3), scientific (3), last (3), written (3), most (3), has (3), questions (3), similar (3), paper (3), those (3), true (3), thu (3), later (3), private (3), million (3), just (3), were (3), dict (3), brandon (3), words (3), two (3), them (3), their (3), line (3), longer (3), 140 (3), help (3), using (3), intricacies (3), december (3), always (3), average (3), arrive (3), each (2), video (2), approximately (2), youtube (2), provide (2), interactive (2), common (2), question (2), get (2), though (2), trial (2), error (2), style (2), within (2), fri (2), matplotlib (2), naturally (2), immediately (2), wondered (2), result (2), pointed (2), original (2), library (2), chaos (2), triple (2), different (2), wed (2), split (2), value (2), apply (2), particular (2), aggregation (2), combine (2), via (2), api (2), today (2), take (2), ways (2), example (2), section (2), approaches (2), build (2), intuition (2), over (2), couple (2), cases (2), also (2), particularly (2), these (2), address (2), once (2), pages (2), popular (2), astropy (2), understanding (2), published (2), started (2), guide (2), seen (2), added (2), master (2), counter (2), been (2), there (2), raymond (2), dive (2), cpython (2), both (2), year (2), talks (2), dictionaries (2), talk (2), even (2), ideas (2), home (2), entertaining (2), may (2), statistics (2), wonder (2), display (2), another (2), day (2), 280 (2), would (2), people (2), near (2), installing (2), including (2), platform (2), tools (2), etc (2), well (2), familiar (2), consider (2), third (2), exactly (2), doing (2), operating (2), system (2), think (2), things (2), stop (2), quick (2), answer (2), decided (2), here (2), exhibit (2), problem (2), does (2), requires (2), itself (2), abstractions (2), leaky (2), source (2), software (2), following (2), iterative (2), process (2), ising (2), model (2), mon (2), under (2), square (2), length (2), roll (2), finally (2), 100 (2), basically (2), had (2), sometimes (2), random (2), reasonable (2), assumptions (2), reality (2), comes (2), buses (2), wait (2), why (2), pythonic (2), perambulations (2), past, available, alternatively, below, description, links, relevant, resources, playlist, notebooks, environment, progress, nonlinear, linear, organized, packaged, tested, series, presents, case, study, personally, approach, reproduce, simlulation, edit, reader, creator, 2016, posted, reddit, fermat, fermatslibrary, visualization, pendulums, very, slightly, initial, conditions, pic, com, ctiabfvwhw, earlier, made, rounds, features, nicely, demonstrates, chaotic, dynamical, systems, action, pendulum, benchmarks, basic, groups, subset, results, output, dataframe, generally, turn, type, implemented, effiently, concise, object, oriented, vein, want, fundamental, driven, otherwise, pattern, architypical, summation, shown, borrowed, science, handbook, grouping, best, grow, coding, spend, comparing, efficiency, various, implementing, algorithms, order, performance, building, blocks, ecosystem, scratch, years, implementations, recommend, wrote, subject, led, steady, trickle, emails, students, researchers, asking, advice, applying, interpreting, algorithm, astronomical, noticed, queries, tended, repeat, same, express, misconceptions, attempt, mere, includes, figures, full, references, bad, somewhat, pedagogical, marginally, lombscargle, shows, period, peak, simulated, dataset, observing, cadence, typical, ground, optical, astronomy, reveals, patterns, failure, method, discussed, explicitly, straightforward, explain, developed, detailed, discussion, preprint, manuscript, quickly, grew, medium, practical, ctypes, tutorial, course, saw, implicit, challenge, let, expose, internal, haven, interface, went, say, elsewhere, memory, change, incremented, means, come, back, know, modified, without, reading, maybe, its, hundreds, keys, values, increased, since, pep509, fascinating, guts, implementation, while, focuses, recent, improvements, user, facing, piece, mention, addition, aid, efforts, favorites, complementary, pair, regulars, hettinger, rhodes, followup, his, 2010, mighty, mightier, modern, confluence, dozen, great, got, sixth, was, wonderful, usual, weren, able, attend, wealth, informative, channel, exposing, spoiler, alert, they, details, reveal, insights, programming, habits, stylistic, conventions, communities, who, write, began, numpy, scipy, scikit, learn, anything, lines, maximum, characters, brought, mind, programmers, face, lives, suggested, pep8, takes, evidence, cram, thoughts, suggest, accommodate, desired, upped, single, purportedly, engineering, gist, english, language, tweets, roughly, log, normal, counts, except, spikes, interesting, november, exploring, focus, managers, exist, specific, cross, less, won, remarking, further, enstaller, homebrew, apt, yum, community, smooth, issues, changes, developers, ease, cognitive, load, background, abstraction, interacts, complexities, leaks, thus, better, understand, happening, working, second, bare, bones, general, install, works, first, wake, several, discussions, topic, colleagues, online, off, treat, depth, fundamentally, usually, rooted, fact, installer, points, than, being, used, simplest, contexts, arise, debugging, knowledge, installation, kernels, disconnected, shell, perrennial, stackoverflow, installed, now, import, said, any, manifest, tue, reason, record, screencasts, showing, application, easy, slow, intensive, numerical, operations, translating, helpful, bit, achieve, optimal, contain, lot, information, workflows, done, live, tutorials, illusion, dad, joke, primary, goal, originality, self, entertainment, anyone, else, finds, bonus, treatments, tweeted, twenty, morning, ourselves, endless, cycle, climbing, sliding, down, never, quite, reaching, final, end, wondering, much, expected, heavy, tails, succinctly, point, memoryless, effect, depends, only, modeled, markov, hit, least, head, clicked, weekend, myself, drawn, four, old, pleasure, playing, classic, kids, board, wherein, players, six, sided, die, advance, forward, through, squares, jump, ahead, avoiding, send, backward, glorified, walk, visual, aids, narrative, thrilling, she, having, fun, practicing, counting, learning, win, lose, gracefully, developing, requisite, skills, play, along, passionate, sports, fan, snakes, simulating, encountered, before, actually, match, explore, standpoint, probabilistic, arguments, real, arrival, city, seattle, hopefully, settle, naïvely, expect, coming, something, schedule, turns, reach, startling, conclusion, ready, catch, advertises, arrivals, glance, watch, note, seem, unlucky, frequently, commute, public, transit, situation, image, license, wikipedia, september, 2018, late, archive,

Text of the page (random words):
home pythonic perambulations pythonic perambulations about archive the waiting time paradox or why is my bus always late thu 13 september 2018 image source wikipedia license cc by sa 3 0 if you like me frequently commute via public transit you may be familiar with the following situation you arrive at the bus stop ready to catch your bus a line that advertises arrivals every 10 minutes you glance at your watch and note the time and when the bus finally comes 11 minutes later you wonder why you always seem to be so unlucky naïvely you might expect that if buses are coming every 10 minutes and you arrive at a random time your average wait would be something like 5 minutes in reality though buses do not arrive exactly on schedule and so you might wait longer it turns out that under some reasonable assumptions you can reach a startling conclusion when waiting for a bus that comes on average every 10 minutes your average waiting time will be 10 minutes this is what is sometimes known as the waiting time paradox i ve encountered this idea before and always wondered whether it is actually true how well do those reasonable assumptions match reality this post will explore the waiting time paradox from the standpoint of both simulation and probabilistic arguments and then take a look at some real bus arrival time data from the city of seattle to hopefully settle the paradox once and for all read more simulation statistics simulating chutes ladders in python mon 18 december 2017 this weekend i found myself in a particularly drawn out game of chutes and ladders with my four year old if you ve not had the pleasure of playing it chutes and ladders also sometimes known as snakes and ladders is a classic kids board game wherein players roll a six sided die to advance forward through 100 squares using ladders to jump ahead and avoiding chutes that send you backward it s basically a glorified random walk with visual aids to help you build a narrative thrilling but she s having fun practicing counting learning to win and lose gracefully and developing the requisite skills to be a passionate sports fan so i play along on the approximately twenty third game of the morning as we found ourselves in a near endless cycle of climbing ladders and sliding down chutes never quite reaching that final square to end the game i started wondering how much longer the game could last what is the expected length of a game how heavy are the tails of the game length distribution how succinctly could i answer those questions in python and then at some point it clicked chutes and ladders is memoryless the effect of a roll depends only on where you are not where you ve been and so it can be modeled as a markov process by the time we finally hit square 100 i basically had this blog post written at least in my head when i tweeted about this people pointed me to a number of similar treatments of chutes ladders so i m under no illusion that this idea is original think of this as a blog post version of a dad joke my primary goal is not originality but self entertainment and if anyone else finds it entertaining that s just an added bonus read more simulation animation optimization of scientific code with cython ising model mon 11 december 2017 python is quick and easy to code but can be slow when doing intensive numerical operations translating code to cython can be helpful but in most cases requires a bit of trial and error to achieve the optimal result cython s tutorials contain a lot of information but for iterative workflows like optimization with cython it s often useful to see it done live for that reason i decided to record some screencasts showing this iterative optimization process using an ising model as an example application read more jupyter cython simulation installing python packages from a jupyter notebook tue 05 december 2017 in software it s said that all abstractions are leaky and this is true for the jupyter notebook as it is for any other software i most often see this manifest itself with the following issue i installed package x and now i can t import it in the notebook help this issue is a perrennial source of stackoverflow questions e g this that here there another this one that one and this etc fundamentally the problem is usually rooted in the fact that the jupyter kernels are disconnected from jupyter s shell in other words the installer points to a different python version than is being used in the notebook in the simplest contexts this issue does not arise but when it does debugging the problem requires knowledge of the intricacies of the operating system the intricacies of python package installation and the intricacies of jupyter itself in other words the jupyter notebook like all abstractions is leaky in the wake of several discussions on this topic with colleagues some online exhibit a exhibit b and some off i decided to treat this issue in depth here this post will address a couple things first i ll provide a quick bare bones answer to the general question how can i install a python package so it works with my jupyter notebook using pip and or conda second i ll dive into some of the background of exactly what the jupyter notebook abstraction is doing how it interacts with the complexities of the operating system and how you can think about where the leaks are and thus better understand what s happening when things stop working third i ll talk about some ideas the community might consider to help smooth over these issues including some changes that the jupyter pip and conda developers might consider to ease the cognitive load on users this post will focus on two approaches to installing python packages pip and conda other package managers exist including platform specific tools like yum apt homebrew etc as well as cross platform tools like enstaller but i m less familiar with them and won t be remarking on them further read more jupyter conda pip exploring line lengths in python packages thu 09 november 2017 this week twitter upped their single tweet character limit from 140 to 280 purportedly based on this interesting analysis of tweet lengths published on twitter s engineering blog the gist of the analysis is this english language tweets display a roughly log normal distribution of character counts except near the 140 character limit at which the distribution spikes the analysis takes this as evidence that twitter users often cram their longer thoughts into the 140 character limit and suggest that a 280 character limit would more naturally accommodate the distribution of people s desired tweet lengths this immediately brought to mind another character limit that many python programmers face in their day to day lives the 79 character line limit suggested by python s pep8 style guide limit all lines to a maximum of 79 characters i began to wonder whether popular python packages e g numpy scipy pandas scikit learn matplotlib astropy display anything similar to what is seen in the distribution of tweet lengths spoiler alert they do and the details of the distribution reveal some insights into the programming habits and stylistic conventions of the communities who write them read more python data statistics exposing python 3 6 s private dict version fri 26 may 2017 i just got home from my sixth pycon and it was wonderful as usual if you weren t able to attend or even if you were you ll find a wealth of entertaining and informative talks on the pycon 2017 youtube channel two of my favorites this year were a complementary pair of talks on python dictionaries by two pycon regulars raymond hettinger s modern python dictionaries a confluence of a dozen great ideas and brandon rhodes the dictionary even mightier a followup of his pycon 2010 talk the mighty dictionary raymond s is a fascinating dive into the guts of the cpython dict implementation while brandon s focuses more on recent improvements in the dict s user facing api one piece both mention is the addition in python 3 6 of a private dictionary version to aid cpython optimization efforts in brandon s words pep509 added a private version number every dictionary has a version number and elsewhere in memory a master version counter and when you go and change a dictionary the master counter is incremented from a million to a million and one and that value a million and one is written into the version number of that dictionary so what this means is that you can come back later and know if it s been modified without reading maybe its hundreds of keys and values you just look and see if the version has increased since the last time you were there he later went on to say the version number is internal i haven t seen an interface for users to get to it which of course i saw as an implicit challenge so let s expose it read more python tutorial ctypes a practical guide to the lomb scargle periodogram thu 30 march 2017 this week i published the preprint of a manuscript that started as a blog post but quickly out grew this medium understanding the lomb scargle periodogram figure 24 from understanding the lomb scargle periodogram the figure shows the true period vs the periodogram peak for a simulated dataset with an observing cadence typical of ground based optical astronomy the simulation reveals common patterns of failure of the lomb scargle method that are not often discussed explicitly but are straightforward to explain based on the intuition developed in the paper see section 7 2 for a detailed discussion over the last couple years i ve written a number of python implementations of the lomb scargle periodogram i d recommend astropy s lombscargle in most cases today and also wrote a marginally popular blog post and somewhat pedagogical paper on the subject this all has led to a steady trickle of emails from students and researchers asking for advice on applying and interpreting the lomb scargle algorithm particularly for astronomical data i noticed that these queries tended to repeat many of the same questions and express some similar misconceptions and this paper is my attempt to address those once and for all in a mere 55 pages which includes 26 figures and 4 full pages of references so it s not all that bad read more lomb scargle group by from scratch wed 22 march 2017 i ve found one of the best ways to grow in my scientific coding is to spend time comparing the efficiency of various approaches to implementing particular algorithms that i find useful in order to build an intuition of the performance of the building blocks of the scientific python ecosystem in this vein today i want to take a look at an operation that is in many ways fundamental to data driven exploration the group by otherwise known as the split apply combine pattern an architypical example of a summation group by is shown in this figure borrowed from the aggregation and grouping section of the python data science handbook the basic idea is to split the data into groups based on some value apply a particular operation to the subset of data within each group often an aggregation and then combine the results into an output dataframe python users generally turn to the pandas library for this type of operation where it is is implemented effiently via a concise object oriented api read more pandas python benchmarks triple pendulum chaos wed 08 march 2017 earlier this week a tweet made the rounds which features a video that nicely demonstrates chaotic dynamical systems in action a visualization of chaos 41 triple pendulums with very slightly different initial conditions pic twitter com ctiabfvwhw fermat s library fermatslibrary march 5 2017 edit a reader pointed out that the original creator of this animation posted it on reddit in 2016 naturally i immediately wondered whether i could reproduce this simlulation in python this post is the result read more matplotlib animation simulation reproducible data analysis in jupyter fri 03 march 2017 jupyter notebooks provide a useful environment for interactive exploration of data a common question i get though is how you can progress from this nonlinear interactive trial and error style of exploration to a more linear and reproducible analysis based on organized packaged and tested code this series of videos presents a case study in how i personally approach reproducible data analysis within the jupyter notebook each video is approximately 5 8 minutes the videos are available in a youtube playlist alternatively below you can find the videos with some description and links to relevant resources read more data pandas jupyter past