Meta tags:

Headings (most frequently used words):

what, exactly, does, do, step, more, python, for, taking, advantage, of, sabbatical, so, how, about, some, maps, even, possibilities, computer, program, scholars, this, historian, parsing, the, unparseable, converting, semi, structured, document, into, files,

Text of the page (most frequently used words):
the (720), and (372), you (346), that (224), with (160), for (147), python (136), but (113), can (103), data (98), have (86), your (84), from (81), more (76), code (74), this (72), all (71), are (67), text (63), some (57), what (56), into (56), #jostwald (53), will (52), like (52), not (46), com (45), one (45), wordpress (44), just (44), those (43), which (43), then (43), other (42), want (41), each (39), there (38), history (37), use (36), was (36), their (36), https (35), out (35), list (34), 2018 (32), they (32), could (32), how (32), about (31), time (31), digital (31), few (30), does (29), has (29), them (29), even (29), information (28), word (28), any (27), get (27), would (27), make (27), 2019 (26), historians (26), over (25), png (25), well (25), most (25), another (25), learning (25), than (24), libraries (24), document (24), also (23), create (23), when (22), its (22), early (21), modern (21), need (21), where (21), been (21), already (21), things (21), excel (21), computer (21), see (20), now (20), map (20), own (20), programming (20), analysis (20), had (19), don (19), specific (19), different (19), bit (18), maps (18), these (18), many (18), after (18), much (18), world (18), might (18), way (18), same (18), cleaning (18), words (18), two (17), online (17), were (17), spanish (17), least (17), using (17), because (17), lots (17), know (17), probably (17), historical (17), book (17), schools (17), maybe (17), only (17), results (17), dictionary (17), year (16), learn (16), find (16), lot (16), above (16), several (16), language (16), file (16), documents (16), next (15), process (15), back (15), catalan (15), used (15), his (15), first (15), should (15), clean (15), start (15), school (15), programs (15), files (15), convert (15), new (14), siege (14), who (14), once (14), did (14), people (14), work (14), still (14), exactly (14), available (14), entries (14), part (13), off (13), think (13), better (13), through (13), here (13), mentioned (13), analyze (13), source (13), look (13), war (12), barcelona (12), our (12), every (12), case (12), separate (12), spent (12), since (12), years (12), aha (12), big (12), sure (12), means (12), folio (12), military (11), http (11), 0000 (11), jpg (11), screenshot (11), various (11), long (11), done (11), important (11), yak (11), hand (11), set (11), steps (11), between (11), future (11), really (11), dataset (11), faculty (11), add (11), run (11), easily (11), jupyter (11), read (11), easy (11), doing (11), tools (11), yet (11), etc (11), format (11), automate (11), web (11), letters (11), feed (10), together (10), take (10), spain (10), full (10), number (10), very (10), particularly (10), allows (10), haven (10), department (10), based (10), structured (10), project (10), sabbatical (10), perform (10), program (10), may (9), medieval (9), someone (9), including (9), figure (9), before (9), include (9), course (9), thing (9), count (9), making (9), basic (9), given (9), question (9), change (9), qgis (9), features (9), projects (9), easier (9), example (9), pandas (9), past (9), others (9), string (9), person (9), page (9), date (9), require (9), item (9), errors (9), around (8), doesn (8), let (8), access (8), always (8), small (8), day (8), say (8), point (8), particular (8), authors (8), check (8), databases (8), yourself (8), hard (8), info (8), type (8), search (8), processing (8), without (8), types (8), common (8), sources (8), numbers (8), value (8), journal (8), research (8), zotero (8), help (7), last (7), fortress (7), beginning (7), give (7), right (7), following (7), 2017 (7), museum (7), getting (7), real (7), fact (7), shaving (7), again (7), names (7), something (7), keep (7), looking (7), whatever (7), order (7), allow (7), graph (7), sorts (7), got (7), lists (7), old (7), syntax (7), books (7), little (7), parse (7), notebook (7), library (7), pretty (7), nested (7), multiple (7), website (7), enough (7), dozen (7), machine (7), smh (6), wouldn (6), view (6), why (6), thought (6), sense (6), therefore (6), imagine (6), guess (6), end (6), else (6), rather (6), semi (6), never (6), line (6), note (6), actually (6), almost (6), having (6), post (6), wikipedia (6), recent (6), combine (6), details (6), usable (6), step (6), hosler (6), area (6), network (6), explore (6), field (6), software (6), table (6), complicated (6), original (6), level (6), likely (6), github (6), csv (6), pdf (6), everything (6), curve (6), regex (6), result (6), download (6), learned (6), months (6), become (6), form (6), natural (6), automatically (6), looks (6), good (6), tasks (6), historian (6), working (6), low (6), millner (6), content (6), nlp (6), changes (6), notes (6), created (6), numerous (6), ocr (6), nouns (6), versions (6), texts (6), functions (6), reminder (5), interested (5), finding (5), via (5), blog (5), respond (5), budget (5), worth (5), especially (5), fortunately (5), wife (5), possibly (5), less (5), tell (5), decided (5), starting (5), dozens (5), google (5), main (5), ways (5), difference (5), similar (5), madrid (5), interesting (5), three (5), english (5), during (5), four (5), across (5), until (5), whether (5), simple (5), stuff (5), scraping (5), deal (5), comments (5), further (5), generally (5), directory (5), took (5), database (5), extract (5), pre (5), mapping (5), standard (5), stuck (5), eye (5), encoding (5), utf (5), sample (5), object (5), websites (5), pages (5), put (5), being (5), needed (5), resulting (5), coordinates (5), started (5), within (5), formatting (5), delimiter (5), name (5), 000 (5), terms (5), myself (5), reading (5), parsing (5), regular (5), index (5), ocred (5), free (5), installing (5), science (5), interactive (5), along (5), coding (5), hanging (5), campaign (5), dates (5), structure (5), entry (5), record (5), copy (5), great (5), bunch (5), place (5), individual (5), thousands (5), places (5), output (5), letter (5), manipulate (5), tcp (5), held (4), topic (4), come (4), definitely (4), visit (4), otherwise (4), events (4), citadel (4), force (4), fort (4), 1640 (4), while (4), site (4), sieges (4), either (4), said (4), forward (4), decades (4), heritage (4), catalonia (4), final (4), creating (4), overall (4), somebody (4), instead (4), ended (4), service (4), city (4), succession (4), whole (4), currently (4), scratch (4), john (4), wasn (4), american (4), aug (4), battles (4), location (4), curious (4), wars (4), hopefully (4), open (4), play (4), chart (4), knowledge (4), series (4), move (4), reason (4), images (4), phds (4), phd (4), departments (4), extracted (4), examples (4), questions (4), sometimes (4), offer (4), print (4), understand (4), packaged (4), method (4), manually (4), enter (4), couple (4), certain (4), notebooks (4), best (4), txt (4), editor (4), basics (4), sound (4), humanities (4), task (4), works (4), methods (4), drawn (4), primary (4), digitized (4), computers (4), taking (4), realized (4), geopandas (4), though (4), import (4), days (4), advanced (4), key (4), isn (4), published (4), textual (4), practically (4), ones (4), quantitative (4), anaconda (4), version (4), marlborough (4), devonthink (4), loop (4), cleaned (4), camp (4), transcription (4), expressions (4), precise (4), problem (4), phrases (4), fruit (4), dirty (4), kind (4), proper (4), entities (4), workflow (4), mean (4), makes (4), rename (4), made (4), eebo (4), scholars (4), rear (4), mirror (4), sale (4), coming (3), panels (3), european (3), write (3), conference (3), state (3), karl (3), left (3), culture (3), build (3), article (3), 1705 (3), below (3), correct (3), named (3), initial (3), built (3), community (3), however (3), central (3), recently (3), referendum (3), previously (3), party (3), performing (3), rest (3), languages (3), southern (3), duke (3), third (3), allied (3), fall (3), king (3), memory (3), him (3), close (3), aren (3), familiar (3), unlike (3), today (3), display (3), salses (3), france (3), away (3), pain (3), later (3), present (3), undoubtedly (3), century (3), mention (3), gives (3), mostly (3), down (3), flags (3), far (3), week (3), sparql (3), listed (3), sheets (3), try (3), complete (3), downloaded (3), period (3), thu (3), often (3), eventually (3), yarn (3), kinds (3), ask (3), useful (3), classification (3), lines (3), pattern (3), according (3), geographical (3), didn (3), seem (3), teaching (3), offered (3), degrees (3), layout (3), tool (3), domain (3), matplotlib (3), nltk (3), includes (3), rerun (3), summary (3), update (3), trying (3), finish (3), formats (3), gets (3), call (3), teach (3), discuss (3), arguments (3), talk (3), error (3), wrong (3), baby (3), strings (3), dictionaries (3), such (3), general (3), prose (3), possible (3), readable (3), ability (3), depending (3), courses (3), automating (3), figured (3), values (3), converting (3), items (3), larger (3), bold (3), asked (3), share (3), piece (3), beginner (3), stage (3), tuesday (3), secondary (3), usually (3), entity (3), package (3), academic (3), visualization (3), install (3), specialized (3), focus (3), hundreds (3), visualizations (3), literally (3), thanks (3), tutorials (3), mass (3), tip (3), requires (3), colleague (3), ago (3), posts (3), journals (3), hit (3), bits (3), statistical (3), function (3), returns (3), split (3), notice (3), delete (3), won (3), standardizing (3), preprocessing (3), nor (3), separated (3), adding (3), top (3), relational (3), assuming (3), sophisticated (3), searching (3), compared (3), nice (3), paragraph (3), collection (3), publication (3), compare (3), quickly (3), image (3), institutional (3), matrix (3), answer (3), provides (3), corpus (3), variable (3), repeat (3), folder (3), rules (3), semester (3), stats (3), entire (3), both (3), edit (3), feature (3), behind (3), commands (3), worry (3), microsoft (3), vep (3), 6828 (2), deadline (2), proposals (2), october (2), love (2), alma (2), mater (2), considered (2), panelists (2), wed (2), 1714 (2), crypt (2), detail (2), kapuzinergruft (2), 6810 (2), tolédo (2), castilian (2), tramping (2), olympics (2), ditch (2), taken (2), commemorating (2), random (2), overlooking (2), submission (2), reapers (2), revolt (2), centuries (2), human (2), rights (2), torture (2), political (2), surprise (2), castle (2), exhibition (2), montjuïc (2), interactivity (2), says (2), talking (2), opening (2), maintained (2), art (2), fortresses (2), hill (2), park (2), interest (2), pronounced (2), fast (2), chapter (2), provide (2), account (2), known (2), government (2), support (2), population (2), against (2), non (2), symbol (2), resistance (2), relationship (2), national (2), tourism (2), saw (2), philippe (2), berwick (2), fighting (2), everywhere (2), besiege (2), austrian (2), mind (2), clearly (2), earlier (2), western (2), minute (2), went (2), british (2), experience (2), relevant (2), franco (2), thirty (2), keeping (2), track (2), gateway (2), model (2), trouble (2), nationalism (2), golden (2), video (2), flag (2), region (2), prehistory (2), life (2), stick (2), sticks (2), moderns (2), section (2), guessing (2), speculation (2), historiographical (2), experienced (2), virtual (2), reality (2), moving (2), impression (2), bread (2), infinitely (2), league (2), arrived (2), second (2), 6853 (2), internet (2), goes (2), europe (2), eastern (2), methodology (2), meantime (2), query (2), wikidata (2), skills (2), scientist (2), cool (2), gotta (2), granular (2), speed (2), times (2), faithful (2), spreadsheets (2), warfare (2), consider (2), bonus (2), 6861 (2), sweater (2), sweaters (2), surprising (2), scale (2), possibilities (2), specialization (2), fewer (2), equivalent (2), specialist (2), ranking (2), patterns (2), draw (2), changed (2), rank (2), size (2), extraction (2), passes (2), major (2), appear (2), older (2), description (2), mapped (2), boring (2), per (2), giving (2), sized (2), seeing (2), too (2), feel (2), quick (2), wheel (2), easiest (2), tend (2), additional (2), seaborn (2), spacy (2), true (2), reuse (2), chunks (2), figuring (2), show (2), received (2), tables (2), anymore (2), certainly (2), gold (2), replicability (2), redo (2), harder (2), issues (2), weird (2), issue (2), plain (2), anything (2), saving (2), starts (2), objects (2), modify (2), fit (2), doc (2), realize (2), ideally (2), comes (2), resources (2), techniques (2), parameters (2), told (2), wanted (2), closer (2), frustrating (2), tells (2), patient (2), understanding (2), abbreviations (2), depends (2), alone (2), powerful (2), increasingly (2), tested (2), consistent (2), unstructured (2), yes (2), weeks (2), user (2), statistics (2), extra (2), imported (2), cell (2), combined (2), 600 (2), respective (2), organize (2), layers (2), expand (2), variables (2), character (2), organization (2), discovered (2), downloadable (2), self (2), lesson (2), friendly (2), fine (2), tough (2), sorting (2), finally (2), quite (2), humans (2), kindly (2), replicate (2), ememh (2), illustrate (2), doctoral (2), shave (2), thus (2), opportunity (2), calendar (2), analyzing (2), creation (2), chair (2), niche (2), genealogy (2), ner (2), recognition (2), outdated (2), dedicated (2), struggle (2), qualitative (2), visualizing (2), recreate (2), playing (2), humanist (2), youtube (2), conferences (2), critical (2), able (2), title (2), ultimately (2), overview (2), continue (2), 6911 (2), allowing (2), oyster (2), extracting (2), deep (2), keywording (2), cycle (2), parsed (2), metadata (2), modules (2), binary (2), strip (2), associated (2), single (2), empty (2), programmer (2), bbedit (2), officers (2), seems (2), lucky (2), scanned (2), missing (2), changing (2), spelling (2), contractions (2), spaces (2), specifically (2), towards (2), tabs (2), provided (2), argue (2), subject (2), discrete (2), keyword (2), diary (2), explain (2), lack (2), limited (2), focusing (2), dec (2), 6940 (2), email (2), asking (2), frequency (2), criteria (2), duration (2), similarly (2), groups (2), itself (2), class (2), students (2), parameter (2), input (2), desired (2), topics (2), looked (2), algorithms (2), schedule (2), collecting (2), student (2), requests (2), interfaces (2), played (2), identify (2), context (2), calculate (2), 1700 (2), insert (2), records (2), fields (2), relationships (2), nodes (2), topological (2), properties (2), international (2), visual (2), bokeh (2), spreadsheet (2), plot (2), charts (2), hyphenated (2), mistake (2), capitalize (2), uses (2), corrections (2), hundred (2), high (2), originals (2), amount (2), punctuation (2), dealing (2), ideas (2), previous (2), request (2), api (2), brute (2), apis (2), application (2), downloading (2), linked (2), copying (2), paste (2), pay (2), indexed (2), adobe (2), indexing (2), importance (2), giant (2), digitally (2), fantasy (2), custom (2), scenes (2), beyond (2), plenty (2), pass (2), block (2), principles (2), attention (2), matter (2), quote (2), perfect (2), algorithm (2), expertise (2), emphasize (2), purchase (2), environments (2), automated (2), control (2), upon (2), increase (2), inputs (2), manipulations (2), wish (2), nothing (2), lets (2), layer (2), browser (2), variety (2), medium (2), fri (2), 6968 (2), 18th (2), 6992 (2), summer (2), discoveries (2), mon (2), 6995 (2), england (2), transcribed (2), installed (2), rohan (2), cover (2), 7001 (2), volume (2), countries (2), skulking (2), holes (2), corners (2), panel, paper, putting, ohio, columbus, theme, soldiers, civilians, cauldron, although, papers, germane, friends, propose, contact, seeking, forum, society, annual, jul, 20180528_124558, 20180528_122354, 20180528_121045, 20180528_120430, 20180527_132526, 20180527_132507, gds_conflicte005, 20180527_132326, spainanimatedsinglecolor, fortresssalses2, 20180527_131657, 20180527_131227, 20180526_111028, 810px, flag_of_catalonia, svg, caught, dead, hosted, 1992, cue, archer, holdover, team, butts, dry, montjuic, locations, measurements, establish, meter, 1792, plaque, slightly, depressing, facts, occupying, stone, garrison, intimidate, town, coincidence, castilians, broke, bombard, restless, barcelonans, civil, became, violations, imprisonment, execution, prisoners, ends, quotation, universal, declaration, freedom, seek, involved, 1697, moat, monsters, guarding, ravelin, nope, guérite, smell, urine, photos, zoom, descriptions, underwent, street, splay, embrasures, outward, funnel, shot, fortifications, themselves, flower, beds, aerial, bearings, complex, surrounded, cable, car, funicular, walk, sites, mont, jew, jewish, burial, ground, outcropping, mediterranean, transformed, lighthouse, successive, formidable, ethnographic, autonomous, northeast, implicitly, recognised, 1978, constitution, disagreements, tax, situation, hostility, independence, soared, polarising, outcome, remains, uncertain, november, 2014, ruling, rebranded, binding, symbolic, pressure, politically, sensitive, significance, strained, johannes, venetia, identity, observations, edited, catherine, palmer, jacqueline, tivers, routledge, abstract, descriptive, noticed, witness, hot, press, study, catalonians, ours, explanation, oppressive, neighbor, defense, peeved, ordered, commander, slaughter, rebellion, capitulation, 11th, september, privileges, revoked, downhill, digress, surrender, meant, french, loaned, exposing, decade, fourth, brief, unsuccessful, attack, bent, eluded, large, exhibit, catastrophe, largely, 1713, crazy, catalans, kept, candidate, carlos, iii, abandoned, imperial, throne, 1711, becoming, holy, roman, emperor, appears, former, digs, remain, refused, acknowledge, felipe, stayed, death, witnessed, viewing, vienna, tomb, celebrates, 1706, liberation, iberian, front, portuguese, border, animated, holdings, iberia, narrative, forces, managed, repulse, occupations, recapture, territory, battlefield, victory, almansa, obligingly, facilitated, reconquest, choosing, abandon, allies, contingent, captured, brihuega, museo, del, ejército, museu, història, catalunya, irregular, miquelets, rose, taxes, governance, olivares, home, songs, conflict, anthem, els, segadors, reaper, languedoc, visited, excursion, pyrenean, pirates, fun, came, origin, legend, senyera, 897, grateful, dipped, fingers, mortally, wounded, blood, dragged, shield, thanking, requested, killers, reward, differently, apparently, dating, 13c, prefer, stories, pen, sword, parchment, quill, ink, indeed, austrians, tale, crimson, streaked, half, waterfront, randomly, selected, destination, covers, hear, mannequin, cavemen, jump, 19th, rooms, kingdoms, principalities, shake, proverbial, catalonian, speaking, struck, predominance, anglo, bookstores, geoffrey, parker, lynch, elliot, keegan, translations, histories, heavy, seeming, silence, senior, native, imperialism, smartphone, headphone, fated, modernist, casa, bottló, photo, vertigo, anyway, attractions, mandatory, tourist, dollars, visiting, gaudí, standing, narrow, tower, waiting, wanting, contemplate, sagrada, familia, cathedral, tourists, must, proximity, bake, bâtards, rioting, spanked, liverpool, champions, survived, ugly, brooklynite, incident, train, northern, capital, seemed, sort, disagreement, parts, country, clue, proudly, hung, apartment, balconies, outnumbering, spying, vestiges, conquistadors, visits, castile, trip, side, conquerors, conquered, sat, wd_battles_15275_map, lights, decide, belligerent, locating, efficiency, oriented, mindset, wherein, collects, ancient, explores, durations, attributes, graphics, accurate, www, gokhan, collate, heart, desires, wants, bureaucracy, inject, entered, factual, approximation, refine, beta, defined, preparing, appeal, skulkers, assist, quixotic, quest, robust, envision, posting, aspects, combats, queries, history_programs_phd_medieval_map, history_programs_phd_map, history_programs_map, history_programs_ba_map, phrase, describe, alludes, backward, shaved, colorful, analogy, reilly, whetted, appetite, individuals, broader, trends, profession, occasionally, micro, combining, proclivities, relative, subspecialties, specializations, news, report, carnegie, feeder, areas, accepted, categories, granted, distribution, percent, assistant, associate, inclined, double, expanded, messier, missed, anomalies, serve, dump, initially, guessed, grouped, percentages, formatted, degree, chore, category, rule, style, 150, dot, composer, 602, equally, connections, potential, spending, necessary, afraid, revert, hasn, invented, member, consuming, newer, graphing, ultra, customizable, textblob, simpler, foundational, packages, reinvent, sharing, hardest, repurposing, versus, response, emem, requesting, gun, transparency, compatible, discover, consistently, rotate, applied, stack, overflow, gobbledygook, strange, characters, slashes, upside, marks, universe, advice, save, bom, assume, switching, latin, alphabets, frustrated, specs, spend, excluding, tragically, commonly, worst, readability, ditto, slope, steep, flattening, ahead, required, going, rereading, introductory, chapters, intermediate, purposes, typing, consult, blogs, videos, poke, comfortable, documentation, except, fail, silently, fails, necessarily, programmers, failing, success, message, brings, humbling, rarely, safety, net, huge, thinking, finished, comparable, readily, freely, unfortunately, integers, stand, concepts, tips, barely, thoughts, guides, bad, occasional, prediction, hat, bring, becomes, machines, excuse, predicted, 60s, masses, shown, power, semesters, cartography, grad, export, pipeline, tweaked, importing, exporting, overwriting, revision, eliminated, geocoding, procedure, customizability, exported, numeric, perfected, concatenation, extraneous, rows, faster, geocoded, latitude, longitude, ezgeocoder, messy, hours, simplest, test, cheated, replace, font, fonts, 361, current, flagship, migrated, emailed, charge, generously, reported, pays, included, total, relearned, umpteenth, constructing, managing, printed, problematic, row, medievalist, indicated, column, filtering, understandable, bloomington, match, algorithmically, horrible, preserving, trained, digested, nowadays, refer, tidy, link, obliged, straightforward, famous, hmm, sounds, worse, frequent, walking, backwards, figurative, month, figures, implications, april, argues, dearth, cartographical, metaphor, motive, plans, savers, marched, taught, fleeting, fragile, quarter, vard2, gate, gramps, outwit, hub, stanford, edinburgh, geoparser, local, quirks, scrape, fancy, tried, analyses, multicolored, beast, rely, editors, fill, gaps, intro, gruntwork, imagination, fortuitously, oodles, pycon, pydata, pylondon, pyberlin, challenging, visible, installation, command, activate, conda, environment, tutorial, realistic, customized, literate, ben, pauley

Text of the page (random words):
cumented my own research journey away from word documents to digital tools see devonthink tag i even created modified a few applescripts to automate this very problem in devonthink in a rudimentary way one for example can explode i e parse a document by creating a new document for every paragraph in the starting document nice but it can be better python to the rescue the problem lots of text files of notes and transcriptions of letters but not very granular and therefore not easily compared requiring lots of wading through dross with the likelihood of getting distracted this is particularly a problem is you re searching for common terms or phrases that appear in lots of different letters wouldn t it be nice if you could filter your search by date or some other piece of metadata the solution use python code to parse the documents say individual letters or entries for a specific day into separate files making it easy to hone in on the precise subject or period you re searching for as well as precise tagging and keywording step 1 for proof of concept i started with a transcription of a campaign journal kindly provided me by lawrence smith in a word document i m sure you have dozens of similar files he was faithful in his transcription even to the extent of mimicking the layout of the information on the page with the use of tabs spaces and returns great for format fidelity but not great for easily extracting important information particularly if you want for example june to be right next to 20th instead of on the line below separated by a bunch of officers names maastricht and london are actually a bit confusing because i m pretty sure the place names after the dates are that day s passwords at least that s what i ve seen in other campaign journals that some of the entries explicitly list a camp location reinforces my speculation of course people can argue about which information is important which is yet another reason why it s best if you can do this yourself aside as you are examining the layout of the document to be parsed you should also have one eye towards the future in this case that means swearing to yourself that i will never again take unstructured notes that will require lots of regex for parsing in other words if you want to make your own notes usable by the computer and don t already have a sophisticated database set up for data entry use a consistent format scheme across sources that is easy to parse automatically for example judicious use of tabs and unique formatting step 2 clean up the text specifically make the structure more standardized so different bits of info can be easily identified and extracted for this document that means making sure each first line only consists of the date and camp location when available that each entry is separated by two carriage returns and adding a distinctive delimiter in this case two colons between each folio because you ll ultimately have the top level of your structured data organized by folio with entries multiple entries per folio this is a one to many relationship for those of you familiar with relational databases like access cleaning the text can be easily done with regex allowing you to cycle through and make the appropriate changes in minutes assuming you know your regular expressions that is the result looks like this note that this stage is not changing the content i e it s not preprocessing the text doing things like standardizing spelling or expanding contractions or what have you nor did i bother getting rid of extra spaces etc those can be stripped with python as needed for this specific document note as well that some of the formatting for the officers of the day is muddled the use of curly brackets seems odd which might equal loss of information but if that info s important you should take care to figure out how to robustly record it at the transcription stage if you re relying on the kindness of others beggars can t be choosers but if you re lucky you happen to have a scanned reproduction of a partial copy of this journal from another source which tells you what information might be missing from the transcription camp journal sample of above from british library add ms 61404 f 45 you probably could do this standardizing within your python code in jupyter notebook but i find it easier to interact with regex in my text editor bbedit your mileage may vary step 3 once you get the text in a standard format like the above you read it into python and convert it into a structured data set if you don t know python at all the following details won t make sense so go read up on some python one of the big hurdles for the neophyte programmer as i ve discovered over and over is to see how the different pieces fit together into a whole so that s what i ll focus on here in a nutshell the code does the following after you ve cleaned up the structure of the original document in your text editor read the file into memory as one big long string perform any other cleaning of the content you want then you perform several passes to massage the string into a dictionary with a nested list for the values there may be a better more efficient way to do this in fewer lines but my beginner code does it in three main steps convert the document to a list splitting each item at the f delimiter now you have a list with each folio as a separate item always look at your results for some reason the first item of the resulting list is empty it doesn t seem to be an encoding error so just delete that item from the list before moving on now read the resulting list items into a python dictionary with the dictionary key the folio number and all of the entries on the folio as the value of that folio use the as the delimiter here with the following line of code a comprehension as they call it notice how the strip and split methods are chained together performing multiple changes on the item object in that single bit of code now you use a for loop to parse each value into separate list items using the other delimiter of n n two returns between entries using the string of the value since otherwise it s a list item and the strip and split methods only work on strings this gives you a dictionary with the folio as the dict key and the value is now a nested list with each of the entries associated with its folio as a separate item as you can see with folio 40 s four entries that s pretty much it now you have a structure for your text congratulations your text has become data or data ish at least the resulting python dictionary allows you to search any folio and it will return a list of all the letters entries on that folio you can loop through all those entries and perform some function on with them so that s a good thing to pickle i e write it to a binary file so that it can be easily read back as a python dictionary later on once you have your data structured and maybe add some more metadata to it you can do all sorts of analysis with all of python s statistical nlp and visualization modules but if you are still straddling the devonthink python divide like i am then you ll also want to make these parsed bits available in devonthink add a bit of code to write out each dictionary key value pair to a separate file and you end up with several hundreds of files each file will have only the content for that specific entry making it easy to precisely target your search and keywording the last thing you want to do is cycle through several dozen hits in a long document for that one hit you re actually looking for that s it entry of may 8th 1705 in its own file the beauty is that you can add more to the code try extracting the dates and camps change what information you want to include in the filename etc depending on the structure of the data you re using you might need to nest dictionaries or lists several layers deep as discussed in my aha example but that s the basics pretty easy once you figure it out that is even better now you can run the same code with a few minor tweaks on all of those other collections of letters and campaign journals that you have allowing you to combine newhailes entries with deane s and millner s and marlborough s letters and the world s your oyster but like any oyster it takes a little work opening that sucker not that i like oysters https jostwald wordpress com 2018 12 03 from historical source to historical data feed 2 6911 jostwald newhailes_sample1 png early_formatting_ideas png newhailes_sample png newhailes_sample_bl_add61404 png list_items png dictionary png dictionary_nested_list png newhailes_finder_folder png newhailes_sample_entry png where the historians are 2017 https jostwald wordpress com 2018 09 18 where the historians are 2017 https jostwald wordpress com 2018 09 18 where the historians are 2017 comments wed 19 sep 2018 02 14 40 0000 http jostwald wordpress com p 6861 shaving the yak is a phrase used to describe the process of programming it alludes to the fact that you often have to take two or more steps backward in order to eventually move one step forward you want a sweater so first you need to get some yarn but to do that you have to and eventually you find yourself shaving a yak the reason why you even consider shaving a yak is that once you ve shaved said yak you now have lots of yarn which allows you to make many sweaters this colorful analogy has a surprising number of online images and even an o reilly book it s a thing i have been doing a lot of digital yak shaving over the past four months come to think of it most of my blog posts consist of yak shaving so if you re interested in learning to code with python but not sure whether it s worth it or if you just want to read an overview of how i used python and qgis to create a map like this from a big word document then continue reading taking advantage of sabbatical on a meta level i knew that if i were ever to make any sweaters with computer code i would have to shave that particular yak this sabbatical multiple factors converged first this year off would be my one opportunity in the next seven years to delve into python and to learn whatever else would set up my research and digital history teaching several years ago i remember reading a digital historian s blog post on the cool stuff he was doing with some advanced digital tool and i thought yeah but who has time to do all that thumbs pointing at self this guy admittedly i have marlborough s big book of battles working title to finish but some of the coding i learn can help with that ultimately it s about priorities and honestly the world will not end if it s denied one more book on marlborough within the next year and the book will be a lot better with the python tools i m learning second and fortuitously beginner friendly python has arrived literally within the past few years thanks to anaconda jupyter notebooks oodles of websites including the programminghistorian org dozens of books and dozens of youtube tutorials from recent pycon pydata pylondon pyberlin conferences there is a critical mass and you can learn much of it on your own even if you don t take any of the available online courses don t get me wrong learning python has still been challenging the most frustrating part is getting everything set up whether it s installing the right python version in the right directory tip start with a clean install of python 3 using anaconda installing third party python libraries visible to your anaconda installation tip do it from the command line and activate the conda environment first or getting your data into a usable format for analysis see below it also requires a learning process to move from the basic tasks you can perform with a jupyter tutorial downloaded from github or from a website or book to more realistic and therefore more complicated customized tasks that you really want to perform with your data right now i wouldn t have been able to do much of what i wanted in python certainly not within a few months of beginning to learn it without the help of my programming wife and a python literate colleague in eastern s english department ben pauley so there s definitely a learning curve python has become the go to language for text cleaning natural language processing visualizations along with r and increasingly basic machine learning and did i mention it also has mapping libraries like geopandas python will do practically any academic gruntwork a humanist can imagine computers would do and then some and i say that as a humanist with a bit of an imagination having taught my intro to digital history course once already i learned that online tools are fleeting and fragile and will only do a third of what you want them to do you can usually find small niche programs that give you the ability to do another quarter of what you want things like vard2 and gate cleaning ocred text and gramps genealogy and outwit hub web scraping and stanford ner named entity recognition of text and edinburgh geoparser ner mapping they can be very useful but they can also become outdated especially the free online ones and you may well have as much trouble installing them on your local machine as something like python so given the fact that python will do almost everything just about every other dedicated software package will do again i m talking data science and academic tasks here and it will require programming on your part and since everything in python is free and since there are so many python libraries that will perform most of these tasks why struggle installing a dozen different programs and learn each of their quirks just to do one specific thing in each one program to scrape data from a website another program for cleaning data one for doing quantitative analysis another for qualitative analysis or natural language processing another for visualizing your results in a fancy chart excel does not count another for creating a network graph of your data yet another program to map your data still another to create an interactive visualization that you can explore python can do them all don t get me wrong sometimes it will actually be easier to just install a specialized program but you ll only know after you ve tried to recreate part of it in python so after playing around with most of the other programs i decided to focus my struggle on installing and learning one tool python and then use its hundreds of libraries to help me do any number of analyses python s a pretty big yak but there s a lot of multicolored yarn on that beast and you can always rely on your text editors excel and other niche programs to fill in any gaps until you learn more about python and its libraries thus means opportunity motive learn python i already have plans for a few dozen projects for python to automate everything from simple time savers like calendar look up last tuesday we marched what date was last tuesday to analyzing my prose to analyzing primary or secondary sources to ...