Meta tags:
description= Using large digital libraries to advance literary history;
Headings (most frequently used words):
the, of, ai, to, time, can, and, in, what, we, fiction, estimate, culture, posts, stone, shell, more, upside, make, us, intelligence, be, next, does, this, gpt, measure, why, how, well, do, mapping, is, cultural, technology, editing, marionette, theater, interesting, will, overconfident, should, artificial, person, shaped, language, models, predict, twist, story, have, any, relation, human, uncertainty, reason, not, trust, so, could, learn, from, predicting, three, pages, liberally, educated, students, need, than, consumers, save, matters, about, writing, at, price, using, passage, bag, words, methods, train, llms, no, model, latent, spaces, science, hasn, prepared, imagine, machine, learning, pagination, some, better, recent, categories, blogroll, archives, super, for, as, doesn, anyone, less, depressed, wait, good, idea, references,
Text of the page (most frequently used words):
the (769), and (390), that (326), this (166), but (160), for (147), not (132), #models (125), are (120), can (113), about (112), with (107), will (97), have (90), language (85), more (81), they (80), human (79), from (78), what (69), model (67), like (62), because (60), don (58), than (56), you (56), learning (54), has (49), one (49), our (48), new (48), #intelligence (47), could (46), how (45), think (45), when (43), was (43), other (42), may (42), there (42), people (41), time (41), need (39), also (39), their (39), all (39), post (39), just (38), some (36), would (35), good (34), culture (33), writing (32), know (32), them (32), social (31), even (31), fiction (30), make (30), only (30), who (30), been (30), https (29), machine (28), students (28), use (28), which (28), now (27), cultural (27), might (27), see (27), gpt (27), 2021 (26), things (26), space (26), find (26), way (26), where (25), these (25), different (25), reason (25), out (25), those (24), say (24), own (24), problem (24), why (24), should (23), want (23), readers (23), questions (23), still (23), using (23), artificial (22), between (22), data (22), meaning (22), story (21), its (21), here (21), org (21), passage (21), 2023 (20), work (20), next (20), instance (20), hard (20), had (20), question (19), into (19), measure (19), books (18), ways (18), let (18), words (18), something (18), much (18), many (18), get (18), technology (18), were (18), then (18), already (17), july (17), text (17), effects (17), plot (17), world (17), possible (17), same (17), author (17), point (16), research (16), understand (16), itself (16), does (16), researchers (16), each (16), parrots (16), often (16), minutes (16), blog (15), change (15), any (15), thinking (15), process (15), mean (15), years (15), actually (15), map (15), little (15), large (15), self (15), word (15), history (14), person (14), upside (14), take (14), range (14), science (14), yet (14), approach (14), neural (14), task (14), since (14), first (14), answer (14), june (13), 2017 (13), 2024 (13), project (13), after (13), right (13), agents (13), very (13), give (13), over (13), step (13), clip (13), end (13), stochastic (13), paper (13), feel (13), 2012 (12), scale (12), deep (12), predict (12), recent (12), better (12), beings (12), lot (12), whether (12), sense (12), look (12), making (12), really (12), date (12), across (12), specific (12), assignments (12), personality (12), comments (11), january (11), 2011 (11), analysis (11), topic (11), modeling (11), categories (11), literary (11), thing (11), clear (11), maybe (11), create (11), going (11), course (11), two (11), kind (11), less (11), doing (11), most (11), image (11), past (11), bad (11), least (11), enough (11), suspense (11), thought (11), challenge (11), his (11), above (11), apple (11), february (10), march (10), 2016 (10), 2018 (10), 2025 (10), interesting (10), expect (10), future (10), explore (10), latent (10), eventually (10), patterns (10), training (10), toward (10), keep (10), tedunderwood (10), force (10), positive (10), generative (10), used (10), debate (10), openai (10), several (10), prediction (10), help (10), style (10), code (10), references (10), both (10), back (10), working (10), ability (10), september (9), 2015 (9), historical (9), theater (9), reading (9), too (9), open (9), idea (9), through (9), perhaps (9), imagine (9), read (9), produce (9), general (9), run (9), goal (9), doi (9), knowledge (9), technologies (9), likely (9), single (9), while (9), effect (9), doesn (9), week (9), passages (9), started (9), learn (9), ask (9), chapter (9), report (8), com (8), october (8), 2013 (8), humanities (8), character (8), posts (8), tags (8), probably (8), well (8), instead (8), vast (8), stories (8), possibility (8), art (8), wrong (8), play (8), another (8), summary (8), collective (8), makes (8), century (8), pages (8), however (8), side (8), argument (8), believe (8), prompt (8), fully (8), underwood (8), critical (8), long (8), conversation (8), example (8), represent (8), recognize (8), short (8), part (8), describe (8), experiment (8), predictions (8), found (8), page (8), reference (8), real (8), balance (8), professors (8), happen (8), bluesky (8), consciousness (8), content (7), stone (7), december (7), august (7), 2014 (7), 2020 (7), search (7), methods (7), trend (7), transformer (7), shaped (7), marionette (7), seven (7), put (7), boring (7), few (7), number (7), ever (7), either (7), kinds (7), form (7), others (7), novel (7), system (7), being (7), important (7), behavior (7), yes (7), hand (7), sentence (7), humans (7), images (7), old (7), cost (7), your (7), scene (7), arxiv (7), aspects (7), picture (7), big (7), potential (7), agree (7), train (7), public (7), disciplines (7), fact (7), pretend (7), matters (7), asked (7), three (7), chatgpt (7), elapsed (7), narrative (7), press (7), construct (7), require (7), start (7), getting (7), sometimes (7), her (7), days (7), confidence (7), try (7), estimate (7), teach (7), save (7), trained (7), robin (7), education (7), vision (7), surprise (7), anthropic (7), depressed (7), write (6), shell (6), april (6), attention (6), problems (6), twist (6), quickly (6), always (6), worlds (6), version (6), ends (6), sort (6), library (6), door (6), whole (6), seem (6), early (6), suspect (6), add (6), dall (6), writers (6), seems (6), place (6), field (6), especially (6), great (6), similar (6), nothing (6), means (6), prefer (6), made (6), hasn (6), did (6), conference (6), value (6), ted (6), abs (6), rather (6), book (6), anyone (6), down (6), usually (6), decide (6), tools (6), mapping (6), produced (6), surprising (6), role (6), around (6), understanding (6), mostly (6), llms (6), simple (6), api (6), method (6), provide (6), before (6), assistant (6), reasons (6), easier (6), experience (6), described (6), every (6), themselves (6), github (6), best (6), voice (6), term (6), university (6), uncertainty (6), bum (6), someone (6), novels (6), breaks (6), moral (6), super (6), media (6), aesthetic (6), persona (6), required (5), reader (5), account (5), victorian (5), article (5), digital (5), 19c (5), teaching (5), analytics (5), saying (5), become (5), useful (5), reality (5), storytelling (5), matter (5), surprised (5), pattern (5), never (5), game (5), exploring (5), support (5), characters (5), styles (5), users (5), starting (5), trying (5), without (5), individual (5), created (5), larger (5), mental (5), resemble (5), title (5), gets (5), claim (5), interested (5), doubt (5), test (5), isn (5), bert (5), certain (5), job (5), haven (5), carbon (5), feature (5), dreams (5), artists (5), white (5), material (5), risks (5), dangers (5), dangerous (5), consequences (5), biases (5), pictures (5), position (5), behind (5), explicitly (5), historians (5), conflict (5), offer (5), texts (5), response (5), given (5), forms (5), far (5), earlier (5), advance (5), second (5), stable (5), jane (5), statistical (5), currently (5), remember (5), measuring (5), called (5), compare (5), core (5), challenges (5), day (5), final (5), address (5), edit (5), reply (5), characterize (5), set (5), average (5), axis (5), until (5), student (5), principles (5), home (5), valid (5), sound (5), suggest (5), distance (5), continue (5), chang (5), 900 (5), necessarily (5), installments (5), hound (5), difficult (5), askell (5), personhood (5), claude (5), power (5), invented (5), void (5), bots (5), stateful (5), name (4), log (4), november (4), creative (4), era (4), theory (4), genre (4), strategies (4), phenomenon (4), humanists (4), libraries (4), decade (4), direction (4), cyberpunk (4), hidden (4), wider (4), moment (4), analogous (4), news (4), helpful (4), allow (4), everything (4), works (4), familiar (4), interactive (4), fictional (4), tend (4), thousands (4), friends (4), understood (4), having (4), plans (4), sure (4), thoughts (4), reach (4), assume (4), presumably (4), living (4), impressive (4), order (4), below (4), reasoning (4), evidence (4), limited (4), access (4), trust (4), knows (4), four (4), turn (4), bit (4), guess (4), hope (4), prepared (4), low (4), against (4), mark (4), journal (4), perspective (4), handle (4), notes (4), trans (4), bender (4), proceedings (4), spaces (4), imagined (4), intent (4), merely (4), suggested (4), easy (4), essay (4), closely (4), air (4), failed (4), respond (4), internet (4), looks (4), unreal (4), engine (4), consider (4), didn (4), discourse (4), critics (4), scholars (4), universities (4), tech (4), seemed (4), move (4), nsf (4), begin (4), perspectives (4), cover (4), develop (4), multiple (4), fine (4), structure (4), corpora (4), reflect (4), web (4), wide (4), exactly (4), genres (4), wouldn (4), published (4), mind (4), pose (4), tends (4), center (4), tasks (4), yauney (4), david (4), running (4), spite (4), agreement (4), she (4), memory (4), hours (4), hour (4), gave (4), results (4), slightly (4), autonomy (4), tune (4), estimates (4), bag (4), experiments (4), valuable (4), focused (4), cases (4), compared (4), provided (4), tell (4), assumptions (4), honest (4), instruction (4), year (4), thirty (4), explain (4), asks (4), won (4), anything (4), sciences (4), mla (4), cccc (4), thesis (4), gentle (4), effectively (4), break (4), assistance (4), essays (4), asking (4), ourselves (4), insight (4), feelings (4), initial (4), advice (4), guide (4), comes (4), ago (4), expected (4), jobs (4), automated (4), implies (4), exciting (4), talk (4), review (4), encouraging (4), serial (4), accuracy (4), roughly (4), baskervilles (4), holmes (4), avoid (4), names (4), scaling (4), birds (4), sky (4), theatrical (4), mask (4), signals (4), last (4), altman (4), persons (4), ease (4), retrieved (4), golden (4), gate (4), sentimental (4), marionettes (4), agent (4), email (3), view (3), site (3), sign (3), wordpress (3), 2026 (3), 2019 (3), planet (3), dan (3), correlation (3), mining (3), diction (3), interpretive (3), criticism (3), discovery (3), building (3), liberally (3), educated (3), consumers (3), overconfident (3), distant (3), emergence (3), babel (3), places (3), invent (3), almost (3), fear (3), worry (3), doors (3), piece (3), thread (3), corporations (3), autonomous (3), loosely (3), robots (3), alien (3), taught (3), within (3), examples (3), genuinely (3), original (3), authors (3), universe (3), bigger (3), call (3), borrowed (3), maps (3), interactively (3), build (3), worse (3), reproduce (3), solving (3), primates (3), life (3), predicting (3), nineteenth (3), twitter (3), boxes (3), mistake (3), leave (3), off (3), smart (3), chimpanzee (3), individuals (3), quite (3), complex (3), association (3), certainly (3), taking (3), supposed (3), fill (3), released (3), clearly (3), verbal (3), obvious (3), energy (3), importantly (3), weigh (3), footprint (3), argue (3), successful (3), spatial (3), charlie (3), edu (3), generation (3), house (3), national (3), black (3), net (3), katherine (3), foundation (3), transparency (3), systems (3), enjoy (3), shared (3), concerns (3), note (3), gives (3), allowed (3), analogy (3), political (3), everyone (3), expertise (3), negative (3), absolutely (3), existing (3), impossible (3), attempt (3), practice (3), together (3), features (3), particular (3), visual (3), although (3), graduate (3), tuning (3), purpose (3), companies (3), differences (3), publication (3), service (3), somehow (3), free (3), bias (3), typically (3), immediate (3), templates (3), creatively (3), necessary (3), study (3), consistent (3), definition (3), captioned (3), phrase (3), traces (3), communication (3), hear (3), state (3), simply (3), computer (3), discuss (3), event (3), roundtable (3), consensus (3), stanford (3), implications (3), simon (3), willison (3), greg (3), computational (3), measured (3), 1980 (3), accurate (3), interpretable (3), defined (3), relatively (3), concepts (3), thousand (3), seen (3), show (3), needs (3), assistants (3), bamman (3), explains (3), nature (3), man (3), night (3), again (3), close (3), whatever (3), narrator (3), childhood (3), replies (3), amount (3), turbo (3), plus (3), ambiguity (3), providing (3), high (3), learned (3), front (3), count (3), yesterday (3), ideal (3), notice (3), cautious (3), needed (3), explicit (3), events (3), spoken (3), passed (3), weeks (3), user (3), interaction (3), durations (3), counting (3), precisely (3), feels (3), length (3), represents (3), entity (3), specify (3), documents (3), based (3), corey (3), changed (3), fall (3), willing (3), once (3), got (3), reflection (3), preserve (3), solve (3), ideas (3), further (3), wheel (3), along (3), fake (3), spin (3), cheat (3), metacognition (3), suggesting (3), midjourney (3), harder (3), angle (3), suggestions (3), beginning (3), wanted (3), putting (3), technical (3), lives (3), html (3), concept (3), allows (3), framed (3), available (3), talking (3), case (3), poses (3), plato (3), novelty (3), family (3), serialization (3), ryan (3), anders (3), group (3), arguing (3), narratives (3), turns (3), expectation (3), mystery (3), installment (3), encourage (3), divergence (3), summaries (3), tempting (3), written (3), interfaces (3), laugh (3), cringe (3), telling (3), products (3), provides (3), caddies (3), interacting (3), argues (3), abstract (3), demo (3), pygmalion (3), chatbot (3), evans (3), obsessed (3), flight (3), spectrum (3), alignment (3), passive (3), accessible (3), passively (3), textbook (3), points (3), wait (3), kapoor (3), farrell (3), gopnik (3), monosemanticity (3), singularity (3), editing (3), manipulate (3), printing (3), modes (3), durable (3), protean (3), legal (3), freedom (3), journey (3), kleist (3), constructed (3), distasteful (3), advantage (3), protagonist (3), inner (3), penny (3), network (3), morality (3), moltbook (3), animals (3), familiars (3), subscribed (2), subscribe (2), 2010 (2), google (2), cohen (2), american (2), under (2), politics (2), uncategorized (2), statistics (2), literature (2), reproducibility (2), replication (2), poetic (2), pace (2), disciplinary (2), collection (2), 20c (2), gender (2), broad (2), older (2), extrapolating (2), deepfakes (2), fears (2), borges (2), optimistically (2), money (2), portal (2), fantasy (2), 2020s (2), subgenre (2), sinister (2), etc (2), led (2), considering (2), decades (2), inscrutable (2), transforming (2), containing (2), robot (2), collaborative (2), script (2), protein (2), folding (2), intelligent (2), music (2), activity (2), expectations (2), increasing (2), fan (2), developed (2), variety (2), arthur (2), devoted (2), safe (2), children (2), symbolic (2), coordinates (2), desires (2), structures (2), bananas (2), advances (2), shrug (2), none (2), envision (2), includes (2), jerome (2), lived (2), true (2), generates (2), expressing (2), intention (2), inference (2), implied (2), contest (2), pursuing (2), unfamiliar (2), manner (2), broadly (2), learns (2), photo (2), television (2), detail (2), plausible (2), recently (2), unlikely (2), threat (2), ready (2), struggling (2), lately (2), said (2), contemporary (2), efforts (2), cause (2), reduction (2), actively (2), agenda (2), renewable (2), sham (2), deceptive (2), campaign (2), mashable (2), skeptical (2), lies (2), institute (2), skeptically (2), limits (2), interest (2), berkeley (2), gray (2), chen (2), ilya (2), sutskever (2), zero (2), resource (2), gov (2), wang (2), autoregressive (2), martin (2), daniel (2), jacob (2), eisenstein (2), william (2), corr (2), size (2), improve (2), discussed (2), bommasani (2), 100 (2), soni (2), lauren (2), abolitionist (2), networks (2), activist (2), newspapers (2), culturalanalytics (2), list (2), production (2), bode (2), non (2), oct (2), recap (2), gebru (2), mcmillan (2), major (2), 610 (2), 623 (2), 1145 (2), 3442188 (2), 3445922 (2), acm (2), fairness (2), accountability (2), information (2), reframe (2), imply (2), illuminate (2), fascinating (2), transform (2), deeply (2), contrast (2), connects (2), plan (2), intelligences (2), tropes (2), conscious (2), various (2), care (2), ethical (2), dilemma (2), controlled (2), movable (2), type (2), war (2), meant (2), shape (2), common (2), observation (2), machines (2), angel (2), diffusion (2), region (2), fun (2), themes (2), differently (2), embeddings (2), dizzying (2), devoid (2), rhetorical (2), broader (2), outside (2), alternatives (2), forward (2), danger (2), computing (2), resources (2), huggingface (2), helping (2), languages (2), distinguish (2), flexibility (2), associate (2), statement (2), challenging (2), requires (2), correctly (2), contrary (2), acceptable (2), audiences (2), delicate (2), share (2), token (2), practices (2), disappointing (2), starts (2), instructions (2), select (2), dictionary (2), named (2), subjects (2), tendency (2), grounded (2), communicative (2), 616 (2), intentional (2), sequences (2), defines (2), trajectory (2), parts (2), anyway (2), natural (2), changing (2), academics (2), credit (2), discussion (2), applications (2), princeton (2), prepare (2), follows (2), mimno (2), away (2), loud (2), automatically (2), document (2), fast (2), modest (2), measurement (2), error (2), confident (2), separate (2), systematically (2), coming (2), showing (2), addressed (2), signs (2), six (2), heard (2), dreaming (2), child (2), incident (2), couch (2), hands (2), water (2), remembers (2), 10080 (2), moderate (2), passing (2), interpretability (2), myself (2), prompts (2), cheaper (2), correlated (2), 121 (2), inter (2), closer (2), took (2), months (2), llm (2), main (2), pointed (2), complete (2), added (2), allowing (2), extremely (2), answers (2), capacity (2), assess (2), quinn (2), dombrowski (2), query (2), morning (2), window (2), coffee (2), dressed (2), seeing (2), car (2), ego (2), responding (2), slowly (2), twice (2), pretty (2), encouraged (2), responses (2), five (2), summarize (2), include (2), taken (2), description (2), dialogue (2), units (2), times (2), understands (2), higher (2), transformed (2), lower (2), definitely (2), estimating (2), labeled (2), predicted (2), hundred (2), showed (2), manually (2), literally (2), says (2), conclude (2), 250 (2), calculated (2), passes (2), minute (2), automate (2), addresses (2), hundreds (2), watkins (2), marc (2), 1984 (2), rudy (2), scarborough (2), joint (2), overview (2), annoying (2), patching (2), pedagogy (2), sufficient (2), capable (2), framing (2), came (2), fresh (2), puts (2), emphasis (2), turning (2), critique (2), normative (2), venture (2), meta (2), adapting (2), intellectual (2), advanced (2), gather (2), interpret (2), rehearse (2), processes (2), originally (2), simulate (2), assignment (2), simulating (2), replicate (2), attitude (2), wisdom (2), optimistic (2), elegiac (2), opportunity (2), spins (2), weird (2), cynical (2), foreground (2), externalize (2), focus (2), imitate (2), trace (2), adjusting (2), practical (2), college (2), avoids (2), class (2), exams (2), solution (2), daughter (2), team (2), price (2), imperfect (2), department (2), critically (2), admittedly (2), illustration (2), variance (2), basic (2), level (2), 2030s (2), courses (2), recommend (2), consumer (2), looking (2), collar (2), choose (2), emerge (2), solved (2), discover (2), today (2), coherent (2), warhol (2), tobin (2), harvard (2), smuts (2), 2009 (2), paradox (2), sinykin (2), sap (2), choi (2), acl (2), meeting (2), piper (2), revelation (2), mcgrath (2), modernist (2), haugtvedt (2), magazines (2), periodicals (2), garrett (2), dames (2), twenty (2), speak (2), beekman (2), woman (2), previously (2), presented (2), multi (2), chunks (2), appropriate (2), related (2), crude (2), trusting (2), cliffhanger (2), standard (2), ensure (2), epistemic (2), rhythms (2), doctor (2), footprints (2), answered (2), predictable (2), steel (2), inferences (2), draw (2), diss (2), cosine (2), check (2), supply (2), masked (2), moreover (2), flying (2), blind (2), shouldn (2), studying (2), reveals (2), tended (2), actual (2), drawn (2), johnson (2), claiming (2), correlate (2), sees (2), ernie (2), happens (2), sesame (2), street (2), beyond (2), impulse (2), warner (2), presents (2), spike (2), jonze (2), mallarmé (2), stéphane (2), neuromancer (2), amanda (2), richie (2), stuart (2), creeped (2), reminds (2), full (2), truth (2), anthropomorphize (2), pure (2), sounding (2), interface (2), impersonal (2), lie (2), reminders (2), owners (2), sounded (2), scarlett (2), johansson (2), eyes (2), opinions (2), determine (2), react (2), grumpy (2), anthropocentric (2), gendered (2), laughter (2), creature (2), emotion (2), caddy (2), quietly (2), extension (2), personal (2), siri (2), coalesce (2), woo (2), sam (2), insist (2), fair (2), risk (2), ambitious (2), downsides (2), lean (2), league (2), boots (2), frame (2), monolith (2), stack (2), overflow (2), threads (2), repos (2), entry (2), daunting (2), prove (2), 1990s (2), program (2), disagree (2), overconfidence (2), bugs (2), evolve (2), strong (2), adding (2), couldn (2), wilder (2), bryanwilder (2), ebooks (2), 2022 (2), jowett (2), phaedrus (2), narayanan (2), normal (2), mollick (2), manning (2), shalizi (2), circuits (2), index (2), extracting (2), sonnet (2), samaltman (2), dark (2), depressing (2), sounds (2), costs (2), benefits (2), purely (2), sharpen (2), swing (2), forth (2), grok (2), heat (2), eliminate (2), load (2), bearing (2), ideological (2), weapons (2), chance (2), habits (2), produces (2), outweigh (2), sustained (2), similarly (2), adjectives (2), directions (2), neurons (2), vertigo (2), bridge (2), able (2), among (2), sref (2), stylistic (2), dimensional (2), vector (2), james (2), fail (2), survey (2), skeptics (2), potentially (2), mature (2), generally (2), stage (2), orders (2), magnitude (2), greater (2), ill (2), substitute (2), false (2), agi (2), generalizable (2), entities (2), path (2), capability (2), independent (2), command (2), external (2), else (2), decisions (2), agency (2), nerds (2), recognized (2), naive (2), tree (2), grace (2), personas (2), shtick (2), struggle (2), sentimentality (2), wince (2), voyage (2), puppet (2), sitting (2), lap (2), pinocchio (2), schiller (2), staged (2), humanity (2), flaw (2), actors (2), alas (2), weakness (2), inhabit (2), accounts (2), riff (2), dancer (2), gesture (2), truly (2), aesthetically (2), discomfort (2), cookie (2), puppets (2), cheesy (2), morally (2), maltese (2), audience (2), patron (2), memento (2), menu (2), website, comment, loading, collapse, bar, manage, subscriptions, privacy, join, 596, subscribers, top, archives, scottbot, irregular, seasr, presentations, sapping, parezco, digo, monk, meandre, installation, ngram, viewer, geoffrey, nunberg, digitally, assisted, collocates, english, 1810, artfl, 18thconnect, blogroll, licensed, commons, attribution, sharealike, unported, license, visualization, unsupervised, undigitized, sociology, romantic, representativeness, reception, poetry, ocr, correction, nonconsumptive, ngrams, methodology, metadata, math, lsa, linguistics, impressionistic, historicism, comparison, bayesian, 18c, rss, dataset, 1700, 1922, structuralist, theories, computers, instability, standards, outlines, jdh, newer, pagination, movements, qanon, eager, parallel, armored, endlessly, impenetrable, servant, refuses, pod, bay, heroines, chalk, chosen, leading, bookshelf, personally, mashup, wardrobes, encylopedias, tlön, futuristic, eric, desmazieres, emotions, servile, excite, terrify, labyrinth, infinite, eye, alternate, shelves, god, extrapolation, weirder, journalism, invariably, invite, servants, antagonists, terminator, hal, paternal, condescension, dread, reactions, domains, creators, collaboratively, emerges, exploration, solitary, undertaken, geniuses, scratch, origin, folk, tales, rearranging, moves, riffing, established, involved, creating, replace, interconnected, proposed, grown, ups, settings, illustrations, rackham, graphic, welcome, collaborators, happening, sites, tens, competitors, enter, startup, motto, subreddit, dungeon, endeavors, including, needn, attempts, imagining, collaborator, amino, acids, infancy, improved, foolish, ignore, clues, bizarrely, parochial, writer, stacking, supposedly, overrated, buzz, relax, realist, goals, achieve, paragraph, wander, understandably, ultimately, wolfgang, köhler, 1925, mentality, apes, driven, mario, klingemann, byline, extrapolate, expressed, symbols, opinion, probability, sentences, shivery, crossed, significant, gary, marcus, ernest, davies, correlations, fusing, abstractions, lemur, hold, umbrella, surprisingly, 1910s, output, baby, daikon, radish, tutu, walking, dog, involving, underway, developments, imaginary, translate, descriptions, opera, zombie, apocalypse, receding, fight, rescue, replicants
Text of the page (random words):
e of a story from beginning to end i made a coarse attempt of that kind in a blog post a few years ago other articles use better methods and give us new ways to think about form mcgrath et al 2018 piper et al 2023 but how closely does the pace of verbal change correlate with readers experience of uncertainty or surprise we don t know autoregressive language models offer a tempting new angle on this problem because they re trained specifically to predict how a given text will continue intuitively it feels like we could measure the predictability of a plot by first asking a model to continue the story and then measuring the divergence between predicted continuation and real text even if this isn t exactly how readers form expectations and experience surprise it might begin to give us some leverage on the question researchers have run a loosely similar experiment on very short stories contributed by experimental subjects sap et al 2020 but scaling that up to published novels poses a challenge for one thing language models may not be equally good at imitating every style a contemporary model s failure to predict the next sentence by jane austen might just mean that it s bad at channeling the regency so to factor style out of the question let s ask a model to predict what will happen in say the next three pages of a story and then compare those predictions to its own summaries of the pages when it sees them readers of a certain age will recognize this as a game ernie invites bert to play on sesame street ernie asks bert what happens next in this picture bert anticipates that the man will step in the pail and disaster will ensue to spell the method out more precisely we move through a novel roughly 900 words at a time on each pass we give a language model both a recap of earlier events and a new 900 word passage we ask the model to summarize the new passage and also ask it to predict what will happen next then we compare its prediction to the summary it generates when it actually sees the next passage and measure cosine distance between the two sentence embeddings a large distance means the model did a poor job of predicting what would happen next does this have any relation to human uncertainty i m not claiming that this is a good model of the way readers experience plot we don t have a good model of that yet the more appropriate question to ask is does this correlate at all with anything human readers do we can check by asking a reader to do the same thing read roughly 900 word passages and make predictions about the future then we can compare the human reader s predictions to automatically generated summaries passages were drawn from now in november by josephine johnson and murder is dangerous by saul levinson n 51 passages pearson s r 41 p 01 human predictions are more variable in quality than the model s when i did this for two novels that were complete blanks to me my predictions tended to diverge from the actual course of the story in roughly the same places where the model found prediction difficult so there does seem to be some relationship between a language model s in ability to see what s coming and a human reader s the image above also reveals that there are broad consistent differences between books for both people and models some stories are easier to predict than others a reason not to trust this readers of this story may already anticipate the next twist which is that of course we shouldn t use llms to study uncertainty because these models have already read many of the books we re studying and will presumably already know the plot this is a particularly nasty problem because we don t have a list of the books commercial models were trained on we re flying blind but before we give up let s test how much of a problem this really poses researchers at berkeley have defined a convenient test of the extent to which a model has memorized a book chang et al 2023 in essence they ask the model to fill in missing names running this test chang et al find that gpt 4 remembers many books in detail moreover its ability to fill in masked names correlates with its accuracy on certain other tasks like its ability to estimate date of publication this could be a problem for questions about plot to avoid this problem and also save money i ve been using gpt 3 5 which chang et al find is less prone to memorize books but is that enough to address the problem let s check below i ve plotted the average divergence between prediction and summary for 25 novels on the y axis and gpt 3 5 s ability to supply masked names in those texts on the x axis if memorization was making prediction more accurate we would expect to see a negative correlation predictions divergence from summaries should go down as name_cloze accuracy goes up the y axis is average cosine distance between prediction and summary x axis is gpt 3 5 s accuracy on the name cloze test defined in chang et al 25 books is not enough for a conclusive answer but so far i cannot measure any pattern of that kind if anything there is a faint trend in the opposite direction in an ideal world researchers would use language models trained on open data sets that they know and control but until we get to an ideal world it looks like it may be possible to run proof of concept experiments with things like gpt 3 5 at least if we avoid extremely famous books scrutinizing the image above readers will probably notice that the most predictable book in this sample was zoya by danielle steel although steel has a reputation that may encourage disparaging inferences see dan sinykin big fiction for why i don t think we re in a position to draw those inferences yet the local rhythms that make prediction possible across three pages are not necessarily what critics mean when they use predictable to diss a book so what could we learn from predicting the next three pages to consider one possible payoff it might give us a handle on the way chapter breaks and other divides structure the epistemic rhythms of fiction for instance many readers have noticed that the installments of novels originally published in magazines tend to end with an explicit mystery to ensure that you keep reading haugtvedt 2016 and beekman 2017 in the first installment of arthur conan doyle s hound of the baskervilles which covers two chapters watson and holmes learn about a legendary curse that connects the family of the baskervilles to a fiendish hound in the final lines of the first installment holmes asks the family doctor about footprints found near the body of sir charles baskerville a man s or a woman s holmes asks the doctor s voice sank almost to a whisper as he answered mr holmes they were the footprints of a gigantic hound end installment novels don t have soundtracks but unexplained suggestive new information is as good as a sting bum bum bum the hound of the baskervilles by sidney paget 1902 it appears that we can measure this cliffhanger effect the serial installments of the hound of the baskervilles often end with a moment of heightened mystery at least if inability to predict the next three pages is any measure of mystery when we measure predictive accuracy throughout the story making four different passes to ensure we have roughly 900 word chunks aligned with all the chapter breaks we find that predictions are farther from reality at the ends of serial installments there is no similar effect at other chapter breaks the mean for breaks at serial installments is more than one standard deviation above the mean for other chapter breaks in spite of tiny n this is actually p 05 now this is admittedly a cherry picked example so far i have only looked at seven novels where we can distinguish the ends of serial installments from other kinds of chapter break using data from warhol et al and i don t see this pattern in all of them so i m not yet making any historical argument about serialization and the rise of the cliffhanger i m just suggesting that it s the kind of question someone could eventually address using this method a doctoral student could do it for instance with a locally hosted model i don t recommend doing it with gpt 3 5 because i dropped 150 or so on this post and that might add up across a dissertation some initial tests suggest to me that this approach will produce results significantly different than we re getting with lexical methods since i m explicitly encouraging people to run with this let me also say that someone actually writing a paper using this method might want to tinker with several things before trusting it measuring the distance between the embedding of one prediction sentence and one summary sentence is a crude way to measure expectation and surprise readers don t necessarily form a single expectation about plot maybe it would be better to model expectation as a range of possibility related to this models may need to be nudged to speculate and not just predict that current actions will continue 900 word chunks may not be the only appropriate scale of analysis when readers talk about narrative surprise they re often thinking about larger arcs like who will he marry or who turns out to be the murderer we need a way to handle braided narratives where each chapter is devoted to a different group of characters garrett 1980 in a multi plot story the b or c plot will often not continue across a chapter break but we re in a multi plot narrative ourselves so those problems may be solved by a different group of characters this was just a blog post to share an idea and get people arguing about it tune in next time for our thrilling conclusion bum bum bum code and data used for this post are available on github the ideas discussed here were previously presented in paris at a workshop on ai for the analysis of literary corpora and in copenhagen in a conference on generative methods in the social sciences and humanities i d like to thank the organizers of those events esp thierry poibeau anders munk and rolf lund for stimulating conversation and also many people in attendance especially david bamman lynn cherny and meredith martin in writing code to query the openai api i borrowed snippets from quinn dombrowski and also of course from gpt 4 itself oh brave new world c my thinking about 19c serialization was advanced by conversation with david bishop and eleanor courtemanche and by suggestions from ryan cordell and elizabeth foxwell on bluesky references beekman g 2017 emotional density suspense and the serialization of the woman in white in all theyear round victorian periodicals review 50 1 chang k cramer m soni s bamman d 2023 speak memory an archaeology of books known to chatgpt gpt 4 https arxiv org abs 2305 00118 dames n 2023 the chapter a segmented history from antiquity to the twenty first century princeton university press garrett p 1980 the victorian multiplot novel studies in dialogical form yale university press haugtvedt e 2016 the sympathy of suspense gaskell and braddon s slow and fast sensation fiction in family magazines victorian periodicals review 49 1 mcgrath l higgins d hintze a 2018 measuring modernist novelty journal of cultural analytics https culturalanalytics org article 11030 measuring modernist novelty piper a xu h kolaczyk e d 2023 modeling narrative revelation computational humanities research 2023 https ceur ws org vol 3558 paper6166 pdf sap m horvitz e choi y smith n a pennebaker j 2020 recollection versus imagination exploring human memory and cognition via neural language models proceedings of the 58th annual meeting of the acl https aclanthology org 2020 acl main 178 sinykin dan big fiction how conglomeration changed the publishing industry and american literature columbia university press 2023 smuts a 2009 the paradox of suspense stanford encyclopedia of philosophy https plato stanford edu entries paradox suspense sussur tobin v 2018 the elements of surprise our mental limits and the satisfactions of plot harvard university press warhol robyn et al reading like a victorian https readinglikeavictorian osu edu tags ai artificial intelligence chatgpt digital humanities categories deep learning social effects of machine learning liberally educated students need to be more than consumers of ai post author by tedunderwood post date september 10 2023 5 comments on liberally educated students need to be more than consumers of ai the initial wave of controversy over large language models in education is dying down we haven t reached consensus about what to do yet retreat to in class exams make tougher writing assignments just forbid the use of ai but it s clear to everyone now that the models will require some response in a month or so there will be a backlash to this debate as professors discover that today s models are still not capable of writing a coherent footnoted twelve page research paper on their own we may tell ourselves that the threat to education was overhyped and congratulate ourselves on having addressed it that will be a mistake we haven t even begun to discuss the challenge ai poses for education for professors yes the initial disruption will be easy to address we can find strategies that allow us to continue evaluating student work while teaching the courses we re accustomed to teach problem solved but the challenge that really matters here is a challenge for students who will graduate into a world where white collar work is being redefined some things will get easier we may all have assistants to help us handle email but by the same token students will be asked to tackle bigger challenges our vision of those challenges is confined right now by a discourse that treats models as paper writing machines but that s hardly the limit of their capacity for instance models can read so a lawyer in 2033 may be asked to use a model to do a quick scan of new case law in these thirty jurisdictions and report back tomorrow on the implications for our project but then come to think of it a report is bulky and static so you know what don t write a report what i really need is a model that s prepared to provide an overview and then answer questions on this topic as they emerge in meetings over the next week a decade from now in short we will probably be using ai not just to gather material and analyze it but to communicate interactively with customers and colleagues all the forms of critical thinking we currently teach will still have value in that world it will still be necessary to ask questions about social context about hidden assumptions and about the uncertainty surrounding any estimate but our students won t be prepared to address those questions unless they also know enough about machine learning to reason about a model s assumptions and uncertainty at higher levels of responsibility this will require more than being a clever prompter and savvy consumer white collar professionals are likely to be fine tuning their own models they will need to choose a base model asses...
|
