Meta tags:
description=;
author=;

Headings (most frequently used words):

ben, newman, publications, large, scale, collaborations, course, projects, class, notes,

Text of the page (most frequently used words):
and (33), the (27), pdf (15), for (14), models (13), #newman (12), language (11), benjamin (10), from (8), tldr (8), can (7), evaluation (7), framework (7), with (6), code (6), scientific (6), stanford (5), nlp (5), final (4), project (4), decision (4), name (4), 2020 (4), conducted (4), during (4), emnlp (4), level (4), scholar (4), question (4), answering (4), decontextualizing (4), snippets (4), documents (4), space (3), 2018 (3), julia (3), gong (3), what (3), under (3), how (3), using (3), are (3), based (3), likely (3), metrics (3), election (3), research (3), that (3), 2021 (3), disinformation (3), human (3), cite (3), investigate (3), this (3), abilities (3), they (3), llms (3), suggestions (3), 2023 (3), semantic (3), propose (3), kyle (3), raymond (3), fok (3), linguist (2), splash (2), student (2), zero (2), width (2), learn (2), about (2), analysis (2), computational (2), without (2), pomdps (2), chinese (2), translating (2), different (2), large (2), two (2), machine (2), neural (2), representations (2), pretrained (2), syntax (2), but (2), training (2), course (2), projects (2), was (2), misinformation (2), bommasani (2), internship (2), information (2), not (2), method (2), evaluating (2), society (2), linguistics (2), communication (2), generation (2), sequence (2), extrapolate (2), many (2), end (2), has (2), john (2), hewitt (2), evaluate (2), syntactic (2), systems (2), blog (2), prompts (2), mask (2), canada (2), capital (2), methods (2), message (2), sentence (2), pyiship (2), performance (2), state (2), art (2), arman (2), cohan (2), luca (2), soldaini (2), user (2), facing (2), literature (2), tables (2), process (2), technologies (2), ben (2), washington (2), dot (2), designed, templatewire, 21n, linguistic, diversity, universals, slides, manner, speaking, class, notes, initiative, ssi, hiding, secret, messages, html, characters, demo, here, encrypter, semantics, pragmatics, 130a, people, use, swears, looking, their, word2vec, glove, embeddings, swear, making, uncertainty, levi, lian, suvir, mirchandani, cs238, you, teach, agents, follow, directions, defining, each, instruction, means, spatial, reference, game, artificial, intelligence, principles, techniques, cs221, your, article, because, translations, phoenetics, lack, corpera, explore, approaches, english, transliteration, search, networks, learning, 2019, cs229, incorporate, access, supervised, probes, unsupervised, recovery, tree, learned, integrity, partnership, coalition, groups, tracked, run, site, long, fuse, shelby, grossman, antoine, bousselut, misuse, section, opportunities, risks, foundation, led, design, implementation, scenarios, including, leading, holistic, scale, collaborations, summer, center, study, csli, cs224u, gram, overlap, like, bleu, capture, whether, successful, communicators, really, introduce, new, communicative, effectiveness, rational, speech, acts, computation, lsa, christopher, potts, reuben, cohn, gordon, natural, why, struggle, reasons, train, tokens, examples, one, them, visualize, effect, extrapolative, outstanding, paper, blackbox, chris, manning, percy, liang, eos, length, extrapolation, should, understanding, argue, build, off, minimal, pair, address, these, goals, find, better, conjugating, verbs, deem, systematicty, behavior, naacl, kai, siang, ang, refining, targeted, salesforce, querying, semantically, equivalent, which, city, necessarily, lead, same, predictions, low, parameter, fine, tuning, increase, model, consistency, yes, don, need, annotations, other, iclr, 2022, poster, prafulla, kumar, choubey, nazneen, rajani, adapters, robustly, extracting, factual, diverse, best, help, email, writing, assistants, legislative, staff, members, when, provide, compared, ones, chi, sarah, kreps, maurice, jakesch, liye, comparing, mediated, dataset, test, llm, ability, generate, comparison, few, table, algorithms, data, 2024, joseph, chee, chang, daniel, weld, juho, kim, pao, siangliulue, aakanksha, naik, yoonjoo, lee, arxivdigestables, synthesizing, into, publications, worked, analyzing, communicate, also, thought, usefully, augment, both, discovery, testing, psycholinguistic, hypotheses, political, science, countering, minsinformation, been, assistant, classes, have, taught, courses, introductory, computing, fundamentals, high, schoolers, previously, pre, doctoral, young, investigator, member, group, cs224n, cs124, hello, currently, phd, advised, interested, building, reliably, studying, roles, emerging, play, more, broadly, yejin, choi, university, allen, school, google, github, blnewman, edu,

Text of the page (random words):
benjamin newman ben newman blnewman at cs dot washington dot edu github google scholar blog hello i m ben newman currently a phd student at the allen school at the university of washington advised by yejin choi i m interested in building computational systems that reliably process human language and studying the roles emerging language technologies can play in society more broadly i ve worked on projects analyzing models abilities to extrapolate process syntax and communicate i ve also thought about how language technologies can usefully augment human language abilities both for scientific discovery e g testing psycholinguistic hypotheses and in political science e g countering minsinformation i ve been a course assistant for two of stanford s nlp classes cs124 and cs224n and have co taught courses in introductory linguistics and computing fundamentals to high schoolers at stanford splash i was previously a pre doctoral young investigator at semantic scholar research and a member of the stanford nlp group publications arxivdigestables synthesizing scientific literature into tables using language models benjamin newman yoonjoo lee aakanksha naik pao siangliulue raymond fok juho kim daniel s weld joseph chee chang kyle lo emnlp 2024 pdf code data tldr we propose a dataset and evaluation method to test llm s ability to generate literature comparison tables and we evaluate a few different table generation algorithms a question answering framework for decontextualizing user facing snippets from scientific documents benjamin newman luca soldaini raymond fok arman cohan kyle lo emnlp 2023 pdf tldr we propose a question answering framework for decontextualizing snippets from scientific documents and investigate the performance of state of the art llms under this framework conducted during a pyiship at semantic scholar a question answering framework for decontextualizing user facing snippets from scientific documents benjamin newman luca soldaini raymond fok arman cohan kyle lo emnlp 2023 pdf tldr we propose a question answering framework for decontextualizing snippets from scientific documents and investigate the performance of state of the art llms under this framework conducted during a pyiship at semantic scholar comparing sentence level suggestions to message level suggestions in ai mediated communication liye fu benjamin newman maurice jakesch sarah kreps chi 2023 pdf tldr llms can best help as email writing assistants for legislative staff members when they provide message level suggestions compared to sentence level ones p adapters robustly extracting factual information from language models with diverse prompts benjamin newman prafulla kumar choubey nazneen rajani iclr 2022 poster pdf tldr querying pretrained nlp models with semantically equivalent prompts e g mask is canada s capital and canada which has capital city mask do not necessarily lead to the same predictions can low parameter fine tuning methods increase model consistency yes and they don t need as many annotations as other methods conducted during an internship at salesforce research refining targeted syntactic evaluation benjamin newman kai siang ang julia gong john hewitt naacl 2021 pdf code cite blog tldr how should we evaluate the syntactic understanding of nlp systems we argue for evaluating models likely behavior and systematicty and build off of minimal pair evaluation to address these goals we find models are better at conjugating verbs they deem likely the eos decision and length extrapolation benjamin newman john hewitt percy liang and chris manning blackbox nlp emnlp 2020 outstanding paper pdf code cite tldr why do sequence models struggle to extrapolate for many reasons but the decision to train models with end of sequence tokens at the end of training examples is one of them we investigate and visualize the effect that this decision has on neural models extrapolative abilities communication based evaluation for natural language generation benjamin newman reuben cohn gordon and christopher potts society for computation in linguistics lsa 2020 pdf code cite tldr do n gram overlap metrics like bleu capture whether the models are successful communicators not really so we introduce a new method for evaluating communicative effectiveness based on the rational speech acts framework conducted during cs224u and the center for the study of language and information csli summer internship large scale collaborations holistic evaluation of language models bommasani et al 2021 disinformation evaluation led the design implementation and analysis of the disinformation scenarios and metrics including leading the human evaluation for disinformation pdf on the opportunities and risks of foundation models bommasani et al 2021 misuse section antoine bousselut shelby grossman and benjamin newman pdf the long fuse misinformation and the 2020 election the election integrity partnership was a coalition of research groups that tracked misinformation in the run up to the 2020 us election site pdf course projects unsupervised recovery of tree metrics from learned representations representations from pretrained language models likely incorporate syntax but can we access it without training supervised probes pdf cs229 machine learning final project 2019 english chinese name machine transliteration using search and neural networks what s your name in chinese translating a name is different from translating an article because name translations are based in phoenetics and lack large corpera we explore two approaches pdf code cs221 artificial intelligence principles and techniques final project with julia gong 2018 using pomdps to learn language in a spatial reference game how can you teach computational agents to follow directions without defining what each instruction means pomdps pdf code cs238 decision making under uncertainty final project with suvir mirchandani and levi lian 2018 swear analysis what we can learn about people s use of swears by looking at their word2vec and glove embeddings pdf linguist 130a semantics and pragmatics final project with julia gong 2018 zero width space encrypter hiding secret messages in html zero width space characters demo here stanford student space initiative ssi class notes stanford splash slides in a manner of speaking linguist 21n linguistic diversity and universals designed by templatewire