Meta tags:
author=Faeze Brahman;
description=A simple, whitespace theme for academics. Based on [*folio](htt????/github.com/bogoli/-folio) design. ;
keywords=jekyll, jekyll-theme, academic-website, portfolio-website;

Headings (most frequently used words):

faeze, brahman, news, selected, publications,

Text of the page (most frequently used words):
and (140), the (45), models (34), for (27), with (26), language (25), that (25), 2024 (25), #brahman (21), 2025 (19), are (15), faeze (12), abs (12), more (12), prompting (11), this (11), 2023 (11), can (11), emnlp (10), https (10), choi (10), generation (10), arxiv (10), humans (10), may (9), url (9), year (9), booktitle (9), yejin (9), author (9), title (9), inproceedings (9), problem (9), bib (9), learning (9), our (9), capabilities (9), generative (9), human (9), org (8), maieutic (8), code (8), authors (8), iclr (8), ipa (8), gpt (8), knowledge (8), understanding (8), reasoning (7), inference (7), even (7), their (7), show (7), scale (7), not (7), counterfactual (7), llms (7), solving (7), 2022 (6), proceedings (6), explanations (6), work (6), over (6), international (6), conference (6), representations (6), west (6), such (6), fine (6), time (6), tailoring (6), large (6), planning (6), naacl (6), appear (6), talk (6), accepted (6), jung (5), qin (5), self (5), these (5), correct (5), while (5), outputs (5), extreme (5), present (5), model (5), tasks (5), constrained (5), ravichander (5), tuning (5), procedural (5), both (5), primaryclass (5), eprint (5), intelligence (5), macgyver (5), neurips (5), colm (5), university (5), oct (5), slides (5), working (5), main (4), welleck (4), bhagavatula (4), bras (4), trained (4), lms (4), has (4), which (4), often (4), challenging (4), better (4), ximing (4), sequence (4), using (4), base (4), propose (4), chandu (4), abhilasha (4), jiang (4), policy (4), without (4), but (4), small (4), pyatkin (4), paper (4), 2311 (4), paradox (4), expert (4), been (4), nomination (4), creative (4), problems (4), 2407 (4), top (4), llm (4), haicosystem (4), safety (4), nlp (4), invited (4), papers (4), new (4), evaluation (4), ai2 (4), research (4), from (3), jaehun (3), lianhui (3), ronan (3), consistent (3), generations (3), test (3), require (3), art (3), approach (3), peter (3), making (3), task (3), imperfect (3), generator (3), off (3), corrector (3), training (3), upon (3), smaller (3), khyathi (3), liwei (3), adapters (3), have (3), performance (3), degree (3), through (3), results (3), plasma (3), hwang (3), into (3), about (3), they (3), novel (3), setting (3), dziri (3), create (3), understand (3), would (3), hypothesis (3), ability (3), best (3), tian (3), dataset (3), coconot (3), data (3), track (3), archiveprefix (3), contextual (3), noncompliance (3), agreement (3), publications (3), washington (3), workshop (3), senior (3), apr (3), wildteaming (3), sep (3), post (3), allen (3), institute (3), 2026 (2), pages (2), aclanthology (2), association (2), computational (2), linguistics (2), publisher (2), address (2), dec (2), sean (2), chandra (2), logically (2), recursive (2), struggle (2), generate (2), promising (2), approaches (2), develop (2), true (2), then (2), three (2), complex (2), than (2), methods (2), fully (2), unsupervised (2), supervised (2), also (2), improves (2), eleventh (2), shen (2), generating (2), sequences (2), constraints (2), few (2), iteratively (2), inefficient (2), infeasible (2), parameters (2), specific (2), correction (2), shelf (2), use (2), either (2), lexically (2), toxicity (2), control (2), jang (2), variety (2), directly (2), them (2), costly (2), broader (2), lightweight (2), user (2), five (2), text (2), consistently (2), brings (2), sometimes (2), outperform (2), potential (2), preprint (2), valentina (2), jena (2), xiang (2), level (2), machines (2), situations (2), current (2), requires (2), 00059 (2), nouha (2), what (2), recent (2), superhuman (2), artificial (2), now (2), exceed (2), same (2), basic (2), errors (2), non (2), how (2), contingent (2), across (2), capability (2), 09682 (2), yufei (2), marjieh (2), solvers (2), functional (2), fixedness (2), familiar (2), ways (2), real (2), world (2), thinking (2), out (2), abilities (2), being (2), step (2), neural (2), information (2), processing (2), systems (2), benchmark (2), 12043 (2), kumar (2), balachandran (2), dasigi (2), saying (2), oral (2), presentation (2), 18370 (2), trust (2), escalate (2), judges (2), provable (2), guarantees (2), 2409 (2), 16427 (2), zhou (2), kim (2), zhu (2), ecosystem (2), sandboxing (2), risks (2), interactions (2), selected (2), feb (2), umass (2), led (2), intern (2), area (2), chair (2), creativity (2), was (2), attending (2), jul (2), serving (2), released (2), family (2), open (2), nov (2), agents (2), jan (2), seminar (2), series (2), you (2), just (2), olmo (2), developing (2), strategies (2), limitations (2), computer (2), science (2), master (2), scientist (2), copyright, powered, theme, hosted, photos, last, updated, unsplash, github, folio, jekyll, please, conatct, fae, gmail, com, 1266, 1279, 18653, doi, abu, dhabi, united, arab, emirates, month, etal, pre, recently, guide, emerged, direction, amend, however, fundamentally, bounded, correctness, themselves, noisy, inconsistent, aims, infer, answer, question, unreliable, induces, tree, abductively, because, recursively, frames, satisfiability, logical, relations, false, benchmarks, commonsense, achieves, accuracy, state, performs, competitively, robustness, providing, interpretable, rationales, openreview, net, forum, hh36jeqzdao, tianxiao, khashabi, daniel, welleck2023generating, applications, satisfying, semantic, ensuring, programs, certain, keywords, avoiding, undesirable, content, whether, tuned, prompted, shot, demonstrations, frequently, violate, lack, mechanism, revise, moreover, some, powerful, inaccessible, update, adaptation, decouples, separate, learns, train, online, procedure, scalar, natural, feedback, intermediate, diverse, mathematical, program, synthesis, when, much, 424, ammanabrolu, prithviraj, ramnath, sahana, others, lu2023inference, demonstrated, exceptional, pure, limited, effective, extremely, feasible, community, efficiently, tailors, guides, during, decoding, adapter, optimize, arbitrary, objective, reinforcement, reduction, significant, improvements, outperforms, competitive, baseline, including, expensive, particular, major, boost, highlight, alternative, 2305, 19472, journal, lorraine, arai, hirona, sanyal, soumya, sakaguchi, keisuke, ren, brahman2023plasma, entails, decomposing, high, goal, temporally, ordered, steps, important, yet, intricate, involves, integrating, common, sense, reason, contextualized, scheduling, doctor, appointment, phone, encouraging, hindered, drawbacks, api, calls, reproducibility, issues, advocate, two, pronged, endow, concretely, symbolic, distillation, enhance, implicit, algorithm, facilitate, structured, accurate, addition, introduce, revision, plan, cope, situation, original, orders, magnitude, 770m, 11b, compete, surpass, larger, teacher, linjie, fisher, jillian, newman, benjamin, koh, pang, wei, ettinger, allyson, west2023generative, wave, sparked, unprecedented, global, attention, excitement, concern, potentially, levels, take, only, seconds, produce, challenge, still, expected, presents, apparent, reconcile, seemingly, persistence, make, posit, tension, reflects, divergence, configuration, today, relative, specifically, having, reproduce, like, acquire, therefore, those, types, contrasts, whom, almost, always, precedes, controlled, experiments, analyzing, image, modalities, although, fall, short, measures, well, weaker, correlation, between, brittleness, adversarial, inputs, findings, support, call, caution, interpreting, analogy, raja, peng, nanyun, griffiths, thomas, tian2023macgyver, explore, modern, circumventing, cognitive, bias, known, psychology, objects, innovative, unconventional, end, automatically, generated, consisting, 600, deliberately, trigger, box, collection, compare, contrast, groups, unique, complementary, example, typically, excel, requiring, domain, leading, higher, variance, other, hand, exposed, highly, specialized, attempt, prone, overconfidence, actions, physically, provide, detailed, error, analysis, demonstrate, enhancing, techniques, iterative, wise, reflection, divergent, convergent, provides, insight, illustrates, psychological, paradigms, extended, comparing, huggingface, datasets, allenai, sachin, vidhisha, pradeep, wiegreffe, sarah, hessel, jack, tsvetkov, yulia, smith, noah, hajishirzi, hannaneh, brahman2024artsayingnocontextual, blog, jung2024trustescalatellmjudges, xuhui, hyunwoo, hao, frank, lin, bill, yuchen, mireshghallah, niloofar, sap, maarten, zhou2024haicosystemecosystemsandboxingsafety, website, guest, lectures, cse, 599, seminars, mar, hanjie, chen, acl, rev, narrative, honored, serve, ubc, group, bocconi, will, vienna, check, here, wildbench, supper, excited, cascaded, selective, aug, coling, rising, star, attend, genai, amherst, preference, annotation, hybridpref, multi, turn, every, pipeline, tülu, vancouver, trusthworthy, come, say, liedar, studying, utility, truthfulness, trade, works, benchmarking, queries, based, guarantee, ucla, keynote, singapore, symposium, ssnlp, raise, jun, accepeted, see, montreal, mitigating, hallucination, 32b, rival, qwen3, checkout, play, playground, technical, report, news, feel, free, get, touch, want, chat, collaborate, brainstorm, previously, interned, microsoft, controllable, grounded, rationale, monotonic, featured, press, techcrunch, greekwire, robust, meaningful, investigate, measure, emergent, bahaviors, frameworks, build, system, various, contexts, reliable, safe, centered, investigating, behaviors, function, alignment, wild, under, beyond, scaling, includes, design, collaborative, hci, interfaces, combine, efficient, algorithms, unseen, changing, team, prior, doctoral, researcher, did, california, santa, cruz, hold, 2018, 2014, bachelor, 2012, electrical, engineering, broad, set, interests, snigdha, chaturvedi, seattle, submenus, mentees, toggle, navigation,

Text of the page (random words):
ts developing robust and meaningful evaluation frameworks to investigate the emergent bahaviors of llms that are challenging to measure previously i interned at microsoft research working on controllable grounded text generation and at ai2 working on unsupervised rationale generation for non monotonic reasoning my work has been featured in press by greekwire and techcrunch feel free to get in touch if you want to chat collaborate or brainstorm news nov 2025 we just released olmo 3 family of 7b 32b models rival qwen3 while being fully open checkout the technical report and play with it on ai2 playground oct 2025 new preprint on rl for mitigating hallucination in reasoning models sep 2025 five papers accepeted to colm 2025 see you in montreal jun 2025 i am serving as an ac for colm 2025 may 2025 invited talk at university of washington raise seminar series slides apr 2025 keynote talk at 2025 singapore symposium on nlp ssnlp slides feb 2025 invited talk at ucla nlp seminar series slides jan 2025 our works on lm based evaluation with human agreement guarantee top 1 8 and benchmarking llms using real world user queries top 5 have been accepted to appear at iclr 2025 jan 2025 our work on studying utility truthfulness trade off in llm agents has been accepted to appear at naacl 2025 dec 2024 i am attending neurips 24 in vancouver to present wildteaming main coconot d b track haicosystem and ai liedar safety trusthworthy agents workshop come say nov 2024 we released tülu 3 a family of post trained models with every step in the pipeline open oct 2024 new papers on multi turn safety evaluation haicosystem and preference annotation hybridpref oct 2024 selected as a rising star to attend the genai workshop at umass amherst sep 2024 serving as area chair for coling 2025 sep 2024 wildteaming and coconot are now accepted at neurips 2024 aug 2024 supper excited about our recent work on cascaded selective evaluation jul 2024 new paper on contextual noncompliance jul 2024 new papers on wildteaming at scale and wildbench may 2024 i will be attending iclr 24 in vienna to present 3 papers check them out here may 2024 invited talk at bocconi university on creativity and constrained problem solving slides apr 2024 macgyver led by my intern yufei tian was accepted at naacl 2024 apr 2024 invited talk at ubc nlp group on creativity and constrained problem solving slides oct 2023 i m honored to serve as a senior area chair for naacl 2024 oct 2023 our workshop on narrative understanding is accepted to emnlp 2024 may 2023 rev led by our intern hanjie chen is accepted at acl 2023 mar 2023 talk at umass nlp seminars feb 2023 guest lectures at the university of washington cse 599 selected publications colm haicosystem an ecosystem for sandboxing safety risks in human ai interactions x zhou f brahman h kim l jiang h zhu and 7 more authors to appear at colm 2025 bib code website inproceedings zhou2024haicosystemecosystemsandboxingsafety title haicosystem an ecosystem for sandboxing safety risks in human ai interactions author zhou xuhui and brahman faeze and kim hyunwoo and jiang liwei and zhu hao and lu ximing and xu frank and lin bill yuchen and choi yejin and mireshghallah niloofar and bras ronan le and sap maarten year 2025 eprint 2409 16427 archiveprefix arxiv primaryclass cs ai booktitle to appear at colm url https arxiv org abs 2409 16427 iclr trust or escalate llm judges with provable guarantees for human agreement j jung f brahman and y choi to appear at iclr 2025 bib code oral presentation top 1 8 inproceedings jung2024trustescalatellmjudges title trust or escalate llm judges with provable guarantees for human agreement author jung jaehun and brahman faeze and choi yejin year 2025 eprint 2407 18370 archiveprefix arxiv primaryclass cs lg url https arxiv org abs 2407 18370 booktitle to appear at iclr nomination oral presentation top 1 8 neurips the art of saying no contextual noncompliance in language models f brahman s kumar v balachandran p dasigi v pyatkin and 9 more authors neural information processing systems neurips dataset and benchmark track 2024 bib blog code data inproceedings brahman2024artsayingnocontextual title the art of saying no contextual noncompliance in language models author brahman faeze and kumar sachin and balachandran vidhisha and dasigi pradeep and pyatkin valentina and ravichander abhilasha and wiegreffe sarah and dziri nouha and chandu khyathi and hessel jack and tsvetkov yulia and smith noah a and choi yejin and hajishirzi hannaneh year 2024 eprint 2407 12043 archiveprefix arxiv primaryclass cs cl url https arxiv org abs 2407 12043 booktitle neural information processing systems neurips dataset and benchmark track data https huggingface co datasets allenai coconot naacl macgyver are large language models creative problem solvers y tian a ravichander l qin r bras r marjieh and 4 more authors proceedings of naacl 2024 abs bib code best paper nomination we explore the creative problem solving capabilities of modern large language models llms in a constrained setting the setting requires circumventing a cognitive bias known in psychology as functional fixedness to use familiar objects in innovative or unconventional ways to this end we create macgyver an automatically generated dataset consisting of 1 600 real world problems that deliberately trigger functional fixedness and require thinking out of the box we then present our collection of problems to both llms and humans to compare and contrast their problem solving abilities we show that macgyver is challenging for both groups but in unique and complementary ways for example humans typically excel in solving problems that they are familiar with but may struggle with tasks requiring domain specific knowledge leading to a higher variance on the other hand llms being exposed to a variety of highly specialized knowledge attempt broader problems but are prone to overconfidence and propose actions that are physically infeasible or inefficient we also provide a detailed error analysis of llms and demonstrate the potential of enhancing their problem solving ability with novel prompting techniques such as iterative step wise reflection and divergent convergent thinking this work provides insight into the creative problem solving capabilities of humans and ai and illustrates how psychological paradigms can be extended into large scale tasks for comparing humans and machines inproceedings tian2023macgyver title macgyver are large language models creative problem solvers author tian yufei and ravichander abhilasha and qin lianhui and bras ronan le and marjieh raja and peng nanyun and choi yejin and griffiths thomas l and brahman faeze year 2024 booktitle proceedings of naacl eprint 2311 09682 url https arxiv org abs 2311 09682 primaryclass cs cl nomination best paper nomination iclr the generative ai paradox what it can create it may not understand f brahman p west x lu n dziri l li and 9 more authors international conference on learning representations 2024 abs bib code the recent wave of generative ai has sparked unprecedented global attention with both excitement and concern over potentially superhuman levels of artificial intelligence models now take only seconds to produce outputs that would challenge or exceed the capabilities even of expert humans at the same time models still show basic errors in understanding that would not be expected even in non expert humans this presents us with an apparent paradox how do we reconcile seemingly superhuman capabilities with the persistence of errors that few humans would make in this work we posit that this tension reflects a divergence in the configuration of intelligence in today s generative models relative to intelligence in humans specifically we propose and test the generative ai paradox hypothesis generative models having been trained directly to reproduce expert like outputs acquire generative capabilities that are not contingent upon and can therefore exceed their ability to understand those same types of outputs this contrasts with humans for whom basic understanding almost always precedes the ability to generate expert level outputs we test this hypothesis through controlled experiments analyzing generation vs understanding in generative models across both language and image modalities our results show that although models can outperform humans in generation they consistently fall short of human capabilities in measures of understanding as well as weaker correlation between generation and understanding performance and more brittleness to adversarial inputs our findings support the hypothesis that models generative capability may not be contingent upon understanding capability and call for caution in interpreting artificial intelligence by analogy to human intelligence inproceedings west2023generative title the generative ai paradox what it can create it may not understand author brahman faeze and west peter and lu ximing and dziri nouha and li linjie and hwang jena d and jiang liwei and fisher jillian and ravichander abhilasha and chandu khyathi and newman benjamin and koh pang wei and ettinger allyson and choi yejin year 2024 eprint 2311 00059 url https arxiv org abs 2311 00059 booktitle international conference on learning representations primaryclass cs ai iclr plasma making small language models better procedural knowledge models for counterfactual planning f brahman c bhagavatula v pyatkin j hwang x li and 5 more authors international conference on learning representations 2024 abs bib code procedural planning which entails decomposing a high level goal into a sequence of temporally ordered steps is an important yet intricate task for machines it involves integrating common sense knowledge to reason about complex contextualized situations that are often counterfactual e g scheduling a doctor s appointment without a phone while current approaches show encouraging results using large language models llms they are hindered by drawbacks such as costly api calls and reproducibility issues in this paper we advocate planning using smaller language models we present plasma a novel two pronged approach to endow small language models with procedural knowledge and counterfactual planning capabilities more concretely we develop symbolic procedural knowledge distillation to enhance the implicit knowledge in small language models and an inference time algorithm to facilitate more structured and accurate reasoning in addition we introduce a novel task counterfactual planning that requires a revision of a plan to cope with a counterfactual situation in both the original and counterfactual setting we show that orders of magnitude smaller models 770m 11b parameters can compete and often surpass their larger teacher models capabilities inproceedings brahman2023plasma author brahman faeze and bhagavatula chandra and pyatkin valentina and hwang jena d and li xiang lorraine and arai hirona j and sanyal soumya and sakaguchi keisuke and ren xiang and choi yejin journal arxiv preprint title plasma making small language models better procedural knowledge models for counterfactual planning url https arxiv org abs 2305 19472 booktitle international conference on learning representations year 2024 emnlp inference time policy adapters ipa tailoring extreme scale lms without fine tuning x lu f brahman p west j jang k chandu and 6 more authors proceedings of emnlp 2023 abs bib code while extreme scale language models have demonstrated exceptional performance on a variety of language tasks the degree of control over these language models through pure prompting can often be limited directly fine tuning such language models can be effective for tailoring them but it can be either extremely costly e g gpt 3 or not even feasible for the broader community e g gpt 4 we propose inference time policy adapters ipa which efficiently tailors a language model such as gpt 3 without fine tuning it ipa guides a large base model during decoding time through a lightweight policy adapter trained to optimize an arbitrary user objective with reinforcement learning on five challenging text generation tasks such as toxicity reduction and lexically constrained generation ipa consistently brings significant improvements over off the shelf language models it outperforms competitive baseline methods sometimes even including expensive fine tuning in particular tailoring gpt 2 with ipa can outperform gpt 3 while tailoring gpt 3 with ipa brings a major performance boost over gpt 3 and sometimes even over gpt 4 our promising results highlight the potential of ipa as a lightweight alternative to tailoring extreme scale language models inproceedings lu2023inference title inference time policy adapters ipa tailoring extreme scale lms without fine tuning author lu ximing and brahman faeze and west peter and jang jaehun and chandu khyathi and ravichander abhilasha and qin lianhui and ammanabrolu prithviraj and jiang liwei and ramnath sahana and others booktitle proceedings of emnlp year 2023 publisher association for computational linguistics url https aclanthology org 2023 emnlp main 424 iclr generating sequences by learning to self correct s welleck x lu f brahman p west t shen and 2 more authors the eleventh international conference on learning representations 2023 abs bib sequence generation applications require satisfying semantic constraints such as ensuring that programs are correct using certain keywords or avoiding undesirable content language models whether fine tuned or prompted with few shot demonstrations frequently violate these constraints and lack a mechanism to iteratively revise their outputs moreover some powerful language models are of extreme scale or inaccessible making it inefficient if not infeasible to update their parameters for task specific adaptation we present self correction an approach that decouples an imperfect base generator an off the shelf language model or supervised sequence to sequence model from a separate corrector that learns to iteratively correct imperfect generations to train the corrector we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations we show that self correction improves upon the base generator in three diverse generation tasks mathematical program synthesis lexically constrained generation and toxicity control even when the corrector is much smaller than the base generator inproceedings welleck2023generating title generating sequences by learning to self correct author welleck sean and lu ximing and brahman faeze and west peter and shen tianxiao and khashabi daniel and choi yejin booktitle the eleventh international conference on learning representations year 2023 url https openreview net forum id hh36jeqzdao emnlp maieutic prompting logically consistent reasoning with recursive explanations j jung l qin s welleck f brahman c bhagavatula and ...