Meta tags:
description=A community blog devoted to technical AI alignment research;

Headings (most frequently used words):

ai, alignment, posts, popular, comments, recent, discussion, the, safe, to, dangerous, shift, is, fundamental, problem, for, eval, realism, forum, af,

Text of the page (most frequently used words):
the (105), and (60), that (41), are (25), for (23), this (22), more (19), have (19), some (17), can (16), with (15), you (13), about (13), rational (11), deployment (10), but (10), model (9), than (9), cooperate (9), they (9), decision (9), just (9), would (8), defect (8), then (8), now (8), like (8), out (8), alignment (8), fdt (7), from (7), one (7), also (7), think (7), problem (7), their (7), people (7), gpt (7), neel (6), nanda (6), awareness (6), data (6), otherwise (6), here (6), what (6), things (6), better (6), even (6), which (6), there (6), eval (6), all (6), josh (5), engels (5), theory (5), reply (5), proof (5), say (5), though (5), was (5), not (5), your (5), use (5), policy (5), could (5), scheming (5), released (5), already (5), them (5), new (5), being (5), why (4), evaluation (4), read (4), long (4), opponent (4), cooperation (4), another (4), prudentbot (4), been (4), should (4), two (4), dilemma (4), where (4), get (4), itself (4), other (4), won (4), enough (4), words (4), code (4), into (4), try (4), version (4), its (4), academic (4), learning (4), models (4), see (4), training (4), most (4), strategies (4), agi (4), help (4), posts (4), fake (4), dangerous (4), basically (4), egregious (4), misalignment (4), llm (3), before (3), how (3), matters (3), case (3), vojtakovarik (3), stuart_armstrong (3), pragmatic (3), algorithm (3), way (3), length (3), let (3), bigger (3), our (3), payor (3), agree (3), first (3), very (3), directly (3), find (3), provability (3), different (3), clear (3), branch (3), continue (3), reading (3), löb (3), has (3), behavior (3), prisoner (3), situation (3), doesn (3), viewed (3), want (3), post (3), good (3), don (3), continual (3), over (3), months (3), many (3), memories (3), differences (3), between (3), research (3), will (3), memory (3), might (3), follow (3), safe (3), shift (3), evaluations (3), reliably (3), few (3), harness (3), trained (3), assistant (3), chatgpt (3), tasks (3), researchers (3), because (3), load (2), naive (2), safety (2), fail (2), arthur (2), conmy (2), rohin (2), shah (2), control (2), senthooran (2), rajamanoharan (2), bilalchughtai (2), 11d (2), predictors (2), game (2), works (2), lot (2), expect (2), rosehadshar (2), shortform (2), geoffrey (2), irving (2), limit (2), shortest (2), james (2), quite (2), own (2), part (2), kripke (2), under (2), logic (2), feels (2), much (2), setup (2), trying (2), still (2), answer (2), relevant (2), when (2), theorem (2), right (2), beginning (2), fairbot (2), call (2), miri (2), axelrod (2), competitors (2), were (2), programs (2), description (2), turns (2), tournament (2), position (2), often (2), depends (2), apply (2), word (2), asking (2), whether (2), ransom (2), probably (2), future (2), makes (2), irrational (2), decisions (2), certain (2), situations (2), describing (2), across (2), questions (2), isn (2), published (2), may (2), fashion (2), theorists (2), who (2), hand (2), someone (2), deployed (2), task (2), structural (2), difference (2), increased (2), misaligned (2), happens (2), concerned (2), risk (2), coherent (2), via (2), written (2), term (2), additional (2), only (2), taxonomy (2), deployments (2), discussion (2), signals (2), make (2), progress (2), both (2), distinguish (2), making (2), human (2), using (2), great (2), charlie (2), griffin (2), however (2), past (2), realistic (2), distribution (2), difficult (2), deploy (2), once (2), fundamental (2), realism (2), scenario (2), come (2), stage (2), 2024 (2), lab (2), said (2), agent (2), autogpt (2), running (2), write (2), fact (2), debug (2), 2025 (2), games (2), getting (2), continually (2), sometimes (2), course (2), seem (2), year (2), comes (2), system (2), etc (2), labs (2), next (2), access (2), doing (2), 2026 (2), worried (2), forum (2), javascript (2), 23d, sft, filters, properties, 22d, callummcdougall, synthetic, document, finetuning, instilling, positive, traits, 21d, hannah, sheahan, cameron, raymond, micahcarroll, marcus, williams, tomek, korbak, predicting, release, simulating, 19d, seb, farquhar, erik, jenner, mary, phuong, gdm, roadmap, 17d, jános, kramár, callum, mcdougall, transparent, diffusiongemma, 15d, driven, feature, discovery, gersonkroiz, aditya, singh, forensics, mateusz, bagiński, tomáš, gavenčiak, 16h, rosser, dohun, lee, filtering, worse, elegant, name, collision, resolution, epistemic, meristem, derived, searches, increasingly, proofs, obtain, budget, looking, harder, symbols, something, grows, bit, encounter, man, okay, wondering, following, implementation, calling, writing, perspective, continuing, puzzling, wholesome, instantiation, exists, finite, modality, project, ground, absent, feel, confident, intuition, stands, processes, above, outlining, terms, realizable, rambly, thoughts, offer, included, below, interest, fwiw, place, arrived, thinking, steps, modelling, prisoners, bothered, notice, checking, shorcut, quantity, counterfact, kinda, surprised, encodes, modal, reasonable, behaviour, poor, imitation, counterfactual, thought, obviously, able, real, encoding, little, ask, does, defecting, lead, cooperating, leads, yep, runs, afoul, consistency, inner, counterfactuals, nothing, save, hard, agents, clever, reason, activate, carefully, skirting, around, nice, waiting, expressive, capture, clean, translation, eluding, ontol, 2262, instead, calls, lemma, andrew, critch, suggested, proving, self, fun, result, paper, program, whose, specified, provably, despite, appearance, circularity, cooperates, involves, löbian, context, twist, facing, each, iterated, rather, gödel, determine, shot, 4mo, transhumanist_atom_understander, payorian, easy, frames, tend, defend, preferred, discussions, degenerate, fruitless, semantic, debates, fixed, particular, contradiction, never, pays, blackmailed, pay, assuming, cause, blackmail, attempts, increase, cdt, current, overall, branches, theories, perfectly, compatible, sentence, sounds, contradictory, arguably, newcomb, paradoxical, cubefox, 3232, sketch, designed, sidestep, barely, keep, small, finger, count, too, problems, definitions, definedness, build, defined, variants, achieve, attempted, theorist, back, defining, blog, bentham, bulldog, against, contrasting, rationalist, enthusiasm, scepticism, number, adopt, counted, missing, four, fingers, functional, sort, specialize, timeline, weeks, distributions, thi, round, structure, happening, times, parallel, leading, variance, chance, selection, rounds, evaluated, according, rewards, 1826, spell, threat, including, novel, arguments, plausibility, describe, promising, mitigating, while, plausible, mitigations, reasonably, cheap, effective, reducing, arising, mechanism, likely, substantially, productive, various, ais, pose, risks, notably, seth, herd, aligned, occasionally, start, become, consistent, updates, behavioral, schemer, changes, alex, mallen, countermeasures, memetic, spread, values, agreed, frame, important, underlying, evals, these, translate, reliable, signal, attempt, gesture, excited, strong, give, cents, thank, flag, yeah, test, cost, sorry, slow, response, luckily, waited, posted, definitely, points, argue, easier, kicking, down, rode, somewhat, crucially, captures, crucial, challenging, suggesting, bounding, level, red, teaming, plenty, ruled, perhaps, seen, recently, diving, broader, range, 13h, 751, core, toolemu, spoof, tool, responses, sad, openai, osworld, webarena, petri, create, environments, appear, approaches, faking, rule, possibility, common, approach, black, box, reassuring, extent, evidence, anything, catastrophically, suppose, capable, potentially, 2mo, patrick, leask, measuring, recent, blast, years, stages, mini, indeed, pass, albeit, later, roughly, multimodal, take, screenshot, input, output, tokens, keystrokes, mouseclicks, images, emergent, capabilities, everything, qualitatively, super, couldn, fine, tuned, strapped, useful, tons, general, purpose, lots, raving, moment, used, bunch, stuff, unlike, leave, background, working, away, docs, edit, check, same, again, sophisticated, invented, play, board, video, decently, serious, kinks, generally, worked, newer, oodles, huge, base, customers, sorts, tries, fails, succeeds, succeed, gradually, set, expands, top, sorta, continuous, plateauing, keeps, skills, accumulates, millions, treating, coworker, virtual, giving, passwords, such, letting, handle, life, admin, shopping, big, swear, rumor, public, claims, typical, engineers, approximately, doubled, productivity, mostly, oversee, manage, lightning, fast, labor, debugging, online, refuses, leaks, indicate, refuse, internally, heavily, automating, process, parent, corporation, work, humans, watching, metrics, suggestions, understand, experiments, architectures, proposing, superintelligent, whatever, definition, favorite, eyes, caveat, true, yet, updating, month, anthropic, claim, solved, rest, seems, timelines, 1mo, daniel, kokotajlo, through, race, dynamics, careless, programmers, bad, actors, opposed, simpler, yudkowsky, soares, story, nobody, foggiest, idea, avoid, citation, needed, pretty, emerging, naturally, least, moderately, prevent, sympathy, sides, debate, 24d, ryan_greenblatt, popular, comments, library, home, login, website, requires, properly, function, consider, activating, site, functionality,

Text of the page (random words):
ment forum af login home library questions all posts about ai alignment posts popular comments ryan_greenblatt 24d 13 8 sympathy for both sides of the egregious misalignment debate and if they re concerned about egregious misalignment and scheming they ll probably say that it would come about through race dynamics careless programmers bad actors etc as opposed to the simpler yudkowsky soares story of we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that citation needed i think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent daniel kokotajlo 1mo 13 1 ai timelines blast from the past now that it s been a few years some of the stages in my mini scenario have indeed come to pass albeit a few months later than in the scenario i d say we are roughly at stage 4 now 1 q1 2024 a bigger better model than gpt 4 is released by some lab it s multimodal it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images just like with gpt 4 vs gpt 3 5 vs gpt 3 it turns out to have new emergent capabilities everything gpt 4 can do it can do better but there are also some qualitatively new things that it can do though not super reliably that gpt 4 couldn t do 2 q3 2024 said model is fine tuned to be an agent it was already better at being strapped into an autogpt harness than gpt 4 was so it was already useful for some things but now it s being trained on tons of data to be a general purpose assistant agent lots of people are raving about it it s like another chatgpt moment people are using it for all the things they used chatgpt for but then also a bunch more stuff unlike chatgpt you can just leave it running in the background working away at some problem or task for you it can write docs and edit them and fact check them it can write code and then debug it 3 q1 2025 same as 1 all over again an even bigger model even better also it s not just autogpt harness now it s some more sophisticated harness that someone invented also it s good enough to play board games and some video games decently on the first try 4 q3 2025 ok now things are getting serious the kinks have generally been worked out this newer model is being continually trained on oodles of data from a huge base of customers they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often gradually the set of tasks it can do reliably expands over the course of a few months it doesn t seem to top out progress is sorta continuous now even as the new year comes there s no plateauing the system just keeps learning new skills as the training data accumulates now many millions of people are basically treating it like a coworker and virtual assistant people are giving it their passwords and such and letting it handle life admin tasks for them help with shopping etc and of course quite a lot of code is being written by it researchers at big agi labs swear by it and rumor is that the next version of the system which is already beginning training won t be released to the public because the lab won t want their competitors to have access to it already there are claims that typical researchers and engineers at agi labs are approximately doubled in productivity because they mostly have to just oversee and manage and debug the lightning fast labor of their ai assistant and it s continually getting better at doing said debugging itself 5 q1 2026 the next version comes online it is released but it refuses to help with ml research leaks indicate that it doesn t refuse to help with ml research internally and in fact is heavily automating the process at its parent corporation it s basically doing all the work by itself the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it s running and architectures it s proposing 6 q3 2026 superintelligent agi happens by whatever definition is your favorite and you see it with your own eyes the caveat is that we don t seem to have true continual learning yet on the other hand they are updating the models about once a month by training them on additional data and anthropic researchers claim that continual learning will be solved this year and the rest of the stage 4 description seems basically right recent discussion the safe to dangerous shift is a fundamental problem for eval realism but also for measuring awareness 21 charlie griffin patrick leask 2mo 1 the safe to dangerous shift is a fundamental problem for eval realism suppose we have a capable and potentially scheming model and before we deploy it we want some evidence that it won t do anything catastrophically dangerous once we deploy it a common approach is to use black box alignment evaluations however alignment evaluations are only reassuring to the extent that the model can t reliably 1 distinguish the deployment distribution from the evaluation distribution as it is otherwise difficult to rule out the possibility of alignment faking there are many approaches one could use to try to make evaluations appear more realistic you can try to create realistic environments e g petri webarena osworld use data from past deployments e g openai sad and spoof tool call responses e g toolemu however the core difference between an alignment evaluation and a see more 751 more words 2 charlie griffin 13h i m sorry for the slow response luckily i waited long enough that we posted the follow up i definitely agree with both of your points that you could argue that your fake eval and fake deployment are easier to distinguish and that this is kicking the can down the rode somewhat crucially i think making a fake eval and fake deployment that captures some of the crucial strategies an ai could follow is challenging we just released a follow up suggesting bounding eval awareness of human level ai across the safe to dangerous shift using human red teaming there are plenty of other strategies signals that are not ruled out i agree this would be great perhaps you have already seen them but i think vojtakovarik et al have released some great posts recently diving into a broader range of strategies vojtakovarik 2h 1 0 thank you for the flag yeah i do think the posts deployment awareness matters more than evaluation awareness and if this were a test how much would it cost are relevant here 1 to give me two cents on this i d be very excited to see a strong taxonomy of eval deployment signals it might help make progress here agreed though i would frame this more as taxonomy of the most important underlying differences between evals and deployments discussion of when these differences translate into a reliable signal an attempt to gesture at some read more reply the case for countermeasures to memetic spread of misaligned values 46 alex mallen 1y as various people have written about before ais that have long term memory might pose additional risks most notably llm agi will have memory and memory changes alignment by seth herd even if an ai is aligned or only occasionally scheming at the start of a deployment the ai might become a consistent and coherent behavioral schemer via updates to its long term memories in this post i ll spell out the version of the threat model that i m most concerned about including some novel arguments for its plausibility and describe some promising strategies for mitigating this risk while i think some plausible mitigations are reasonably cheap and could be effective at reducing the risk from coherent scheming arising via this mechanism research here will likely be substantially more productive in the future continue reading 1826 more words josh engels 4h 1 0 if the selection of memories happens in clear rounds where memories are evaluated according to rewards then i don t see clear structural differences between training and deployment isn t another structural difference that in deployment this round structure may be happening many times in parallel leading to increased variance in deployed models and increased chance that one of the models is misaligned e g say we have some sort of continual learning where deployed models specialize over the timeline of weeks months to different task distributions i thi read more reply pragmatic fdt and predictors as game theory 15 stuart_armstrong 4d decision theory is back in fashion defining fashion as one good post on a good ea blog bentham s bulldog bb has published a case against fdt functional decision theory contrasting rationalist enthusiasm with academic scepticism academic decision theorists don t like the theory the number of academic decision theorists who adopt it could be counted on one hand by someone missing four of their fingers i am just barely a published academic decision theorist so you can keep a small finger to count me too my position is that though fdt may have problems with its definitions and under definedness we can build defined variants that achieve what fdt attempted to i want to do two things in this post first sketch a pragmatic version of fdt designed to sidestep the continue reading 3232 more words 2 cubefox 4d it depends on what we apply the word rational to are we asking whether a fixed policy is rational or are we asking whether a decision in a particular situation branch is rational it is no contradiction to say that a policy which never pays ransom is rational and that in a situation where you have been blackmailed the decision to pay the ransom is also rational assuming this doesn t cause the probably of future blackmail attempts to increase a rational policy makes irrational decisions in certain situations cdt can be viewed as describing which decision is rational in the current branch decision situation and fdt can be viewed as describing which policy is overall rational to have across branches viewed like this the two theories are perfectly compatible they try to answer different questions the sentence a rational policy makes irrational decisions in certain situations sounds contradictory but isn t this is arguably why newcomb s problem feels paradoxical stuart_armstrong 4d 2 0 it depends on what we apply the word rational to people tend to use rational it to defend their preferred position so i find discussions often degenerate into fruitless semantic debates reply payorian cooperation is easy with kripke frames 22 transhumanist_atom_understander 4mo the context is miri s twist on axelrod s prisoner s dilemma tournament axelrod s competitors were programs facing each other in an iterated prisoner s dilemma miri s tournament is a one shot prisoner s dilemma but the programs get to read their opponent s code or rather a description of the behavior of the code in gödel löb provability logic which turns out to be enough to determine their behavior in the setup one fun result right in the beginning of the paper is about a program fairbot whose behavior is specified by i ll cooperate with you if you provably cooperate with me despite the appearance of circularity fairbot cooperates with itself the proof involves löb s theorem so we call this löbian cooperation andrew critch has suggested another way of proving self cooperation instead of löb s theorem we use what he calls payor s lemma it continue reading 2262 more words 4 james payor 5d writing to say that i quite like and agree with the perspective here i have been continuing to do my own puzzling about what would be a more wholesome instantiation would be than the exists finite length proof modality as part of a project to ground things out better though even absent that i feel confident that the intuition stands up and the processes above you re outlining in kripke terms should be realizable i also have two rambly thoughts to offer here which i ve included below if they re of interest 1 fwiw in the first place i arrived at by thinking about what the algorithm steps should be in the modelling my prisoners dilemma opponent i was very bothered by the part where we get to i notice they are checking and then fail to shorcut that s a quantity directly under our control why not counterfact on it i was then kinda surprised to find this directly encodes in modal provability logic with reasonable behaviour even though feels very much like a poor imitation of a counterfactual i thought that then obviously we should be able do the real prudentbot the naive encoding for that would be like defect if otherwise cooperate if otherwise defect this is a little different than your setup i think i m trying to ask does defecting lead to defect cooperate if so defect otherwise if cooperating leads to cooperate cooperate then cooperate otherwise defect but yep this runs directly afoul of the consistency problem inner counterfactuals do nothing to save it from that problem and to be clear the problem is that it s hard for itself and other agents to model that it won t find some clever reason to activate the defect branch and the og prudentbot is uh carefully skirting around this i still expect there s a nice answer waiting here provability should be expressive enough to capture the limit of a relevant algorithm in a clean way the translation is still eluding me 2 when i think about the ontol james payor 5d 2 0 man okay i m now wondering if the following works for a prudentbot implementation calling our prudentbot and our opponent in the encounter let or something that grows a bit bigger than let be the length of the shortest proof that let be the length of the shortest proof that if and then cooperate otherwise defect in symbols this is derived from the limit of an algorithm that searches for increasingly long proofs that we can obtain the opponent s cooperation one way or another with some budget for looking harder for a defect cooperate proof s read more reply geoffrey irving s shortform geoffrey irving 7d epistemic meristem 5d 1 0 an elegant resolution of the name collision reply rosehadshar s shortform rosehadshar 7d load more 14 data filtering works a lot worse than you would expect dohun lee j rosser josh engels neel nanda 16h 0 15 pragmatic fdt and predictors as game theory stuart_armstrong 4d 2 14 deployment awareness matters more than evaluation awareness vojtakovarik tomáš gavenčiak mateusz bagiński 11d 6 17 the case for model forensics aditya singh gersonkroiz senthooran rajamanoharan neel nanda 11d 0 18 llm driven feature discovery josh engels bilalchughtai neel nanda 15d 1 32 how transparent is diffusiongemma and why it matters josh engels callum mcdougall bilalchughtai jános kramár senthooran rajamanoharan arthur conmy rohin shah neel nanda 17d 2 43 gdm ai control roadmap mary phuong erik jenner rohin shah seb farquhar 19d 2 13 predicting llm safety before release by simulating deployment tomek korbak marcus williams micahcarroll cameron raymond hannah sheahan 2...