Meta tags:
description= A community blog devoted to technical AI alignment research;
Headings (most frequently used words):
ai, alignment, the, for, posts, popular, comments, recent, discussion, frontmatter, safe, to, dangerous, shift, is, fundamental, problem, eval, realism, forum, af, backstory, this, post, conversation, begins,
Text of the page (most frequently used words):
the (67), and (50), that (30), for (23), more (21), are (18), #alignment (18), this (17), you (15), about (11), risk (11), would (10), think (10), with (10), like (10), very (9), can (9), not (9), have (9), from (9), will (9), how (8), our (8), what (8), some (7), human (7), then (7), 2040 (7), eval (6), deployment (6), awareness (6), but (6), words (6), model (6), future (6), they (6), than (5), dangerous (5), use (5), problem (5), plan (5), expect (5), all (5), humans (5), averse (5), because (5), ais (5), good (5), possible (5), reply (4), into (4), read (4), see (4), make (4), here (4), posts (4), evaluation (4), long (4), could (4), fake (4), there (4), out (4), however (4), one (4), evidence (4), grant (4), research (4), coefficient (4), giving (4), has (4), daniel (4), thinking (4), get (4), mistakes (4), reading (4), ruthless (4), asi (4), steven (4), byrnes (4), plausible (4), stuff (4), technical (4), statements (4), nlas (4), natural (4), language (4), robust (4), neel (4), nanda (4), egregious (4), misalignment (4), most (3), these (3), much (3), matters (3), vojtakovarik (3), follow (3), both (3), strategies (3), just (3), level (3), safe (3), shift (3), already (3), data (3), evaluations (3), box (3), won (3), scheming (3), also (3), high (3), why (3), 160m (3), continue (3), reason (3), true (3), should (3), really (3), optimist (3), its (3), company (3), values (3), probability (3), satisfies (3), ultimately (3), leads (3), social (3), drives (3), initialized (3), implausible (3), autoencoders (3), claude (3), guesses (3), turntrout (3), com (3), called (3), ryan_greenblatt (3), josh (3), engels (3), load (2), though (2), taxonomy (2), important (2), differences (2), between (2), deployments (2), discussion (2), excited (2), signals (2), might (2), progress (2), give (2), thank (2), sorry (2), enough (2), agree (2), your (2), points (2), distinguish (2), making (2), released (2), across (2), other (2), great (2), charlie (2), griffin (2), core (2), past (2), realistic (2), try (2), many (2), distribution (2), difficult (2), deploy (2), approach (2), only (2), reliably (2), once (2), want (2), before (2), fundamental (2), realism (2), base (2), non (2), resolution (2), sources (2), higher (2), hard (2), 19h (2), stan (2), van (2), wingerden (2), marco (2), cozzi (2), murfet (2), jacob (2), pfau (2), alex (2), jesse (2), hoogland (2), geoffrey (2), irving (2), announcing (2), thanks (2), come (2), first (2), serial (2), those (2), next (2), well (2), absence (2), yup (2), superintelligence (2), techniques (2), sociopath (2), anyone (2), own (2), programmers (2), say (2), assuming (2), constraints (2), don (2), aversion (2), under (2), neutral (2), assumptions (2), takeover (2), matter (2), cooperation (2), capabilities (2), reduced (2), take (2), over (2), plans (2), constraint (2), problems (2), since (2), while (2), sufficiently (2), agis (2), probably (2), brain (2), agi (2), point (2), notes (2), via (2), nla (2), reconstruction (2), accuracy (2), llm (2), robustness (2), linkpost (2), https (2), michaelzhang (2), initialization (2), scenario (2), step (2), happen (2), 10h (2), bhalstead (2), romeo (2), thomas (2), larsen (2), elifland (2), kokotajlo (2), senthooran (2), rajamanoharan (2), bilalchughtai (2), 13d (2), worried (2), forum (2), javascript (2), agreed, frame, underlying, evals, when, translate, reliable, signal, attempt, gesture, strong, help, two, cents, flag, yeah, relevant, were, test, cost, slow, response, luckily, waited, posted, definitely, argue, easier, kicking, down, rode, somewhat, crucially, captures, crucial, challenging, suggesting, bounding, using, red, teaming, plenty, ruled, perhaps, seen, them, recently, diving, broader, range, 751, difference, toolemu, spoof, tool, call, responses, sad, openai, osworld, webarena, petri, create, environments, appear, approaches, faking, otherwise, rule, possibility, common, black, reassuring, extent, anything, catastrophically, suppose, capable, potentially, 2mo, patrick, leask, measuring, 501, structured, 108m, plus, 52m, conditional, combination, hiring, success, compute, needs, includes, small, regranting, budget, which, quality, back, shared, community, infrastructure, depend, sole, funder, start, goal, raise, larger, scale, funds, mixture, put, rigorous, closer, even, footing, frontier, labs, accelerate, towards, confidence, find, obstacles, showing, fka, sequent, announce, quick, clarification, presume, talking, systemat, helpful, argument, bootstrapping, issue, 100, perfect, sometimes, sft, ing, confident, round, incorporate, build, repeat, billion, times, sealed, spiral, nonsense, dumber, smarter, 16h, tom, davidson, 2143, tell, specific, strange, thing, especially, any, concrete, whatsoever, sociopathy, nature, intelligence, whatever, look, yourself, mirror, despite, right, front, eyes, llms, alas, artificial, default, yet, invented, happy, lie, cheat, steal, whenever, doing, selfishly, beneficial, callous, indifference, whether, including, users, lives, dies, fictional, conversation, begins, 5mo, let, brand, new, solved, maybe, somewhere, earth, term, optimizing, control, eats, world, near, question, else, supposed, solve, pro, said, everything, outputs, checkable, yes, skeptical, strategic, understand, helps, actually, longer, holds, seeking, constant, absolute, almost, never, plot, instead, cooperate, object, rebellion, succeeding, ensure, later, scales, imo, best, william, macaskill, elliott, thornley, comment, details, compress, insights, elicit, cheap, prices, compared, where, example, costs, getting, result, universe, importantly, verify, subjects, safely, delegate, tasks, care, reducing, catastrophic, eliminating, risks, reduce, eliminate, variance, fail, accept, smaller, amounts, guaranteed, resources, shot, terms, unlike, doesn, weak, brittleness, exploit, either, immune, negligible, reward, misspecification, malign, generalization, pointer, interpretability, known, method, motivation, applies, generally, trained, trivial, basically, noosphere89, 12563, screwed, certainly, need, prosocial, motivations, code, write, after, working, hypothetical, particular, focus, treating, innate, moral, jumping, off, intro, safety, discussed, backstory, post, frontmatter, 3679, entirely, nevertheless, achieve, nearly, same, emitting, does, train, slightly, increasing, plausibility, meant, activation, vector, describe, plain, text, training, collection, involves, asking, guess, change, various, ways, measure, impact, show, qwen2, irrelevant, prevailing, sentiments, autoencoder, total, transparency, feels, bit, too, galaxy, brained, makes, newly, discovered, usable, enhance, existing, open, weights, models, excessively, capability, levels, disincentive, overstated, neglects, mover, advantage, chris_leong, 2027, summaries, justice, whole, was, detailed, comprehensive, work, things, rather, rely, abstractions, doom, utopia, delay, creation, happened, sooner, 2030, precise, decisive, action, part, chinese, governments, aim, prediction, recommendation, year, futures, project, been, sinking, time, big, now, done, www, recent, 19d, rohin, shah, arthur, conmy, jános, kramár, callum, mcdougall, transparent, diffusiongemma, 17d, driven, feature, discovery, gersonkroiz, aditya, singh, case, forensics, mateusz, bagiński, tomáš, gavenčiak, stuart_armstrong, pragmatic, fdt, predictors, game, theory, rosser, dohun, lee, filtering, works, lot, worse, concerned, through, race, dynamics, careless, bad, actors, etc, opposed, simpler, yudkowsky, soares, story, nobody, foggiest, idea, avoid, citation, needed, people, who, pretty, emerging, naturally, being, least, moderately, prevent, sympathy, sides, debate, 1mo, popular, comments, questions, library, home, login, website, requires, properly, function, consider, activating, access, site, functionality,
Text of the page (random words):
x ai alignment forum this website requires javascript to properly function consider activating javascript to get access to all site functionality ai alignment forum af login home library questions all posts about ai alignment posts popular comments ryan_greenblatt 1mo 13 8 sympathy for both sides of the egregious misalignment debate and if they re concerned about egregious misalignment and scheming they ll probably say that it would come about through race dynamics careless programmers bad actors etc as opposed to the simpler yudkowsky soares story of we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that citation needed i think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent 15 how robust are natural language autoencoders to initialization michaelzhang turntrout 2h 0 83 ai 2040 plan a daniel kokotajlo elifland thomas larsen romeo bhalstead ryan_greenblatt 10h 1 17 announcing our 160m grant from coefficient giving geoffrey irving jesse hoogland alex ht jacob pfau daniel murfet marco cozzi stan van wingerden 19h 0 17 notes on technical alignment via human like social drives steven byrnes 1d 2 16 data filtering works a lot worse than you would expect dohun lee j rosser josh engels neel nanda 3d 0 15 pragmatic fdt and predictors as game theory stuart_armstrong 7d 2 14 deployment awareness matters more than evaluation awareness vojtakovarik tomáš gavenčiak mateusz bagiński 13d 6 17 the case for model forensics aditya singh gersonkroiz senthooran rajamanoharan neel nanda 13d 0 18 llm driven feature discovery josh engels bilalchughtai neel nanda 17d 1 32 how transparent is diffusiongemma and why it matters josh engels callum mcdougall bilalchughtai jános kramár senthooran rajamanoharan arthur conmy rohin shah neel nanda 19d 2 load more recent discussion ai 2040 plan a 83 daniel kokotajlo elifland thomas larsen romeo bhalstead ryan_greenblatt 10h this is a linkpost for https www ai 2040 com for the past year we at the ai futures project have been sinking most of our time into our next big scenario now it s done it s called ai 2040 plan a it s called plan a because it s a recommendation not a prediction it s what we think should happen not what will happen though we think it s plausible enough to aim for it s called ai 2040 because in it they delay the creation of superintelligence to 2040 it would have happened much sooner in 2030 to be precise if not for decisive action on the part of the us and chinese governments as with ai 2027 summaries don t really do it justice since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high level abstractions like doom or utopia read the scenario at ai 2040 com you can see more 68 more words chris_leong 7h 3 3 total research transparency feels a bit too galaxy brained for me it makes non robust assumptions that newly discovered techniques won t be usable to enhance already existing open weights models to excessively dangerous capability levels i also think the disincentive for research is overstated as it neglects first mover advantage reply how robust are natural language autoencoders to initialization 15 michaelzhang turntrout 2h this is a linkpost for https turntrout com natural language autoencoder robustness natural language autoencoders are meant to take in an llm s activation vector and describe in plain text what the model is thinking however its training data collection involves asking claude to guess what a model might be thinking how robust are nlas to these guesses we change claude s guesses in various ways and measure the impact on the nla s statements as well as on reconstruction accuracy we show that qwen2 5 7b nlas have some robustness to irrelevant statements and prevailing sentiments in claude s guesses however if an nla is initialized with entirely implausible statements it can nevertheless achieve nearly the same reconstruction accuracy as plausible initialized nlas while emitting 99 3 implausible statements rl does train implausible initialized nlas to be slightly more plausible increasing from 0 08 to 0 7 but the plausibility continue reading 3679 more words notes on technical alignment via human like social drives 17 steven byrnes 1d 1 frontmatter 1 1 backstory for this post as discussed in intro to brain like agi safety i m working on the technical alignment problem for a hypothetical future brain like agi with a particular focus on treating human innate social and moral drives as a possible jumping off point for our technical alignment approach after all if it s possible for humans to do stuff that ultimately leads to a good future then it s probably also possible for sufficiently human like agis to do stuff that ultimately leads to a good future or if it s not possible for humans to do stuff that ultimately leads to a good future then we re screwed no matter what but assuming it s possible the sufficiently human like agis would certainly need to have good prosocial motivations what code do we write that would continue reading 12563 more words 3 noosphere89 1d i actually just no longer think this is true and the reason this is true is because this only holds under risk neutral or risk seeking assumptions about it s values and under risk averse values like constant absolute risk aversion the ai will almost never want to plot takeover and will instead cooperate with humans no matter what it s object level values are so long as the cooperation probability is higher than the probability of a rebellion succeeding and we can ensure that the probability of cooperation is very very high and later on scales with ai capabilities imo the best reading sources for this are risk averse ais by william macaskill and elliott thornley and my own very long comment about the details of it all but to compress the insights that are important it s that we can elicit ai capabilities for very cheap prices compared to a risk neutral ai where in one example it reduced the costs of getting a good alignment result from 2 of the universe to 1 99 and importantly we can do this on hard to verify subjects like alignment and the reason we can safely delegate ais to these tasks is that they care as much about reducing catastrophic risk as about eliminating risks so they won t take over to reduce risk to 0 or eliminate variance and risk averse ais are just very averse to plans that could fail so they will accept smaller amounts of guaranteed resources over a plausible shot of takeover in terms of what constraints it satisfies it satisfies constraint 1 because unlike other alignment plans it doesn t really have many weak points brittleness to exploit and it s either immune to or is reduced to a negligible problem like reward misspecification malign generalization and pointer problems interpretability problems and there s a known method to make ais have the motivation that applies very very generally to ais trained with rl it satisfies constraint 2 since with some more stuff that while not trivial to do is basically re steven byrnes 8h 6 0 thanks sorry if you already said this but are you assuming that everything the ai outputs is reliably checkable by humans if yes i m skeptical about the strategic constraints of 6 1 or if no i don t understand how risk aversion helps let s say company a has its brand new risk averse asi in a box the problem to be solved is maybe some company b somewhere on earth will make a long term optimizing ruthless out of control asi that then eats the world in the near future and my question is how is company a or anyone else supposed to solve this pro read more reply why we should expect ruthless sociopath asi 54 steven byrnes 5mo the conversation begins fictional optimist so you expect future artificial superintelligence asi by default i e in the absence of yet to be invented techniques to be a ruthless sociopath happy to lie cheat and steal whenever doing so is selfishly beneficial and with callous indifference to whether anyone including its own programmers and users lives or dies me yup alas optimist despite all the evidence right in front of our eyes from humans and llms me yup optimist ok well i m here to tell you that is a very specific and strange thing to expect especially in the absence of any concrete evidence whatsoever there s no reason to expect it if you think that ruthless sociopathy is the true core nature of intelligence or whatever then you should really look at yourself in a mirror and continue reading 2143 more words tom davidson 16h 1 0 the first one bootstrapping has the issue that if the serial thinking is not 100 perfect then it will sometimes get mistakes and then you re sft ing on the mistakes making the model more confident in those mistakes and then the next round of serial thinking will incorporate and build on those mistakes repeat a billion times in a sealed box and i think it would spiral into nonsense it would get dumber not smarter thanks this is helpful and not an argument i ve come across before one quick clarification i presume you re here talking about systemat read more reply announcing our 160m grant from coefficient giving 17 geoffrey irving jesse hoogland alex ht jacob pfau daniel murfet marco cozzi stan van wingerden 19h we are excited to announce that resolution fka sequent has a 160m grant from coefficient giving cg to put rigorous alignment research on a closer to even footing with the frontier labs we will use it to accelerate progress towards higher confidence alignment or to find evidence and obstacles showing why alignment is hard the grant is structured as a 108m base plus 52m conditional on a combination of hiring success and compute needs the base includes a small regranting budget which we plan to use both for high quality non resolution alignment research and to give back to shared community infrastructure that we depend on coefficient giving will be our sole funder to start thank you our goal is to raise larger scale funds from a mixture of sources once we see more 501 more words the safe to dangerous shift is a fundamental problem for eval realism but also for measuring awareness 21 charlie griffin patrick leask 2mo 1 the safe to dangerous shift is a fundamental problem for eval realism suppose we have a capable and potentially scheming model and before we deploy it we want some evidence that it won t do anything catastrophically dangerous once we deploy it a common approach is to use black box alignment evaluations however alignment evaluations are only reassuring to the extent that the model can t reliably 1 distinguish the deployment distribution from the evaluation distribution as it is otherwise difficult to rule out the possibility of alignment faking there are many approaches one could use to try to make evaluations appear more realistic you can try to create realistic environments e g petri webarena osworld use data from past deployments e g openai sad and spoof tool call responses e g toolemu however the core difference between an alignment evaluation and a see more 751 more words 2 charlie griffin 3d i m sorry for the slow response luckily i waited long enough that we posted the follow up i definitely agree with both of your points that you could argue that your fake eval and fake deployment are easier to distinguish and that this is kicking the can down the rode somewhat crucially i think making a fake eval and fake deployment that captures some of the crucial strategies an ai could follow is challenging we just released a follow up suggesting bounding eval awareness of human level ai across the safe to dangerous shift using human red teaming there are plenty of other strategies signals that are not ruled out i agree this would be great perhaps you have already seen them but i think vojtakovarik et al have released some great posts recently diving into a broader range of strategies vojtakovarik 2d 1 0 thank you for the flag yeah i do think the posts deployment awareness matters more than evaluation awareness and if this were a test how much would it cost are relevant here 1 to give me two cents on this i d be very excited to see a strong taxonomy of eval deployment signals it might help make progress here agreed though i would frame this more as taxonomy of the most important underlying differences between evals and deployments discussion of when these differences translate into a reliable signal an attempt to gesture at some read more reply load more
|
