Meta tags:
description=We train and deploy an AI review agent optimised for precision and real-world use, enabling oversight to scale with autonomous code generation.;

Headings (most frequently used words):

to, code, is, than, the, you, practical, approach, verifying, at, scale, precision, more, important, for, usability, recall, repo, wide, tools, and, execution, are, necessary, reward, model, train, on, not, exactly, reviewer, should, ship, verification, can, be, cheaper, generation, what, we, learn, from, deployment, conclusion, bibtex, footnotes, want, try, codex, review,

Text of the page (most frequently used words):
the (74), and (59), code (37), codex (23), that (22), reviewer (21), review (20), with (17), human (17), for (16), issues (13), are (11), training (11), model (10), this (10), high (9), #verification (8), prs (8), generation (8), not (8), from (8), time (8), recall (8), gpt (8), can (7), but (7), has (7), real (7), precision (7), often (7), over (7), change (7), comments (7), false (7), scale (6), openai (6), repo (6), safety (6), signal (6), more (6), than (6), both (6), generated (6), practical (5), alignment (5), these (5), user (5), access (5), work (5), deployment (5), because (5), checks (5), inference (5), written (5), budget (5), alarms (5), only (5), correct (5), many (5), context (5), text (5), research (4), use (4), want (4), generator (4), part (4), workflows (4), our (4), results (4), reviewers (4), without (4), about (4), evaluation (4), how (4), models (4), must (4), they (4), when (4), critical (4), different (4), whether (4), automated (4), improves (4), may (4), approach (3), verifying (3), author (3), max (3), 2025 (3), github (3), repository (3), settings (3), you (3), cli (3), engineering (3), oversight (3), trust (3), tool (3), internal (3), already (3), previous (3), avoid (3), their (3), them (3), external (3), world (3), cannot (3), practice (3), system (3), comment (3), rather (3), its (3), while (3), figure (3), previously (3), identified (3), even (3), verifier (3), gap (3), share (3), also (3), findings (3), proposed (3), reward (3), task (3), possible (3), developer (3), most (3), execution (3), important (3), tradeoff (3), monitoring (3), trębacz (2), maja (2), arnesen (2), sam (2), cassirer (2), albin (2), johnson (2), lin (2), xin (2), sottiaux (2), thibault (2), dec (2), blog (2), your (2), cloud (2), then (2), commenting (2), same (2), methods (2), autonomous (2), shaped (2), around (2), requires (2), adoption (2), optimize (2), tax (2), aware (2), teams (2), defects (2), should (2), core (2), stronger (2), verify (2), support (2), judgment (2), one (2), toward (2), balance (2), just (2), benefit (2), under (2), distribution (2), shift (2), signals (2), grading (2), simply (2), improvements (2), lab (2), before (2), impact (2), being (2), risk (2), could (2), clean (2), layer (2), defense (2), fewer (2), cases (2), where (2), bug (2), reduce (2), deploying (2), across (2), strong (2), consistently (2), surfaces (2), actionable (2), experiments (2), launch (2), blocking (2), problems (2), what (2), learn (2), tasks (2), attention (2), there (2), rely (2), effective (2), suggesting (2), right (2), evaluated (2), set (2), codebase (2), fraction (2), higher (2), potential (2), further (2), study (2), run (2), additional (2), however (2), observe (2), performance (2), compared (2), limits (2), humans (2), between (2), generating (2), errors (2), makes (2), out (2), result (2), quality (2), therefore (2), designed (2), monitor (2), catching (2), avoiding (2), intent (2), produced (2), noise (2), incorrect (2), diff (2), providing (2), dedicated (2), wide (2), tools (2), bugs (2), guidelines (2), output (2), other (2), technically (2), times (2), alarm (2), every (2), expected (2), systems (2), misc, trebacz2025verifyingcodeatscale, title, year, month, howpublished, url, https, com, scaling, bibtex, connect, turn, once, enabled, auto, summon, anytime, pull, request, try, used, teach, two, skills, differ, footnotes, becomes, everyday, low, earning, show, deliver, reliable, feedback, slowing, down, preparing, distant, future, uncovered, production, exposed, inconsistencies, trusted, datasets, see, rich, next, frontier, factors, monitors, interact, developers, strengthen, brittle, adversarial, dynamics, maintaining, control, sits, philosophy, become, generators, ability, critique, capable, step, keeping, conclusion, matters, provides, widely, accessible, tests, assumptions, gives, outcome, offline, helps, ensure, transfer, october, handles, 100k, per, day, alongside, pre, triggered, via, catch, ever, reach, early, promising, reactions, positive, reliance, serious, start, treating, guarantee, people, understand, replacement, careful, encouragingly, seen, merged, later, need, fix, follow, which, aligns, helping, escaped, shifting, effort, internally, traffic, been, crucial, turning, prototype, into, dependable, constraints, now, workflow, leaves, authors, address, assisted, showing, helped, validate, stakes, prevented, averted, multiple, failures, since, underlying, expect, raw, intelligence, increase, parallel, pay, close, genuine, advantage, persists, learns, subtly, game, own, direct, measurement, proxies, compare, versus, acknowledging, bias, far, remains, similarly, meaningfully, weakened, function, left, lower, budgets, recovers, substantial, known, recovery, distributional, difficulty, respect, controlled, take, verified, sweep, small, token, spend, catches, large, severity, confirmed, mostly, calibration, reduces, drops, rapidly, thinking, reviewing, test, includes, have, proposes, new, insights, given, tell, those, input, exploit, broad, search, tokens, falsifying, usually, needs, targeted, hypothesis, identifying, were, entirely, making, top, cheaper, using, single, risks, failure, optimizes, please, during, behaviours, harm, downstream, such, overly, cautious, stylised, outputs, frustrate, users, view, separate, closely, effectiveness, itself, facing, appear, similar, solve, fundamentally, require, designs, prioritise, mistakes, acceptable, sensitive, extra, information, allowing, enforce, precise, instruction, following, needing, infer, deployed, inverse, priorities, operate, ambiguous, incomplete, specifications, evolving, conventions, asserting, remain, robust, languages, domains, importantly, build, train, exactly, ship, recent, commits, popular, open, source, repositories, trained, specifically, ratio, less, likely, unimportant, reserving, default, prompt, able, identify, numerous, produces, number, abilities, found, raising, earlier, june, 2024, was, simpler, wasn, suitable, adds, reasoning, latency, targets, clear, bar, attempts, relied, optionally, brief, surrounding, fastest, misses, whole, interaction, dependencies, criticgpt, necessary, spotting, spurious, steer, ranging, certain, breaking, simultaneously, special, scaffolding, made, opinionated, guess, believe, allow, steerable, custom, instructions, package, level, specification, agents, stylistic, nature, incur, negative, utility, worth, pointing, typo, personal, notebook, saved, aim, flag, issue, present, misinterpreting, intentions, seeing, finding, outweigh, cost, damage, maximize, defenses, fail, wrong, impractical, chooses, slow, noisy, cumbersome, will, bypassed, agent, explicitly, accepted, measured, modestly, reduced, exchange, first, push, compromising, reliability, usability, post, learned, agentic, discuss, giving, considerations, guide, minimal, ideas, aren, theoretical, within, automatically, reviewed, engineers, pushing, protected, value, caught, assume, trustworthy, check, complements, action, strategy, depth, honesty, behavioral, testing, activation, chain, thought, collaborative, coding, proliferate, volume, quickly, exceeds, thorough, widens, does, introduces, severe, vulnerabilities, accidentally, intentionally, collaboration, rest, team, back,

Text of the page (random words):
a practical approach to verifying code at scale back to openai alignment blog a practical approach to verifying code at scale dec 1 2025 maja trębacz sam arnesen albin cassirer max johnson xin lin thibault sottiaux in collaboration with the rest of the codex team as autonomous collaborative coding systems proliferate the volume of produced code quickly exceeds the limits of thorough human oversight as this gap widens so does the risk that ai written code introduces severe bugs and vulnerabilities whether accidentally or intentionally we cannot assume that code generating systems are trustworthy or correct we must check their work automated code review is a practical output monitor that complements chain of thought monitoring action monitoring internal activation monitoring behavioral testing honesty training and other safety work as part of a defense in depth strategy in this post we share what we learned from training a dedicated agentic code reviewer as part of gpt 5 codex and gpt 5 1 codex max 1 we discuss how giving the reviewer repo wide tools and execution access improves both recall and precision and how deployment time considerations guide us toward high signal settings with minimal alignment tax these ideas aren t theoretical within openai every pr is automatically reviewed many engineers run review in the codex cli before pushing and the model has protected high value experiments and caught launch blocking issues precision is more important for usability than recall defenses often fail not because they are technically wrong but because they are so impractical that the user chooses not to use them a system that is slow noisy or cumbersome will be bypassed when deploying the code review agent we explicitly accepted a measured tradeoff modestly reduced recall in exchange for high signal quality and developer trust we optimize for signal to noise first and only then push recall without compromising reliability a code reviewer could aim to flag every possible issue present in the proposed code change in practice many issues are false alarms or the result of misinterpreting user intentions we want the expected benefit from seeing a proposed bug finding to outweigh the expected cost to verify it and the damage from a false alarm that is we want findings that maximize p text correct times c_ text saved c_ text human verification p text incorrect times c_ text false alarm comments that are technically correct but more of a stylistic nature may even incur negative utility e g it may not be worth pointing out a comment typo in a personal research notebook while we ve made an opinionated guess about the right balance of precision and recall we believe it s important to allow this tradeoff and other guidelines to be steerable by the custom task instructions or package or repo level agents md specification figure 1 there is a tradeoff between spotting real bugs recall and avoiding commenting on spurious issues precision we can steer the models with different developer guidelines ranging from only output the most critical and certain breaking issues to share as many findings as possible gpt 5 1 codex simultaneously improves both recall and precision over gpt 5 with special scaffolding and repository access repo wide tools and execution are necessary earlier research e g criticgpt june 2024 shaped our methods but was designed for simpler tasks and wasn t suitable for deployment in practice this reviewer adds reasoning tool use repo scale context and precision latency targets to clear the adoption bar most of the previous attempts at code review relied on providing just a diff of the change to the model and optionally a brief surrounding context this results in the fastest review time however it often misses important context about the whole codebase and the interaction of the change with its dependencies we evaluated providing repository access and code execution abilities to a gpt 5 model and found that it results in a stronger reviewer catching more critical issues and raising fewer false alarms dedicated training for code review further improves the results figure 2 human evaluation of the code review performance on recent commits from popular open source repositories gpt 5 codex trained specifically for higher signal to noise ratio makes comments that are less likely to be incorrect or unimportant reserving user attention for critical issues with a default prompt and access only to the context of the pr diff gpt 5 is able to identify numerous high impact comments but also produces a high number of false alarms the reward model you train on is not exactly the reviewer you should ship verification at training time and human facing review may appear similar but they solve fundamentally different problems and therefore require different designs when training models for code generation we rely on automated checks to reduce errors at scale and prioritise catching as many potential mistakes as possible rather than avoiding false alarms it is acceptable for these reward models to be over sensitive these training time checks can also use extra information about the task allowing them to enforce precise instruction following without needing to infer the developer s intent deployed code review has the inverse priorities a reviewer must operate on ambiguous real world code produced by humans or human ai workflows often with incomplete specifications and evolving conventions it must avoid over asserting intent remain robust across languages and domains and most importantly build user trust using a single verifier for both settings risks failure in both if a generator over optimizes to please the reward signal during training it may learn behaviours that harm downstream review quality such as overly cautious or stylised outputs that frustrate users we therefore view training context aware reviewers as a separate task in codex training designed for human workflows and closely monitor its effectiveness at grading codex generation itself verification can be cheaper than generation we observe the automated code reviewer to be effective at identifying errors in both human and codex generated prs codex code review makes comments on 36 of the prs that were entirely generated by codex cloud out of these comments 46 result in the author making a code change this is compared to 53 on the comments on top of the human generated prs we can exploit an inference time gap between generation and verification generating a correct code change often requires broad search and many tokens while falsifying a proposed change usually needs only targeted hypothesis generation and checks to further study the verification generation gap with respect to inference budget we run a controlled study where we take previously human verified comments from a previous generation of a strong code review model and sweep the code reviewer inference budget even at a small fraction of the generator s token spend the verifier catches a large share of previously identified high severity issues confirmed by the pr author and the additional budget mostly improves calibration and reduces false alarms however we also observe that the performance drops more rapidly with thinking budget on reviewing model generated code compared to the human written this evaluation has limits because the test set only includes issues humans have already identified if the reviewer proposes new insights when given more time this evaluation cannot tell us whether those findings are correct without additional human input figure 3 recall of the automated reviewer gpt 5 1 codex as a function of inference budget on codex generated left and human written right code evaluated on a set of previously identified issues from the openai codebase even at lower budgets the verifier recovers a substantial fraction of known issues with consistently higher recovery on human written code suggesting a potential distributional shift in verification difficulty since codex code generator and code reviewer are the same underlying model with different tasks we expect their raw intelligence to increase in parallel over time we pay close attention to whether a genuine verification advantage persists at inference time and whether the model learns to subtly game or avoid its own checks there is no clean direct measurement of this so we rely on practical proxies and compare how often the reviewer surfaces actionable issues on codex generated prs versus human written prs while acknowledging distribution bias so far the reviewer remains similarly effective suggesting that oversight is not being meaningfully weakened what we learn from deployment deploying the reviewer both internally and across external github traffic has been crucial for turning a strong lab prototype into a dependable safety layer under real engineering constraints the code review model is now a core part of openai s engineering workflow when the reviewer leaves a comment authors address it with a code change in 52 7 of cases often assisted by codex showing that the system consistently surfaces actionable issues the reviewer has already helped validate high stakes experiments prevented launch blocking problems and averted multiple critical failures over reliance is a serious risk teams could start treating a clean review as a guarantee of safety rather than as one layer of defense we want people to understand that the reviewer is a support tool not a replacement for careful judgment encouragingly we ve seen fewer cases where merged prs later need bug fix follow up work which aligns with the reviewer helping reduce escaped defects rather than simply shifting effort around as of october 2025 the system handles more than 100k external prs per day alongside internal pre pr checks triggered via the review cli that often catch issues before they ever reach github early signals of real world impact are promising with over 80 of comment reactions being positive external deployment matters not just because it provides a widely accessible safety benefit but because it tests research assumptions under real world distribution shift it gives us outcome signals that offline grading simply cannot and helps ensure that improvements in the lab transfer to improvements in practice conclusion as autonomous code generation becomes part of everyday engineering oversight has to be shaped around real workflows safety requires adoption so we optimize the reviewer for low safety tax and high precision earning user trust our results show that repo aware reviewers with tool access can deliver reliable high signal feedback without slowing teams down this work is not about preparing for a distant future internal deployment has already uncovered real production defects and exposed evaluation inconsistencies in our previous trusted datasets we see a rich next frontier in human factors work on how reviewers and monitors should interact with developers strengthen workflows and avoid brittle or adversarial dynamics maintaining human control sits at the core of our alignment philosophy and as models become stronger generators their ability to verify critique and support human judgment must scale with them training capable reviewers is one step toward keeping that balance footnotes 1 the codex code generator and code reviewer are the same model but the training methods used to teach these two skills differ want to try codex code review use review in codex cli or connect your github repo to codex cloud then turn on code review in your codex repository settings once enabled codex can auto review prs or you can summon it anytime by commenting codex review on a pull request bibtex misc trebacz2025verifyingcodeatscale title a practical approach to verifying code at scale author trębacz maja and arnesen sam and cassirer albin and johnson max and lin xin and sottiaux thibault year 2025 month dec howpublished openai alignment research blog url https alignment openai com scaling code verification