Meta tags:
description=Compare AI inference performance across GPUs and frameworks. Real benchmarks on NVIDIA GB200, B200, AMD MI355X, and more. Free, open-source, continuously updated.;
author=SemiAnalysis;
keywords=AI inference benchmark,GPU benchmark,LLM benchmark,inference performance,NVIDIA benchmark,AMD benchmark,GPU performance comparison,AI model benchmarks,tokens per second,TTFT benchmark,time to first token,inference latency,ML inference,vLLM benchmark,TensorRT-LLM benchmark,GPU CI/CD,open source benchmark,SemiAnalysis,InferenceMAX,InferenceX,GB200 benchmark,MI355X benchmark,H100 benchmark,A100 benchmark;

Headings (most frequently used words):

vs, b200, mi355x, disagg, deepseek, sglang, open, source, continuous, inference, benchmark, trusted, by, gigawatt, token, factories, full, dashboard, every, result, is, transparently, done, through, public, github, actions, automation, quick, comparisons, minimax, m3, first, looknew, gb200, nvl72, multi, single, node, h200, blackwell, hopper, amd, mi300x, mi325x, h100, gb300, trtllm, over, time, fp8,

Text of the page (most frequently used words):
the (72), and (68), #inference (29), that (29), open (25), across (21), benchmarks (19), with (17), performance (16), reproducible (16), inferencex (16), real (14), deepseek (14), hardware (14), sglang (13), data (12), fp8 (12), b200 (12), benchmark (12), how (12), for (12), ecosystem (12), per (11), gpu (11), every (11), software (11), inferencemax (10), source (9), world (9), disagg (9), mi355x (9), throughput (9), community (9), transparent (9), this (8), helps (8), dynamo (8), amd (8), minimax (8), models (8), scale (8), new (7), are (7), see (7), vllm (7), from (7), stacks (7), gpus (7), cloud (7), customers (7), github (6), infrastructure (6), nvl72 (6), into (6), full (6), public (6), cost (6), efficiency (6), can (6), continuously (6), pytorch (6), semianalysis (5), dollar (5), like (5), trtllm (5), fp4 (5), gb300 (5), actions (5), built (5), make (5), mission (5), latest (5), frontier (5), entire (5), many (5), tokens (5), about (4), benchmarking (4), meta (4), oracle (4), microsoft (4), h100 (4), h200 (4), gb200 (4), model (4), run (4), progress (4), dashboard (4), qwen (4), when (4), watt (4), updated (4), platforms (4), workloads (4), level (4), platform (4), providing (4), build (4), confidence (4), give (4), more (3), continuous (3), auditable (3), openai (3), disaggregated (3), time (3), mi325x (3), mi300x (3), first (3), most (3), comparisons (3), explore (3), framework (3), through (3), nvidia (3), kimi (3), white (3), what (3), was (3), its (3), industry (3), exactly (3), founder (3), ceo (3), lambda (3), compute (3), teams (3), researchers (3), actually (3), hugging (3), face (3), everyone (3), major (3), stack (3), critical (3), insights (3), forward (3), chief (3), weka (3), accelerators (3), cache (3), having (3), decisions (3), tensorwave (3), because (3), track (3), numbers (3), nebius (3), performs (3), move (3), our (3), coreweave (3), needs (3), strengthens (3), vendor (3), neutral (3), head (3), moonshot (3), all (2), your (2), 229 (2), star (2), policy (2), trusted (2), operators (2), etc (2), over (2), mori (2), blackwell (2), hopper (2), available (2), configurations (2), jump (2), straight (2), runs (2), browse (2), week (2), repo (2), executes (2), logs (2), visible (2), point (2), produced (2), click (2), transparently (2), soon (2), oss (2), other (2), supporters (2), house (2), senior (2), has (2), engineer (2), red (2), hat (2), san (2), diego (2), distserve (2), simple (2), decode (2), whole (2), where (2), exists (2), accessible (2), labs (2), measure (2), perform (2), raises (2), bar (2), stanford (2), simon (2), optimizations (2), provides (2), much (2), evolves (2), moving (2), makes (2), officer (2), memory (2), evolve (2), excited (2), offloading (2), xpus (2), building (2), these (2), smarter (2), product (2), lead (2), engine (2), factories (2), such (2), nightly (2), have (2), today (2), leading (2), marketing (2), actionable (2), peter (2), delivering (2), really (2), support (2), effort (2), brings (2), clarity (2), executive (2), director (2), kernels (2), driven (2), users (2), mode (2), out (2), help (2), tri (2), dao (2), systems (2), strategies (2), vice (2), president (2), reflect (2), transparency (2), essential (2), azure (2), token (2), kind (2), alibaba (2), glad (2), 2026, com, rights, reserved, work, consider, starring, sharing, network, 中文版, reliability, frontend, contribute, cookie, privacy, land, acknowledgement, legal, newsletter, main, site, trillion, timeline, tracks, improvements, three, generations, instinct, generational, scaling, multi, single, node, appear, here, they, come, online, look, popular, curated, ready, quick, works, view, submissions, database, published, release, historical, dataset, stays, weekly, snapshots, precision, parallelism, setting, committed, shell, script, recipes, while, configuration, lands, 000, datapoints, added, average, workflow, recipe, lives, actual, target, artifacts, publicly, viewable, any, chart, result, done, automation, compare, b300, vr200, mi455x, ualoe72, tpuv7, ironwood, deepseekv4, pro, glm, gpt, llama, metric, fully, configurable, charts, date, ranges, concurrency, sweeps, raw, export, advisor, sriram, krishnan, now, commonly, hearing, want, testament, dylan522p, core, maintainer, principal, michael, goin, good, sir, assistant, professor, creator, fastvideo, hao, zhang, introduced, thesis, split, prefill, optimize, each, own, terms, eighteen, months, later, disaggregation, default, architecture, comparing, aggregated, serving, pareto, curve, shows, separation, pays, off, ttft, tpot, stephen, balaban, individual, largest, aligns, giving, matters, informed, choices, grounded, clement, delangue, extends, chip, pulling, directly, hub, them, accelerator, phd, student, guo, working, understanding, techniques, interact, yet, incredibly, hard, needed, field, gaps, val, bercovici, ensembles, precisely, xpu, tpus, lpus, storage, roadmap, include, agentic, coding, stress, cpu, nvme, wall, feeds, visibility, invest, mingyi, behind, production, xai, grok, earning, recognition, king, firsthand, varies, capturing, variance, way, does, reproducibly, darrick, horton, next, generation, believe, innovation, thrives, strong, alternatives, reinforces, vision, cutting, synthetic, highlighting, potential, business, roman, chernin, sets, standard, reveal, measuring, beyond, claims, provider, initiative, ensure, aligned, cto, salanki, measures, turns, abstract, insight, fast, space, foundation, matt, demonstrates, operate, practice, tensorrt, llm, show, innovations, runtimes, frameworks, translate, measurable, range, including, being, running, offers, approach, tracking, mark, saroufim, arguably, important, suite, project, collaborate, team, diverse, scenarios, trust, reference, will, fair, measurements, drive, layer, architectures, engines, scientist, together, inventor, flash, attention, gap, between, theoretical, peak, often, determined, distributed, low, valuable, showing, mtp, speculative, wide, play, various, results, faster, jay, jackson, enterprises, flexibility, choice, skus, confidently, select, best, align, their, super, intelligence, lab, joseph, spisak, belief, tools, accelerate, embodies, same, philosophy, serve, billions, baseline, practitioners, making, scott, guthrie, performant, efficient, effective, supports, under, realistic, ability, tune, helping, industrial, stargate, hoeschele, unprecedented, cut, noise, provide, living, picture, megawatt, datacenters, always, been, putting, capable, hands, developers, possible, rigorous, measurement, questions, matter, qwen3, independent, deploy, driving, dedicated, supporting, advancing, series, tracked, better, understand, encourages, keep, improving, optimizing, devrel, ryan, lee, both, capability, deployment, mind, day, one, reflects, collaborative, spirit, proud, part, gigawatt, live, articles, datasets, home,

Text of the page (random words):
open source ai inference benchmark inferencex by semianalysis inferencex by home dashboard comparisons supporters datasets articles about star 1 229 中文 minimax m3 benchmarks are live new first inference numbers across nvidia and amd gpus click to explore explore open source continuous inference benchmark trusted by gigawatt token factories minimax moonshot kimi alibaba qwen openai microsoft meta oracle tri dao vllm gpu mode pytorch coreweave nebius tensorwave sglang weka stanford hugging face lambda uc san diego red hat white house vendor neutral continuously updated benchmarking is essential as models and inference stacks co evolve minimax m3 was built with both frontier capability and real world deployment efficiency in mind and the day one vllm support from the community reflects the collaborative spirit we re proud to be part of inferencex provides the kind of transparent reproducible data the ecosystem needs ryan lee head of devrel minimax at moonshot ai we are dedicated to supporting the open source ecosystem by advancing frontier open models as the kimi k2 series evolves we are glad to see its performance tracked in inferencex s open and reproducible benchmarks inferencex helps the community better understand industry level performance and encourages the ecosystem to keep improving and optimizing moonshot ai qwen has always been about putting capable models into the hands of as many developers as possible and real world inference efficiency is what makes that scale inferencex brings rigorous vendor neutral measurement to exactly the questions that matter how models like qwen3 5 actually perform across accelerators independent reproducible benchmarks on real hardware give the community the clarity it needs to deploy with confidence and we re glad to see that level of transparency driving the inference ecosystem forward alibaba qwen as we build systems at unprecedented scale it s critical for the ml community to have open transparent benchmarks that reflect how inference really performs across hardware and software inferencemax s head to head benchmarks cut through the noise and provide a living picture of token throughput performance per dollar and tokens per megawatt this kind of open source effort strengthens the entire ecosystem and helps everyone from researchers to operators of frontier datacenters make smarter decisions peter hoeschele vp of infrastructure and industrial compute openai stargate our mission at azure is to give customers the most performant efficient and cost effective cloud for ai semianalysis inferencemax supports that mission by providing transparent reproducible benchmarks that track inference performance across gpus and software stacks under realistic workloads this continuous data on throughput efficiency and cost per watt strengthens our ability to tune azure s inference platform for scale helping customers build with confidence on microsoft cloud scott guthrie executive vice president microsoft cloud ai pytorch was built on the belief that open tools accelerate the entire ai ecosystem inferencex embodies that same philosophy open reproducible and vendor neutral benchmarks that give the community real data on real hardware as inference workloads scale to serve billions of users having a continuously updated transparent performance baseline across accelerators is essential for practitioners and platform teams making critical infrastructure decisions joseph spisak product director meta super intelligence lab oracle cloud infrastructure is built to give frontier labs enterprises flexibility and choice with many gpu skus available for ai at scale inferencemax strengthens that mission by delivering open source reproducible benchmarks that reflect real world performance efficiency and cost on the latest hardware and software with this transparency customers can confidently select the platforms that best align with their ai strategies jay jackson vice president oracle cloud infrastructure the gap between theoretical peak and real world inference throughput is often determined by systems software inference engine distributed strategies and low level kernels inferencemax is valuable because it benchmarks the latest software showing how optimizations like fp4 mtp speculative decode and wide ep actually play out across various hardware open reproducible results like these help the whole community move faster tri dao chief scientist of together ai inventor of flash attention the industry needs many public reproducible benchmarks of inference performance we re excited to collaborate with inferencemax from the vllm team more diverse workloads and scenarios that everyone can trust and reference will help the ecosystem move forward fair transparent measurements drive progress across every layer of the stack from model architectures to inference engines to hardware simon mo vllm project co lead arguably the most important oss benchmark suite out today inferencex mark saroufim gpu mode founder meta pytorch engineer inferencemax demonstrates how an open ecosystem can operate in practice many leading inference stacks such as vllm sglang and tensorrt llm are built on pytorch and benchmarks like this show how innovations across kernels runtimes and frameworks translate into measurable performance on a range of hardware platforms including nvidia and amd gpus by being open source and running nightly inferencemax offers a transparent community driven approach to tracking progress and providing pytorch users with data driven insights matt white executive director pytorch foundation inferencemax raises the bar by delivering open transparent benchmarks that track how inference really performs across the latest gpus and software stacks for customers having reproducible data that measures real world tokens per dollar tokens per watt turns abstract marketing numbers into actionable insight at coreweave we support this effort because it brings clarity to a fast moving space and helps the entire ecosystem build with confidence peter salanki cto coreweave inferencemax sets a new standard by providing open transparent benchmarks that reveal how inference performs across today s leading gpus and software stacks with reproducible data measuring real world tokens per dollar and tokens per watt customers can move beyond marketing claims to actionable insights for us at nebius as a full stack ai cloud provider this initiative helps us build our inference platform with confidence and ensure we are aligned with the ecosystem roman chernin co founder chief business officer nebius at tensorwave we re building a next generation cloud on amd gpus because we believe innovation thrives when customers have strong alternatives inferencemax reinforces that vision by providing open source reproducible benchmarks that track throughput efficiency and cost across the latest hardware and software by cutting through synthetic numbers and highlighting real world inference performance it helps customers see the full potential of amd platforms for ai at scale darrick horton ceo tensorwave sglang is the inference engine behind many production inference factories such as xai s grok earning its recognition as the inference king at scale we see firsthand how much performance varies across hardware models and configurations inferencex benchmarks sglang across every major gpu platform nightly capturing that variance in a way no other benchmark does continuously reproducibly mingyi lu sglang product lead inferencex ensembles precisely that open reproducible benchmarks that are continuously updated as xpu accelerators gpus tpus lpus memory storage and software stacks evolve i m excited to see the inferencex benchmarking roadmap include agentic coding workloads that stress cpu kv cache offloading soon nvme kv cache offloading from xpus as weka helps scale the memory wall by building the kv cache infrastructure that feeds these xpus having this level of visibility into inference performance helps the entire ecosystem make smarter decisions about where to invest val bercovici chief ai officer weka for researchers working on inference optimizations understanding how new techniques interact across the software and hardware stack is critical yet incredibly hard to measure inferencex provides much needed insights into how inference performance evolves across major hardware platforms moving the field forward with open reproducible data that makes the gaps and progress visible simon guo phd student stanford cs hugging face exists to make ai open and accessible to everyone inferencex extends that mission to ai chip performance pulling models directly from the hub and benchmarking them across every major accelerator continuously and transparently when the community can see exactly how frontier open models perform on real hardware in real time it raises the bar for the entire ecosystem clement delangue ceo hugging face lambda exists to make gpu compute simple and accessible for ai teams from individual researchers to the largest labs inferencex aligns with that mission by giving the community open reproducible benchmarks that measure what actually matters real world throughput cost efficiency and performance per watt across the latest hardware and software stacks teams can make informed compute choices grounded in transparent continuously updated data stephen balaban co founder and ceo lambda when we introduced distserve the thesis was simple split prefill and decode and optimize each on its own terms eighteen months later disaggregation is the default architecture across the industry inferencex is the benchmark that comparing disaggregated and aggregated serving across the whole pareto curve inferencex shows exactly when and where p d separation pays off in ttft tpot throughput and cost hao zhang assistant professor uc san diego co creator of distserve vllm and fastvideo the benchmark is good sir michael goin vllm core maintainer senior principal engineer at red hat now commonly hearing we want the semianalysis for x testament to what dylan522p has built sriram krishnan white house senior ai advisor see more supporters full dashboard every model gpu framework and metric fully configurable inference benchmark charts with date ranges concurrency sweeps and raw data export compare nvidia gb300 nvl72 gb200 nvl72 b300 b200 h200 h100 amd mi355x mi325x mi300x and soon vr200 nvl72 amd mi455x ualoe72 tpuv7 ironwood etc across deepseekv4 pro qwen kimi glm minimax gpt oss llama and other models open dashboard every result is transparently done through public github actions automation every data point on the dashboard is produced by a public github actions workflow run the recipe lives in the repo the run executes on the actual target hardware and the full logs and artifacts are publicly viewable click any point on a chart to jump straight to the run that produced it all reproducible auditable and open source 1 000 new benchmark datapoints added per week on average browse every new model gpu framework and configuration as it lands public actions runs every benchmark executes on github actions with full logs visible while the run is in progress open recipes every model framework precision and parallelism setting is committed to the public repo as a shell script weekly db snapshots the full benchmark database is published as a public github release every week so the historical dataset stays auditable browse submissions view benchmark runs on github actions how it works quick comparisons jump straight into the most popular gpu inference benchmark comparisons curated and ready to explore minimax m3 first look new first benchmarks of minimax m3 across every available gpu new configurations appear here as they come online minimax m3 gb200 nvl72 vs b200 multi vs single node gb200 nvl72 dynamo trtllm vs b200 dynamo trtllm on deepseek r1 8k 1k at fp4 deepseek gb200 b200 dynamo fp4 nvl72 b200 vs h200 blackwell vs hopper blackwell b200 vs hopper h200 dynamo trtllm throughput per gpu on deepseek r1 8k 1k at fp8 deepseek b200 h200 dynamo fp8 amd mi300x mi325x mi355x three generations of amd instinct on sglang at fp8 generational throughput scaling on deepseek r1 8k 1k deepseek mi300x mi325x mi355x sglang fp8 h100 vs gb300 disagg deepseek h100 fp8 disagg vs gb300 fp8 disagg vs gb300 fp4 disagg on deepseek r1 8k 1k deepseek h100 gb300 disagg fp8 fp4 disagg b200 sglang vs mi355x vs b200 trtllm disaggregated b200 dynamo sglang vs mi355x mori sglang vs b200 dynamo trtllm on deepseek r1 8k 1k at fp8 deepseek b200 mi355x dynamo mori fp8 disagg mi355x sglang disagg over time deepseek fp8 mi355x sglang disaggregated inference on deepseek r1 8k 1k fp8 tracks throughput improvements over time deepseek mi355x sglang fp8 disagg timeline continuous open source inference benchmarking real world reproducible auditable performance data trusted by trillion dollar ai infrastructure operators like openai meta oracle microsoft etc semianalysis main site newsletter about legal land acknowledgement privacy policy cookie policy contribute benchmarks frontend more gpu reliability performance per dollar 中文版 star 1 229 if this data helps your work consider starring us on github or sharing with your network 2026 semianalysis com all rights reserved