Meta tags:
description=InferenceX initiative is supported by major buyers of compute and prominent members of the ML community including those from MiniMax, Moonshot Kimi, Alibaba Qwen, OpenAI, Microsoft, vLLM, PyTorch Foundation, Oracle and more.;
author=SemiAnalysis;
keywords=AI inference benchmark,GPU benchmark,LLM benchmark,inference performance,NVIDIA benchmark,AMD benchmark,GPU performance comparison,AI model benchmarks,tokens per second,TTFT benchmark,time to first token,inference latency,ML inference,vLLM benchmark,TensorRT-LLM benchmark,GPU CI/CD,open source benchmark,SemiAnalysis,InferenceMAX,InferenceX,GB200 benchmark,MI355X benchmark,H100 benchmark,A100 benchmark;

Headings (most frequently used words):

inferencex, initiative, supporters,

Text of the page (most frequently used words):
the (130), and (126), that (59), open (42), #inference (41), real (36), inferencex (36), across (34), performance (33), benchmarks (33), for (30), hardware (29), software (27), reproducible (26), with (25), inferencemax (23), world (22), transparent (22), this (19), community (19), workloads (19), how (19), scale (19), data (18), our (18), ecosystem (17), efficiency (16), continuously (16), per (15), cost (15), throughput (15), gpus (14), customers (14), amd (14), from (13), ceo (13), can (13), source (12), infrastructure (12), see (12), models (12), stacks (12), gpu (11), make (11), founder (11), into (11), platform (11), what (11), helps (10), are (10), every (10), actually (10), stack (10), cloud (10), updated (10), latest (10), like (9), vllm (9), provides (9), dollar (8), about (8), benchmarking (8), system (8), benchmark (8), built (8), full (8), has (8), systems (8), kind (8), progress (8), compute (8), entire (8), semianalysis (7), cache (7), they (7), gives (7), many (7), teams (7), decisions (7), industry (7), providing (7), frontier (7), nvidia (7), more (6), production (6), through (6), because (6), these (6), labs (6), building (6), everyone (6), exactly (6), give (6), mission (6), tokens (6), nvl72 (6), nightly (6), platforms (6), pytorch (6), microsoft (5), long (5), chief (5), where (5), show (5), major (5), matters (5), vendor (5), having (5), performs (5), level (5), results (5), track (5), watt (5), developers (5), build (5), insights (5), was (5), confidence (5), openai (4), oracle (4), cto (4), serving (4), engine (4), agentic (4), clear (4), context (4), move (4), between (4), memory (4), way (4), measure (4), including (4), much (4), team (4), marketing (4), giving (4), needed (4), diverse (4), understand (4), essential (4), llm (4), such (4), effort (4), clarity (4), help (4), critical (4), needs (4), project (4), most (4), researchers (4), driven (4), best (4), delivering (4), forward (4), next (4), when (4), accelerators (4), sglang (4), neutral (4), work (3), continuous (3), operators (3), meta (3), creator (3), lmcache (3), tensormesh (3), simple (3), not (3), one (3), directly (3), use (3), matter (3), very (3), visible (3), whether (3), works (3), day (3), model (3), kernels (3), engineering (3), enterprises (3), believe (3), important (3), have (3), helping (3), better (3), efficient (3), driving (3), enabling (3), accessible (3), means (3), delivers (3), fast (3), supports (3), brings (3), cutting (3), deliver (3), under (3), engineer (3), time (3), been (3), gap (3), glad (3), makes (3), officer (3), factories (3), evolve (3), confidently (3), their (3), excited (3), running (3), proud (3), support (3), senior (3), director (3), mi300 (3), leading (3), new (3), innovation (3), its (3), coding (3), lead (3), initiative (3), vice (3), president (3), strengthens (3), azure (3), head (3), glm (3), qwen (3), moonshot (3), minimax (3), supporters (3), reserved (2), your (2), sharing (2), 273 (2), star (2), policy (2), state (2), inside (2), repeated (2), shared (2), storage (2), other (2), those (2), prefill (2), scientist (2), same (2), tools (2), conversation (2), get (2), capacity (2), useful (2), right (2), embeddedllm (2), deep (2), maintainer (2), turns (2), need (2), gmi (2), alex (2), become (2), done (2), something (2), runtimes (2), distributed (2), reflects (2), researcher (2), lab (2), engines (2), agents (2), capable (2), improving (2), provide (2), feedback (2), loop (2), field (2), step (2), view (2), tco (2), well (2), periodic (2), each (2), cut (2), practitioners (2), voltage (2), park (2), saurabh (2), access (2), instead (2), guessing (2), verda (2), edge (2), without (2), want (2), token (2), second (2), always (2), gb300 (2), soon (2), adaptive (2), generation (2), rack (2), measurements (2), theoretical (2), firmus (2), energy (2), you (2), prime (2), intellect (2), vultr (2), choose (2), supermicro (2), measures (2), tool (2), crusoe (2), being (2), choice (2), select (2), mi325 (2), mi355 (2), will (2), faster (2), inventor (2), highlights (2), comparisons (2), approach (2), dynamo (2), disaggregated (2), speed (2), demand (2), grace (2), blackwell (2), era (2), megawatt (2), effective (2), trust (2), accelerate (2), visibility (2), white (2), distserve (2), decode (2), optimize (2), architecture (2), whole (2), lambda (2), exists (2), hugging (2), face (2), perform (2), raises (2), bar (2), phd (2), student (2), simon (2), optimizations (2), understanding (2), interact (2), evolves (2), moving (2), weka (2), offloading (2), xpus (2), smarter (2), product (2), tensorwave (2), numbers (2), potential (2), nebius (2), today (2), actionable (2), coreweave (2), peter (2), really (2), executive (2), foundation (2), users (2), out (2), layer (2), together (2), strategies (2), reflect (2), transparency (2), realistic (2), rigorous (2), measurement (2), deploy (2), alibaba (2), kimi (2), 2026, com, all, rights, consider, starring, github, network, 中文版, glossary, reliability, frontend, contribute, cookie, privacy, land, acknowledgement, legal, newsletter, main, site, auditable, trusted, trillion, etc, yihua, cheng, started, observation, should, managed, treated, disposable, traces, especially, create, prefixes, conversations, opportunities, host, workers, behaviors, offload, look, closer, agent, than, fixed, prompt, kuntai, caching, starts, concrete, instructions, documents, reused, steps, recomputed, prefix, adds, latency, reuse, usable, run, compatible, api, tests, behave, sessions, enough, pressure, doing, thing, pin, siang, tan, contributor, depends, runtime, scheduling, openly, discipline, neoclouds, yeh, center, value, creation, turned, black, box, measured, just, but, runs, marketed, sky, computing, cao, shiyi, practical, evaluation, bottlenecks, tradeoffs, scalable, moreover, increasingly, reliable, close, automatic, optimization, further, talal, kaissi, scales, globally, single, region, define, toward, rational, investments, sovereign, healthier, competition, ultimately, worldwide, founding, xander, dunn, scientists, turn, scientific, discoveries, care, deeply, spec, sheets, thousands, say, smart, giri, affordable, goal, confident, happy, back, ruben, bryon, founded, engineers, frictionless, gatekeeping, builders, fluff, baseten, ker, ship, using, million, objective, tracks, gb200, h100, rubin, tpu, trainium, helpful, allowing, wider, landscape, creating, taxonomy, around, julien, launay, ourselves, key, datapoints, large, impacts, market, revenue, signal, missing, developer, author, machine, learning, book, stas, bekman, someone, who, written, extensively, actual, trackable, science, datologyai, matthew, leavitt, even, dylan, patel, nice, little, guy, feelings, tim, rosenfield, only, gigawatts, renewable, powered, asia, pacific, australia, jack, min, ong, pushing, post, training, research, complements, focus, accelerates, svp, nathan, goulding, committed, freedom, gain, showcasing, empower, charles, liang, launch, chase, lochmiller, great, partner, empowering, why, which, cuts, hype, unique, brium, quentin, colombet, highlight, rapid, pace, exciting, witness, birth, tied, here, affects, specific, cases, looking, showcase, continue, week, hyperscale, cuda, ian, buck, cares, welcome, underscore, advantage, nvlink, networking, consistently, roi, anush, elangovan, moat, match, improvement, fantastic, performing, interactivity, levels, jensen, huang, growing, exponentially, reasoning, invented, thinking, meeting, constant, enable, frequently, trt, unmatched, powering, productive, chair, lisa, collaboration, inspire, competitive, instinct, mi325x, mi355x, underscoring, strength, commitment, house, advisor, sriram, krishnan, now, commonly, hearing, testament, dylan522p, core, principal, red, hat, michael, goin, good, sir, assistant, professor, san, diego, fastvideo, hao, zhang, introduced, thesis, split, own, terms, eighteen, months, later, disaggregation, default, comparing, aggregated, pareto, curve, shows, separation, pays, off, ttft, tpot, stephen, balaban, individual, largest, aligns, informed, choices, grounded, clement, delangue, extends, chip, pulling, hub, them, accelerator, transparently, stanford, guo, working, techniques, yet, incredibly, hard, gaps, val, bercovici, ensembles, precisely, xpu, tpus, lpus, roadmap, include, stress, cpu, nvme, wall, feeds, invest, mingyi, behind, xai, grok, earning, recognition, king, firsthand, varies, configurations, capturing, variance, does, reproducibly, darrick, horton, thrives, strong, alternatives, reinforces, vision, synthetic, highlighting, business, roman, chernin, sets, standard, reveal, measuring, beyond, claims, provider, ensure, aligned, salanki, abstract, insight, space, matt, demonstrates, operate, practice, tensorrt, innovations, frameworks, translate, measurable, range, offers, tracking, mode, mark, saroufim, arguably, oss, suite, tsinghua, university, kaichao, pogchamp, chat, public, collaborate, scenarios, reference, fair, drive, architectures, flash, attention, tri, dao, peak, often, determined, low, valuable, showing, fp4, mtp, speculative, wide, play, various, jay, jackson, flexibility, skus, available, align, super, intelligence, joseph, spisak, belief, embodies, philosophy, serve, billions, baseline, making, corporate, strategic, planning, dighe, economics, requires, generating, signals, guide, strategy, silicon, unlock, scott, guthrie, performant, ability, tune, industrial, stargate, hoeschele, unprecedented, noise, living, picture, datacenters, zhipu, horizon, autonomous, execution, everything, multi, calling, pipelines, enormously, putting, hands, possible, questions, qwen3, independent, dedicated, supporting, advancing, series, tracked, encourages, keep, optimizing, devrel, ryan, lee, both, capability, deployment, mind, collaborative, spirit, part, supported, buyers, prominent, members, articles, datasets, dashboard, home,

Text of the page (random words):
e latest hardware and software with this transparency customers can confidently select the platforms that best align with their ai strategies jay jackson vice president oracle cloud infrastructure the gap between theoretical peak and real world inference throughput is often determined by systems software inference engine distributed strategies and low level kernels inferencemax is valuable because it benchmarks the latest software showing how optimizations like fp4 mtp speculative decode and wide ep actually play out across various hardware open reproducible results like these help the whole community move faster tri dao chief scientist of together ai inventor of flash attention the industry needs many public reproducible benchmarks of inference performance we re excited to collaborate with inferencemax from the vllm team more diverse workloads and scenarios that everyone can trust and reference will help the ecosystem move forward fair transparent measurements drive progress across every layer of the stack from model architectures to inference engines to hardware simon mo vllm project co lead inferencemax benchmark is pogchamp w in chat kaichao you vllm project co lead phd student tsinghua university arguably the most important oss benchmark suite out today inferencex mark saroufim gpu mode founder meta pytorch engineer inferencemax demonstrates how an open ecosystem can operate in practice many leading inference stacks such as vllm sglang and tensorrt llm are built on pytorch and benchmarks like this show how innovations across kernels runtimes and frameworks translate into measurable performance on a range of hardware platforms including nvidia and amd gpus by being open source and running nightly inferencemax offers a transparent community driven approach to tracking progress and providing pytorch users with data driven insights matt white executive director pytorch foundation inferencemax raises the bar by delivering open transparent benchmarks that track how inference really performs across the latest gpus and software stacks for customers having reproducible data that measures real world tokens per dollar tokens per watt turns abstract marketing numbers into actionable insight at coreweave we support this effort because it brings clarity to a fast moving space and helps the entire ecosystem build with confidence peter salanki cto coreweave inferencemax sets a new standard by providing open transparent benchmarks that reveal how inference performs across today s leading gpus and software stacks with reproducible data measuring real world tokens per dollar and tokens per watt customers can move beyond marketing claims to actionable insights for us at nebius as a full stack ai cloud provider this initiative helps us build our inference platform with confidence and ensure we are aligned with the ecosystem roman chernin co founder chief business officer nebius at tensorwave we re building a next generation cloud on amd gpus because we believe innovation thrives when customers have strong alternatives inferencemax reinforces that vision by providing open source reproducible benchmarks that track throughput efficiency and cost across the latest hardware and software by cutting through synthetic numbers and highlighting real world inference performance it helps customers see the full potential of amd platforms for ai at scale darrick horton ceo tensorwave sglang is the inference engine behind many production inference factories such as xai s grok earning its recognition as the inference king at scale we see firsthand how much performance varies across hardware models and configurations inferencex benchmarks sglang across every major gpu platform nightly capturing that variance in a way no other benchmark does continuously reproducibly mingyi lu sglang product lead inferencex ensembles precisely that open reproducible benchmarks that are continuously updated as xpu accelerators gpus tpus lpus memory storage and software stacks evolve i m excited to see the inferencex benchmarking roadmap include agentic coding workloads that stress cpu kv cache offloading soon nvme kv cache offloading from xpus as weka helps scale the memory wall by building the kv cache infrastructure that feeds these xpus having this level of visibility into inference performance helps the entire ecosystem make smarter decisions about where to invest val bercovici chief ai officer weka for researchers working on inference optimizations understanding how new techniques interact across the software and hardware stack is critical yet incredibly hard to measure inferencex provides much needed insights into how inference performance evolves across major hardware platforms moving the field forward with open reproducible data that makes the gaps and progress visible simon guo phd student stanford cs hugging face exists to make ai open and accessible to everyone inferencex extends that mission to ai chip performance pulling models directly from the hub and benchmarking them across every major accelerator continuously and transparently when the community can see exactly how frontier open models perform on real hardware in real time it raises the bar for the entire ecosystem clement delangue ceo hugging face lambda exists to make gpu compute simple and accessible for ai teams from individual researchers to the largest labs inferencex aligns with that mission by giving the community open reproducible benchmarks that measure what actually matters real world throughput cost efficiency and performance per watt across the latest hardware and software stacks teams can make informed compute choices grounded in transparent continuously updated data stephen balaban co founder and ceo lambda when we introduced distserve the thesis was simple split prefill and decode and optimize each on its own terms eighteen months later disaggregation is the default architecture across the industry inferencex is the benchmark that comparing disaggregated and aggregated serving across the whole pareto curve inferencex shows exactly when and where p d separation pays off in ttft tpot throughput and cost hao zhang assistant professor uc san diego co creator of distserve vllm and fastvideo the benchmark is good sir michael goin vllm core maintainer senior principal engineer at red hat now commonly hearing we want the semianalysis for x testament to what dylan522p has built sriram krishnan white house senior ai advisor open collaboration is driving the next era of ai innovation the open source inferencemax benchmark gives the community transparent nightly results that inspire trust and accelerate progress it highlights the competitive tco performance of our amd instinct mi300 mi325x and mi355x gpus across diverse workloads underscoring the strength of our platform and our commitment to giving developers real time visibility into our software progress dr lisa su chair and ceo amd inference demand is growing exponentially driven by long context reasoning nvidia grace blackwell nvl72 was invented for this new era of thinking ai nvidia is meeting that demand through constant hardware and software innovation to enable what s next in ai by benchmarking frequently inferencemax gives the industry a transparent view of llm inference performance on real world workloads the results are clear grace blackwell nvl72 with trt llm and dynamo delivers unmatched performance per dollar and per megawatt powering the most productive and cost effective ai factories in the world jensen huang founder ceo nvidia speed is the moat inferencemax s nightly benchmarks match the speed of improvement of the amd software stack it s fantastic to see amd s mi300 mi325 and mi355 gpus performing so well across diverse workloads and interactivity levels anush elangovan vp gpu software amd inferencemax highlights workloads that the ml community cares about at nvidia we welcome these comparisons because they underscore the advantage of our full stack approach from gpus hardware to nvlink networking to nvl72 rack scale to dynamo disaggregated serving that consistently delivers industry leading inference performance and roi at scale ian buck vp gm hyperscale nvidia inventor of cuda inferencemax s nightly results highlight the rapid pace of progress in the amd software stack it s exciting to witness the birth of an open project that provides a tied feedback loop between what the software team works on here at amd and how it affects specific ml use cases across our mi300 mi325 and mi355 gpus i m looking forward to see what s next for inferencemax and to showcase what the amd platform can do amd gpus will continue to get faster every week quentin colombet senior director amd ex brium ceo at crusoe we believe being a great partner means empowering our customers with choice and clarity that s why we re proud to support inferencemax which provides the entire ai community with open source reproducible benchmarks for the latest hardware by delivering transparent real world data on throughput efficiency and cost inferencemax cuts through the hype and helps our customers confidently select the very best platform for their unique workloads chase lochmiller co founder ceo crusoe supermicro is excited about the launch of inferencemax the semianalysis benchmarking system that measures real world throughput performance per dollar and energy efficiency this open source tool provides reproducible benchmarks running on the latest hardware and software enabling ai labs and enterprises to choose the best platforms at scale charles liang founder ceo supermicro vultr is committed to providing an open ecosystem that gives developers freedom in how they build and scale ai whether on nvidia or amd gpus with inferencemax customers gain open reproducible benchmarks that deliver clear insights into throughput efficiency and cost across cutting edge hardware and software by showcasing real world performance we empower teams to confidently choose the right platform for their ai workloads nathan goulding svp of engineering vultr at prime intellect we re pushing the frontier of ai post training and open research inferencex complements that work by providing open reproducible benchmarks that track real world inference performance across hardware and software stacks as they evolve for researchers like us having transparent continuously updated data on throughput and efficiency means we can focus on building better models instead of second guessing infrastructure this is the kind of community driven effort that accelerates progress for everyone jack min ong researcher prime intellect at firmus we re building the most energy efficient ai factories in the world and efficiency only matters if you can measure it inferencex gives the industry open reproducible benchmarks that track real world throughput cost and performance per watt across the latest gpu platforms and software stacks as we scale gigawatts of renewable powered ai infrastructure across asia pacific australia this kind of transparent continuously updated data helps the entire ecosystem understand what these systems actually deliver tim rosenfield co founder co ceo firmus inferencemax has been useful for us even if dylan patel is a nice little guy with feelings matthew leavitt chief science officer datologyai inferencex provides the open source measurements the community needs nightly results across real workloads real hardware and real software stacks as someone who has written extensively about the gap between theoretical and actual system performance i m glad to see a project that makes that gap visible and trackable for everyone sb stas bekman developer author of machine learning engineering open book 17 5k we use inferencex benchmarks ourselves as one of the key datapoints to help us make infrastructure decisions at adaptive ml inference performance is critical for large scale rl workloads where fast generation directly impacts time to market revenue for our customers inferencex benchmarks the full stack continuously engine model software and hardware across rack scale systems like gb300 nvl72 this is the kind of open transparent reproducible signal the ecosystem has been missing julien launay co founder ceo adaptive ml our customers ship ai to production using frontier open source models and at scale every token per second and every dollar per million tokens matters inferencex gives the ecosystem something we ve always needed an objective open benchmark that tracks real inference performance continuously across hardware such as gb300 nvl72 gb200 nvl72 h100 soon rubin tpu trainium very helpful in allowing the wider community to understand the landscape and creating a clear taxonomy around performance alex ker engineer baseten we founded verda to give ai engineers frictionless access to cutting edge compute without gatekeeping inferencex supports this mission by giving ai builders open reproducible benchmarks that show what gpus actually deliver under real inference workloads we want our customers to see transparent continuously updated performance data without marketing fluff inferencex provides exactly that ruben bryon founder ceo verda voltage park is built to give ai teams fast affordable access to gpu compute at scale inferencex supports that goal by providing open reproducible benchmarks that show how inference actually performs across the latest hardware and software stacks with transparent continuously updated data on throughput efficiency and cost teams can make confident compute decisions instead of guessing we re happy to back an effort that brings this level of clarity to the ecosystem saurabh giri cto voltage park at periodic labs we re building ai scientists that turn compute into real world scientific discoveries that means we care deeply about what each gpu actually delivers inferencex provides open reproducible benchmarks that cut through spec sheets and show real world throughput efficiency and cost across the latest hardware and software stacks having done inference across thousands of gpus i can say this kind of transparent continuously updated data is exactly what practitioners need to make smart infrastructure decisions xander dunn founding team periodic labs as ai infrastructure scales globally no single vendor or region can define the benchmarks that matter for everyone inferencex is an important step toward a shared transparent view of inference performance and tco enabling more rational investments for sovereign ai cloud operators as well as healthier competition and ultimately more accessible ai capacity worldwide talal m al kaissi ceo it is important to have an open and continuously updated platform for benchmarking inference engines across real workloads and diverse hardware inferencex provides this kind of transparent and practical evaluation helping the community better understand real system bottlenecks and tradeoffs benchmarks like this are essential for building m...