Meta tags:
description=At twenty past two on a Tuesday morning one customer began a historical backfill against our API.... Tagged with sergeyshinder, sre, reliability, ratelimiting.;
keywords=sergeyshinder, sre, reliability, ratelimiting, software, coding, development, engineering, inclusive, community;

Headings (most frequently used words):

the, our, rate, limit, punished, everyone, except, customer, causing, problem, dev, community, top, comments, more, from, sergey, shinder,

Text of the page (most frequently used words):
the (31), and (17), dev (12), that (8), for (8), our (7), share (6), #customer (6), was (6), with (5), sergey (5), shinder (5), you (5), service (5), who (5), rate (5), key (5), #community (4), about (4), sergeyshinder (4), were (4), from (4), limit (4), had (4), bucket (4), hundred (4), create (3), software (3), two (3), abuse (3), comments (3), what (3), not (3), aggregate (3), customers (3), number (3), its (3), requests (3), which (3), one (3), client (3), causing (3), account (2), log (2), where (2), 2026 (2), other (2), use (2), code (2), conduct (2), your (2), told (2), now (2), api (2), every (2), out (2), redis (2), anything (2), reliability (2), sre (2), more (2), sep (2), this (2), reporting (2), hide (2), comment (2), will (2), post (2), via (2), report (2), answer (2), whoever (2), because (2), never (2), them (2), day (2), own (2), thirty (2), kept (2), rather (2), than (2), call (2), say (2), per (2), can (2), second (2), percent (2), twelve (2), doing (2), nineteen (2), times (2), token (2), asking (2), proportion (2), punished (2), everyone (2), except (2), problem (2), copy (2), link (2), search (2), place, coders, stay, date, grow, their, careers, made, love, 2016, ruby, rails, built, powers, inclusive, communities, open, source, forem, terms, privacy, policy, mlh, shop, contact, showcase, organization, accounts, advertise, help, education, tracks, videos, reading, list, challenges, home, space, discuss, keep, development, manage, career, sms, provider, when, come, back, heard, http, integration, beta, became, version, new, installed, javascript, npm, releaseengineering, users, logged, weeks, before, refused, write, joined, follow, further, actions, may, consider, blocking, person, confirm, child, well, are, sure, want, become, hidden, but, still, visible, permalink, dismiss, preview, submit, templates, let, quickly, faqs, store, snippets, template, trusted, user, personal, subscribe, top, protects, says, nothing, gets, have, said, pushes, hardest, changed, too, publish, worst, tenant, success, next, average, over, once, small, having, completely, different, each, has, sized, trailing, peak, multiplier, global, behind, backstop, allocation, mechanism, carry, class, interactive, outranks, batch, same, responses, hit, record, throttled, fraction, shedding, something, see, happening, somebody, specific, steady, forty, availability, hour, came, ninety, reads, moderate, incident, unusual, closer, existed, nowhere, shared, drained, arrival, flight, takes, tokens, appears, goes, instant, exactly, limiter, did, job, perfectly, standing, distributed, shortage, direct, inverse, deserved, edge, holds, single, whole, thousand, chosen, years, ago, actually, serve, been, read, configuration, many, without, noticing, absence, dimension, decision, fairness, implementation, detail, twenty, past, tuesday, morning, began, historical, backfill, against, within, ten, minutes, being, rejected, started, getting, most, throughput, they, asked, ratelimiting, posted, mastodon, facebook, linkedin, copied, clipboard, pick, gem, boost, save, jump, fire, raised, hands, exploding, head, unicorn, like, add, reaction, close, powered, algolia, navigation, menu, skip, content,

Text of the page (random words):
our rate limit punished everyone except the customer causing the problem dev community skip to content navigation menu search powered by algolia search log in create account dev community close add reaction like unicorn exploding head raised hands fire jump to comments save boost pick as gem more copy link copy link copied to clipboard share to x share to linkedin share to facebook share to mastodon share post via report abuse sergey shinder posted on sep 20 our rate limit punished everyone except the customer causing the problem sergeyshinder sre reliability ratelimiting at twenty past two on a tuesday morning one customer began a historical backfill against our api within ten minutes a hundred and twelve other customers were being rejected and the customer who started it was getting most of the throughput they had asked for our edge holds a single token bucket for the whole service two thousand requests a second a number chosen years ago from what the service can actually serve it is not per key it never had been and i had read the configuration many times without noticing that the absence of a key dimension was a decision about fairness rather than an implementation detail a shared bucket is drained by arrival a client with nineteen hundred requests in flight takes tokens at nineteen hundred times the rate of a client with one because every token that appears goes to whoever is asking at that instant and asking is exactly what that client was doing the limiter did its job perfectly it kept the service standing and it distributed the shortage in direct proportion to who was causing it which is to say in inverse proportion to who deserved it that customer s steady rate is about forty a second our aggregate availability for the hour came out at ninety one percent which reads as a moderate incident for the hundred and twelve customers who were not doing anything unusual it was closer to thirty percent and that number existed nowhere each key now has its own bucket sized from its own trailing thirty day peak with a multiplier with the global bucket kept behind them as a backstop for the service rather than as the allocation mechanism requests carry a class so an interactive call outranks a batch call from the same key responses say which limit was hit and we record the throttled fraction per key so shedding is something we can see happening to somebody specific the reporting changed too we publish the worst tenant s success rate next to the aggregate because the aggregate is an average over customers and had never once told us that a small number of them were having a completely different day a limit protects the service it says nothing about who gets what and where you have not said it the answer is whoever pushes hardest sergey shinder top comments 0 subscribe personal trusted user create template templates let you quickly answer faqs or store snippets for re use submit preview dismiss code of conduct report abuse are you sure you want to hide this comment it will become hidden in your post but will still be visible via the comment s permalink hide child comments as well confirm for further actions you may consider blocking this person and or reporting abuse sergey shinder follow i am sergey shinder joined sep 2 2026 more from sergey shinder our users were logged out for two weeks before redis refused to write anything sergeyshinder sre redis reliability our beta became the version every new customer installed sergeyshinder releaseengineering npm javascript the sms provider told us when to come back and we heard now sergeyshinder api integration http dev community a space to discuss and keep up software development and manage your software career home dev challenges reading list dev videos dev education tracks dev help advertise on dev organization accounts dev showcase about contact dev shop mlh code of conduct privacy policy terms of use built on forem the open source software that powers dev and other inclusive communities made with love and ruby on rails dev community 2016 2026 we re a place where coders share stay up to date and grow their careers log in create account