Meta tags:
description=Improving reliability and performance across your entire observability infrastructure;
keywords=arc,adaptive request concurrency,networking,concurrency,http,rate limiting;

Headings (most frequently used words):

the, adaptive, request, concurrency, how, it, resilient, observability, at, scale, lead, up, problem, rate, limiting, is, not, panacea, solution, works, we, built, more, to, come, vector, site, footer, role, of, configuration, about, components, setup, community, download,

Text of the page (most frequently used words):
the (80), and (46), vector (44), that (29), you (24), for (23), #concurrency (22), this (19), with (14), rate (13), http (11), arc (10), but (10), your (10), request (10), how (9), more (9), are (9), which (9), can (9), performance (9), #adaptive (9), limits (9), our (8), services (8), downstream (7), has (7), service (7), observability (7), algorithm (6), some (6), decision (6), problem (6), data (6), rtt (6), limit (6), always (6), sink (6), like (6), elasticsearch (6), cluster (6), limiting (6), github (5), configuration (5), new (5), users (5), there (5), blog (5), current (5), response (5), based (5), chat (4), sinks (4), team (4), about (4), next (4), even (4), feedback (4), currently (4), also (4), inspired (4), feature (4), process (4), highly (4), own (4), see (4), have (4), here (4), these (4), scenarios (4), need (4), case (4), what (4), clickhouse (4), codes (4), throughput (4), time (4), high (4), scale (4), loop (4), not (4), support (4), rss (3), download (3), prod (3), components (3), updates (3), now (3), quite (3), its (3), their (3), open (3), example (3), netflix (3), work (3), lead (3), provides (3), was (3), approach (3), rust (3), down (3), solution (3), aimd (3), within (3), along (3), may (3), change (3), changes (3), below (3), determines (3), lowers (3), when (3), needs (3), set (3), buffer (3), basis (3), making (3), failure (3), too (3), two (3), provide (3), automatically (3), instances (3), optimal (3), system (3), static (3), does (3), them (3), one (3), pipelines (3), infrastructure (3), all (2), releases (2), going (2), first (2), kubernetes (2), initial (2), should (2), immediately (2), management (2), into (2), underlying (2), providing (2), knobs (2), internal (2), way (2), issue (2), week (2), behind (2), depth (2), bruce (2), visualizations (2), both (2), pretty (2), linear (2), exponential (2), engineering (2), tcp (2), algorithms (2), lot (2), back (2), read (2), congestion (2), control (2), additive (2), increase (2), multiplicative (2), decrease (2), qualitatively (2), better (2), customer (2), since (2), very (2), built (2), fine (2), parameters (2), most (2), cases (2), adjust (2), uses (2), exponentially (2), weighted (2), measurements (2), parameter (2), heavily (2), ewma_alpha (2), failures (2), happens (2), lower (2), thus (2), choose (2), disk (2), memory (2), configured (2), per (2), buffering (2), quietly (2), works (2), out (2), version (2), setting (2), consistently (2), sees (2), requests (2), increases (2), enabled (2), simple (2), continuously (2), information (2), decisions (2), outage (2), avoid (2), any (2), risk (2), while (2), full (2), advantage (2), capacity (2), across (2), than (2), sending (2), number (2), traffic (2), factors (2), raise (2), assess (2), overwhelming (2), reliability (2), could (2), help (2), because (2), over (2), answer (2), question (2), fix (2), user (2), mean (2), questions (2), resilient (2), guides (2), docs (2), 2026, datadog, inc, rights, reserved, community, administration, deployment, installation, setup, transforms, sources, cookies, privacy, contact, site, footer, test, class, integration, previous, thank, joining, newsletter, sign, receive, emails, latest, content, page, confident, state, improve, experience, rely, forward, continue, listening, incorporating, roadmap, fully, refining, future, release, serves, calls, exploration, alternative, gradient, lively, discussions, already, pointing, steps, follow, announcement, post, from, timber, engineer, far, look, implemented, intricate, story, tell, great, urge, tune, guenter, come, took, several, months, hefty, prs, handful, think, end, result, wholly, consistent, fastidious, strive, building, dead, ends, testing, mostly, relied, house, straightforward, customizable, server, written, http_test_server, largely, java, shop, nothing, usable, off, shelf, ecosystem, needed, create, implementation, module, special, importance, responsible, alluded, above, controller, adaptive_concurrency, engineers, ultimately, opted, deeply, analogous, beautifully, summarized, piece, respective, approaches, utilize, say, want, discussion, drove, repo, epic, forth, slew, 3255, under, load, foray, addressing, came, called, option, possible, inspirations, gestured, toward, rfc, 1858, pinpointed, pain, point, early, life, development, methodical, driven, summarize, defaults, just, know, call, targeted, general, shouldn, resort, three, available, minimal, necessary, respond, threshold, ignored, rtt_threshold_ratio, moving, average, ewma, past, reference, compare, compared, older, ones, rapidly, higher, latency, decrease_ratio, itself, throttle, output, sent, later, lets, between, default, maximizes, use, requires, stronger, durability, guarantees, max_size, room, tuning, find, yourself, needing, additional, log, table, 8123, host, log_stream_2, log_stream_1, inputs, type, clickhouse_internal, never, stops, background, box, zero, beyond, enabling, opt, enable, role, tree, active, knows, extreme, total, increasing, indicate, decreases, 503, unavailable, 429, many, declining, constant, successful, 200, 299, linearly, watches, significant, movements, things, round, trip, success, diagram, shows, chart, fairly, level, posts, via, keeps, track, make, precise, require, human, intervention, hums, speed, granularity, simply, cannot, dramatically, backpressure, events, experiences, temporary, destabilizing, still, taking, decreased, bandwidth, take, redistributes, deploy, plays, feel, strongly, path, given, network, environment, conditions, adjusts, light, don, forget, course, chasing, dragon, cognitive, operational, costs, especially, pronounced, elastic, environments, essentially, let, tweak, topologies, much, little, friction, compounds, designed, volume, only, perform, balancing, act, each, instance, application, points, elusive, target, shifts, low, capable, handling, least, waste, resources, compromise, vicious, constantly, outcomes, nice, fallback, blunt, instrument, half, dynamic, core, configuring, locks, perpetual, panacea, versions, prior, address, outbound, certainly, prevent, certain, worst, revealed, deep, limitations, crux, matter, presents, major, challenge, those, handle, event, payloads, quickly, send, interdependent, almost, letting, overwhelm, wide, degradation, cascading, found, were, second, complex, depends, variety, covered, detail, telling, would, irresponsible, then, causing, changing, dramatic, limited, processing, respect, particular, other, words, intentionally, reduced, stay, rate_limit_num, rate_limit_duration_secs, internally, typically, trace, tower_limit, exceeded, disabling, common, get, involves, logs, excited, announce, includes, away, optimizes, responses, mechanism, become, critical, technological, landscape, why, extremely, tiniest, resource, footprint, manage, huge, enough, real, world, pipeline, efficiency, being, good, citizen, playing, nicely, luc, perkins, lucperkins, min, november, 2020, improving, entire, navbar, dropdown, menu, search, toggle, dark, mode,

Text of the page (random words):
adaptive request concurrency resilient observability at scale vector docs guides components download blog support observability pipelines toggle dark mode search x 𝕏 github chat rss updates open navbar dropdown menu docs guides components download blog support observability pipelines x 𝕏 github chat rss updates vector blog adaptive request concurrency adaptive request concurrency resilient observability at scale improving reliability and performance across your entire observability infrastructure november 24 2020 8 min read luc perkins lucperkins observability pipelines have become critical infrastructure in the current technological landscape which is why we ve built vector to provide extremely high throughput with the tiniest resource footprint we can manage rust is a huge help here but this is not enough in the real world your observability pipeline needs to provide optimal performance and efficiency while also being a good infrastructure citizen and playing nicely with services like elasticsearch and clickhouse and so we re excited to announce that vector version 0 11 includes support for adaptive request concurrency arc in all of its http based sinks this feature does away with static rate limits and automatically optimizes http concurrency limits based on downstream service responses the underlying mechanism is a feedback loop inspired by tcp congestion control algorithms the lead up one of the most common support questions we get about vector involves logs like this trace tower_limit rate service rate limit exceeded disabling service users typically have two questions about this what does it mean how can i fix it the answer to the first question is simple vector has internally rate limited processing to respect user configured limits request rate_limit_duration_secs and request rate_limit_num for that particular sink in other words vector has intentionally reduced performance to stay within static limits the answer to the second question how to fix it is more complex because it depends on a variety of factors that change over time covered in more detail below telling the user to raise their limits would be irresponsible since we d then risk overwhelming the downstream service and causing an outage but not changing them could mean limiting performance in a dramatic way in one case we found that rate limits were limiting performance by over 80 the crux of the matter is that vector s high throughput presents a major challenge for http based sinks like elasticsearch because those services can t always handle event payloads as quickly as vector can send them and when data services are heavily interdependent which is almost always letting vector overwhelm one of them can lead to system wide performance degradation or even cascading failures in versions of vector prior to 0 11 you could address this problem by setting rate limits on outbound http traffic to downstream services rate limiting certainly does help prevent certain worst case scenarios but customer feedback and our own internal qa has revealed that this approach also has deep limitations the problem rate limiting is not a panacea rate limiting is nice to have as a fallback but it s a blunt instrument a static half solution to a dynamic problem the core problem is that configuring your own rate limits locks you into a perpetual loop within this vicious loop you need to constantly avoid two outcomes you set the limit too high and thus compromise system reliability by overwhelming your services time to lower the limit and re assess you set the limit too low and waste resources your elasticsearch cluster may be capable of handling more concurrency than you re providing at least for now time to raise the limit and re assess not only do you need to perform this balancing act on a per sink basis and on each vector instance that may be a lot of application points in your system but the optimal rate is an elusive target that shifts along with changes in a number of factors the number of vector instances currently sending traffic the current capacity of downstream services the volume of data you re currently sending these changes are especially pronounced in highly elastic environments like kubernetes that are essentially designed to let you tweak cluster topologies configuration and much more with very little friction which compounds the problem and don t forget of course that this chasing the dragon decision loop has its own cognitive and operational costs the solution adaptive request concurrency we feel strongly that vector s adaptive request concurrency arc feature provides a qualitatively better path than rate limiting with arc enabled on any given sink vector determines the optimal network concurrency based on current environment conditions and continuously re adjusts in light of new information here s how that plays out in some example scenarios change response you deploy more vector instances vector automatically redistributes http throughput across both current and new instances you scale up your elasticsearch cluster vector automatically increases concurrency to take full advantage of the new capacity you scale your elasticsearch cluster back down vector lowers concurrency to avoid any risk of destabilizing the cluster while still taking full of advantage of the now decreased bandwidth your elasticsearch cluster experiences a temporary outage vector lowers concurrency dramatically and provides backpressure by buffering events with arc these scenarios require no human intervention vector quietly hums along making these decisions for you with a speed and granularity that rate limits simply cannot provide how it works arc in vector is based on a decision making process that s fairly simple at a high level when vector posts data to downstream services via http it continuously keeps track of downstream service performance and uses that information to make precise concurrency decisions the diagram below shows vector s decision chart with arc enabled vector watches for significant movements in two things the round trip time rtt of requests and http response codes failure vs success if the rtt is declining constant and or response codes are consistently successful 200 299 vector sees and increases the throughput linearly this is the additive increase in aimd if the rtt is increasing and or response codes consistently indicate failure codes like 429 too many requests and 503 service unavailable vector sees and exponentially decreases concurrency this is the multiplicative decrease in aimd this decision tree is always active and vector always knows what to do even in extreme cases like total service failure the role of configuration vector never stops quietly making the linear up vs exponential down decision in the background and it works out of the box with zero configuration beyond enabling the feature which is currently on an opt in basis in version 0 11 you can enable arc in an http sink by setting the request concurrency parameter to adaptive here s an example for a clickhouse sink sinks clickhouse_internal type clickhouse inputs log_stream_1 log_stream_2 host http clickhouse prod 8123 table prod log data request concurrency adaptive there s also room for fine tuning if you find yourself needing additional knobs buffering what happens when vector needs to lower concurrency and thus throttle the output what happens to data that needs to be sent later vector lets you choose between an on disk and an in memory buffer and to set a max_size for that buffer the memory buffer is the default which maximizes performance but you can always choose disk if your use case requires stronger durability guarantees as always this can be configured on a per sink basis the adaptive concurrency algorithm itself in general you shouldn t need to adjust the algorithm but in case you need to resort to that there are three parameters available decrease_ratio this determines how rapidly vector lowers the limit in response to failures or higher latency ewma_alpha vector uses an exponentially weighted moving average ewma of past rtt measurements as a reference to compare with the current rtt the ewma_alpha parameter determines how heavily new measurements are weighted compared to older ones rtt_threshold_ratio the minimal change in rtt necessary for the algorithm to respond and adjust concurrency changes below that threshold are ignored the defaults should work just fine for these parameters in most cases but we know that some scenarios may call for a highly targeted approach how we built it the development process behind arc was highly methodical and data driven to summarize customer feedback has pinpointed concurrency management as a pain point since very early in the life of vector the initial foray in addressing the problem came in rfc 1858 which called for a qualitatively better option for users and gestured toward some possible inspirations our engineers ultimately opted for a solution deeply inspired by analogous work on the netflix engineering team which is beautifully summarized in the performance under load piece on their blog our respective approaches utilize an additive increase multiplicative decrease aimd algorithm inspired by tcp congestion control algorithms we ll have a lot more to say about this here on the blog next week if you want to see the in depth discussion that drove this process see github issue 3255 on the vector repo there you ll see a pretty epic back and forth within our engineering team along with a slew of visualizations it s quite the read as netflix is largely a java shop and there was nothing immediately usable off the shelf in the rust ecosystem we needed to create our own rust implementation which you can see in the adaptive_concurrency module of special importance is the concurrency controller which is responsible for the linear up exponential down decision that i alluded to above for testing the team mostly relied on our in house http_test_server a pretty straightforward but highly customizable http server written in go it took several months some hefty prs and even a handful of dead ends but we think that both the process and the end result are wholly consistent with the fastidious approach we strive for in building vector more to come next week we ll follow up on this announcement with a post from timber s bruce guenter the lead engineer behind arc that provides a far more in depth look at how this feature was implemented bruce has quite an intricate story to tell and some great visualizations so we urge you to tune in going forward we ll continue listening to vector users and incorporating their feedback on concurrency management into vector s roadmap we re fully open to refining the underlying algorithm and providing more configuration knobs in a future release if that serves our users there s currently an open issue for example that calls for exploration of an alternative gradient algorithm also inspired by netflix s work and some lively internal discussions are already pointing the way to next steps for now we re quite confident that arc in vector 0 11 even in its initial state should immediately improve the experience of users that rely on downstream http services on this page sign up to receive emails on the latest vector content and new releases thank you for joining our updates newsletter previous first class kubernetes integration for vector next how we test vector vector site footer about vector contact us the team privacy cookies components sources transforms sinks setup installation deployment configuration administration going to prod community github x chat download releases x 𝕏 github chat rss 2026 datadog inc all rights reserved