Meta tags:
description= Deliver events to New Relic;
keywords= new relic,newrelic,component,sink,logs,metrics;
Headings (most frequently used words):
request, adaptive_concurrency, buffer, proxy, batch, encoding, concurrency, health, checks, enabled, acknowledgements, healthcheck, metrics, rate, limits, adaptive, max_events, timeout_secs, deprecated, new, relic, configuration, input, types, telemetry, how, it, works, vector, site, footer, example, configurations, account_id, api, compression, inputs, license_key, region, logs, buffers, and, batches, retry, policy, state, about, components, setup, community, download, max_bytes, max_size, type, when_full, except_fields, only_fields, timestamp_format, http, https, no_proxy, rate_limit_duration_secs, rate_limit_num, retry_attempts, retry_initial_backoff_secs, retry_jitter_mode, retry_max_duration_secs, buffer_byte_size, buffer_discarded_events_total, buffer_events, buffer_received_bytes_total, buffer_received_events_total, buffer_sent_bytes_total, buffer_sent_events_total, buffer_size_bytes, buffer_size_events, component_discarded_events_total, component_errors_total, component_received_event_bytes_total, component_received_events_count, component_received_events_total, component_sent_bytes_total, component_sent_event_bytes_total, component_sent_events_total, utilization, require, disable, arc, static, decrease_ratio, ewma_alpha, initial_concurrency, max_concurrency_limit, rtt_deviation_scale,
Text of the page (most frequently used words):
the (354), vector (114), optional (100), component (73), this (45), #buffer (41), #request (34), events (34), data (30), type (30), from (30), and (29), concurrency (28), default (25), for (24), which (24), that (23), are (23), string (23), metrics (23), number (21), host (20), stage (20), logs (20), hostname (19), process (18), instance (18), pid (18), system (18), running (18), component_type (18), kind (18), component_kind (18), component_id (18), originated (18), can (17), literal (17), enum (17), configuration (15), you (15), sinks (14), sink (14), when (14), xxxx (14), new (13), counter (13), aws (13), file (13), uint (13), timestamp (13), api (13), options (12), adaptive (12), bytes (12), within (12), compression (12), requests (11), not (11), values (11), limit (11), batch (11), event (11), pipeline (11), proxy (11), source (11), will (10), http (10), option (10), used (10), uri (10), transform (10), maximum (10), acknowledgements (10), region (9), mode (9), numbered (9), unique (9), identifier (9), buffer_id (9), relic (9), datadog (8), behavior (8), retry (8), health (8), value (8), description (8), object (8), end (8), set (7), name (7), than (7), time (7), use (7), rtt (7), examples (7), license_key (7), inputs (7), account_id (7), with (6), required (6), service (6), like (6), seconds (6), gauge (6), internal (6), metric (6), log (6), before (6), valid (6), adaptive_concurrency (6), more (6), example (6), represent (6), unix (6), gzip (6), model (6), new_relic (6), my_sink_id (6), gcp (6), all (5), github (5), input (5), backoff (5), limits (5), checks (5), error (5), sent (5), endpoint (5), has (5), full (5), reference (5), matches (5), enabled (5), memory (5), disk (5), prometheus (5), chat (4), components (4), failed (4), rate (4), rate_limit_num (4), rate_limit_duration_secs (4), static (4), performance (4), based (4), healthcheck (4), require (4), check (4), size (4), batches (4), concepts (4), specific (4), how (4), output (4), were (4), origins (4), level (4), true (4), currently (4), deprecated (4), supported (4), types (4), retries (4), jitter (4), current (4), allowed (4), any (4), average (4), measurements (4), range (4), cause (4), float (4), less (4), com (4), list (4), proxying (4), traffic (4), encoding (4), docs (4), architecture (4), rss (3), releases (3), download (3), sources (3), opentelemetry (3), nats (3), updates (3), state (3), other (3), these (3), fixed (3), also (3), feature (3), systems (3), downstream (3), blog (3), false (3), buffers (3), configured (3), max_events (3), timeout_secs (3), flushed (3), them (3), global (3), total (3), configurations (3), may (3), sanitized (3), pod (3), pod_name (3), pathname (3), peer_path (3), peer_addr (3), connection (3), container (3), container_name (3), either (3), note (3), due (3), histogram (3), intentionally (3), filter (3), dropped (3), been (3), specified (3), amount (3), wait (3), none (3), your (3), delay (3), after (3), window (3), past (3), deviation (3), increases (3), scale (3), greater (3), hosts (3), must (3), support (3), bool (3), fields (3), lost (3), buffering (3), websocket (3), splunk (3), hec (3), azure (3), kinesis (3), going (2), administration (2), deployment (2), installation (2), setup (2), transforms (2), about (2), stateless (2), its (2), across (2), each (2), status (2), 500 (2), responses (2), control (2), retry_attempts (2), policy (2), via (2), integer (2), highly (2), enabling (2), such (2), does (2), algorithms (2), arc (2), disable (2), yaml (2), healthy (2), upon (2), failure (2), proceed (2), meets (2), exceeds (2), max_bytes (2), age (2), conditions (2), above (2), treats (2), instead (2), delivery (2), guarantees (2), ratio (2), would (2), indicate (2), idle (2), waiting (2), every (2), emitted (2), protocol (2), absolute (2), path (2), destination (2), only (2), accepted (2), tagged (2), cumulatively (2), batching (2), useful (2), issues (2), topology (2), errors (2), discarded (2), intentional (2), buffer_size_events (2), buffer_size_bytes (2), received (2), favor (2), telemetry (2), following (2), recommends (2), below (2), pile (2), being (2), strategy (2), reduce (2), first (2), fibonacci (2), sequence (2), 223372036854776e (2), one (2), outbound (2), ignore (2), larger (2), algorithm (2), takes (2), initial (2), smaller (2), high (2), down (2), parameters (2), typically (2), meta (2), settings (2), addresses (2), 127 (2), foo (2), bar (2), 3128 (2), https (2), proxied (2), should (2), see (2), whether (2), encoded (2), zlib (2), snappy (2), block (2), acceptance (2), consumption (2), free (2), space (2), performant (2), but (2), durable (2), restarted (2), forcefully (2), crashes (2), buffered (2), synchronized (2), least (2), information (2), connected (2), controls (2), handled (2), open (2), home (2), server (2), statsd (2), socket (2), sematext (2), redis (2), pulsar (2), remote (2), write (2), mqtt (2), formerly (2), kafka (2), influxdb (2), humio (2), greptimedb (2), pubsub (2), stackdriver (2), cloud (2), storage (2), monitoring (2), sqs (2), firehose (2), cloudwatch (2), amqp (2), route (2), remap (2), windows (2), kubernetes (2), docker (2), agent (2), linux (2), observability (2), pipelines (2), guides (2), sidebar, 2026, inc, rights, reserved, community, prod, cookies, privacy, team, contact, site, footer, next, previous, thank, joining, our, newsletter, sign, receive, emails, latest, content, page, meaning, consistent, 408, 429, 501, retried, attempts, retry_backoff_secs, apply, both, addition, limiting, overall, throughput, manually, specifying, recommend, improves, reliability, communicates, have, made, further, away, automatically, optimizes, underlying, mechanism, feedback, loop, inspired, tcp, congestion, checkout, announcement, post, config, etc, exit, immediately, pass, flag, ensure, accessible, ready, accept, performed, initialization, fails, logged, start, controlled, met, shown, diagram, notice, differently, treating, isolates, ensuring, services, disruptions, contained, honored, works, load, completely, simply, never, updated, utilization, component_sent_events_total, component_sent_event_bytes_total, some, send, excluding, query, raw, component_sent_bytes_total, component_received_events_total, separate, mostly, low, debugging, small, passed, component_received_events_count, component_received_event_bytes_total, occurred, error_type, encountered, component_errors_total, component_discarded_events_total, buffer_sent_events_total, buffer_sent_bytes_total, buffer_received_events_total, buffer_received_bytes_total, buffer_events, non, blocking, buffer_discarded_events_total, buffer_byte_size, link, summary, distribution, table, lists, possible, aware, available, differ, codec, lower, timeout, could, create, orphaned, result, duplicate, take, aborted, between, retry_max_duration_secs, incorporating, into, greatly, likelihood, creating, accidental, denial, dos, against, own, many, clients, recovering, random, anywhere, calculated, retry_jitter_mode, select, future, backoffs, attempting, retry_initial_backoff_secs, make, outstanding, given, managed, positive, denotes, calculating, secondary, computed, indicates, variable, those, comparing, expected, factor, appropriate, equal, reasonable, deviations, considered, anomalous, rtt_deviation_scale, 200, bound, put, place, safeguard, max_concurrency_limit, setting, seeing, long, ramp, restart, find, looking, adaptive_concurrency_limit, initial_concurrency, uses, exponentially, weighted, moving, ewma, compare, adjust, slowly, unusually, response, variability, weighting, compared, older, ewma_alpha, rounded, applying, back, rapidly, latency, fraction, decreasing, decrease_ratio, changes, incorrect, lead, stable, unstable, caution, follows, various, timeouts, middleware, splat, 192, 168, blocks, cidr, subdomains, wildcard, domains, domain, names, match, pattern, multiple, patterns, avoid, no_proxy, enables, similar, common, convention, different, proxies, configure, through, making, external, new_relic_license_key, license, key, prefix, array, info, wildcards, upstream, ids, starts, microseconds, unix_us, nanoseconds, unix_ns, milliseconds, unix_ms, floating, point, unix_float, rfc, 3339, rfc3339, format, timestamp_format, included, only_fields, excluded, except_fields, transformations, prepare, serialization, zstandard, zstd, unless, otherwise, highest, priority, preferable, temporarily, lose, rather, slowdown, drops, drop_newest, applies, backpressure, signalling, slow, means, while, edge, handling, when_full, 500ms, then, 256, megabytes, 268435488, allocated, hold, max_size, relevant, individual, found, section, configures, 100, uncompressed, batched, they, serialized, compressed, processed, disabling, precedence, over, supports, waits, acknowledged, acknowledging, acknowledgement, new_relic_account_key, account, json, toml, advanced, minimal, view, egress, yes, once, beta, deliver, versioning, security, glossary, environment, variables, cli, secrets, template, syntax, schema, unit, tests, tls, webhdfs, exporter, postgres, papertrail, mezmo, logdna, loki, keep, honeycomb, chronicle, unstructured, elasticsearch, doris, traces, databricks, zerobus, databend, console, clickhouse, blackhole, monitor, ingestion, blob, axiom, sns, streams, appsignal, trace, throttle, tag, cardinality, sample, lua, incremental, exclusive, dedupe, ec2, metadata, aggregate, vrl, syslog, stdin, scrape, pushgateway, postgresql, okta, nginx, mongodb, logstash, journald, client, heroku, logplex, fluent, descriptor, exec, eventstoredb, dnstap, demo, ecs, apache, expressions, functions, language, pgo, optimization, validating, management, runtime, sizing, capacity, planning, rollout, availability, hardening, architecting, unified, aggregator, architectures, production, topologies, roles, installer, archives, manual, ubuntu, rhel, raspbian, nixos, macos, debian, centos, arch, amazon, operating, platforms, yum, rpm, pacman, nix, msi, homebrew, helm, dpkg, apt, package, managers, quickstart, introduction, navbar, dropdown, menu, search, toggle, dark, documentation,
Text of the page (random words):
ge limit if you re seeing that it takes a long time to ramp up adaptive concurrency after a restart you can find this value by looking at the adaptive_concurrency_limit metric default 1 request adaptive_concurrency max_concurrency_limit optional uint the maximum concurrency limit the adaptive request concurrency limit does not go above this bound this is put in place as a safeguard default 200 request adaptive_concurrency rtt_deviation_scale optional float scale of rtt deviations which are not considered anomalous valid values are greater than or equal to 0 and reasonable values range from 1 0 to 3 0 when calculating the past rtt average a secondary deviation value is also computed that indicates how variable those values are that deviation is used when comparing the past rtt average to the current measurements so we can ignore increases in rtt that are within an expected range this factor is used to scale up the deviation to an appropriate range larger values cause the algorithm to ignore larger increases in the rtt default 2 5 request concurrency optional string literal enum uint configuration for outbound request concurrency this can be set either to one of the below enum values or to a positive integer which denotes a fixed concurrency limit enum options option description adaptive concurrency is managed by vector s adaptive request concurrency feature none a fixed concurrency of 1 only one request can be outstanding at any given time default adaptive request rate_limit_duration_secs optional uint the time window used for the rate_limit_num option default 1 seconds request rate_limit_num optional uint the maximum number of requests allowed within the rate_limit_duration_secs time window default 9 223372036854776e 18 requests request retry_attempts optional uint the maximum number of retries to make for failed requests default 9 223372036854776e 18 retries request retry_initial_backoff_secs optional uint the amount of time to wait before attempting the first retry for a failed request after the first retry has failed the fibonacci sequence is used to select future backoffs default 1 seconds request retry_jitter_mode optional string literal enum the jitter mode to use for retry backoff behavior enum options option description full full jitter the random delay is anywhere from 0 up to the maximum current delay calculated by the backoff strategy incorporating full jitter into your backoff strategy can greatly reduce the likelihood of creating accidental denial of service dos conditions against your own systems when many clients are recovering from a failure state none no jitter default full request retry_max_duration_secs optional uint the maximum amount of time to wait between retries default 30 seconds request timeout_secs optional uint the time a request can take before being aborted datadog highly recommends that you do not lower this value below the service s internal timeout as this could create orphaned requests pile on retries and result in duplicate data downstream default 60 seconds input types the following table lists all telemetry data types supported by the component across possible configurations be aware that the available data types may differ based on the specified codec configuration logs log events are supported metrics the following metrics are supported counter distribution gauge histogram set summary telemetry metrics link buffer_byte_size gauge the number of bytes currently in the buffer deprecated this metric has been deprecated in favor of buffer_size_bytes buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_discarded_events_total counter the number of events dropped by this non blocking buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on intentional true if the events were discarded intentionally like a filter transform or false if due to an error pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_events gauge the number of events currently in the buffer deprecated this metric has been deprecated in favor of buffer_size_events buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_received_bytes_total counter the number of bytes received by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_received_events_total counter the number of events received by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_sent_bytes_total counter the number of bytes sent by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_sent_events_total counter the number of events sent by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_size_bytes gauge the number of bytes currently in the buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_size_events gauge the number of events currently in the buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline component_discarded_events_total counter the number of events dropped by this component component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on intentional true if the events were discarded intentionally like a filter transform or false if due to an error pid optional the process id of the vector instance component_errors_total counter the total number of errors encountered by this component component_id the vector component id component_kind the vector component kind component_type the vector component type error_type the type of the error host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the stage within the component at which the error occurred component_received_event_bytes_total counter the number of event bytes accepted by this component either from tagged origins like file and uri or cumulatively from other origins component_id the vector component id component_kind the vector component kind component_type the vector component type container_name optional the name of the container from which the data originated file optional the file from which the data originated host optional the hostname of the system vector is running on mode optional the connection mode used by the component peer_addr optional the ip from which the data originated peer_path optional the pathname from which the data originated pid optional the process id of the vector instance pod_name optional the name of the pod from which the data originated uri optional the sanitized uri from which the data originated component_received_events_count histogram a histogram of the number of events passed in each internal batch in vector s internal topology note that this is separate than sink level batching it is mostly useful for low level debugging performance issues in vector due to small internal batches component_id the vector component id component_kind the vector component kind component_type the vector component type container_name optional the name of the container from which the data originated file optional the file from which the data originated host optional the hostname of the system vector is running on mode optional the connection mode used by the component peer_addr optional the ip from which the data originated peer_path optional the pathname from which the data originated pid optional the process id of the vector instance pod_name optional the name of the pod from which the data originated uri optional the sanitized uri from which the data originated component_received_events_total counter the number of events accepted by this component either from tagged origins like file and uri or cumulatively from other origins component_id the vector component id component_kind the vector component kind component_type the vector component type container_name optional the name of the container from which the data originated file optional the file from which the data originated host optional the hostname of the system vector is running on mode optional the connection mode used by the component peer_addr optional the ip from which the data originated peer_path optional the pathname from which the data originated pid optional the process id of the vector instance pod_name optional the name of the pod from which the data originated uri optional the sanitized uri from which the data originated component_sent_bytes_total counter the number of raw bytes sent by this component to destination sinks component_id the vector component id component_kind the vector component kind component_type the vector component type endpoint optional the endpoint to which the bytes were sent for http this will be the host and path only excluding the query string file optional the absolute path of the destination file host optional the hostname of the system vector is running on pid optional the process id of the vector instance protocol the protocol used to send the bytes region optional the aws region name to which the bytes were sent in some configurations this may be a literal hostname component_sent_event_bytes_total counter the total number of event bytes emitted by this component component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on output optional the specific output of the component pid optional the process id of the vector instance component_sent_events_total counter the total number of events emitted by this component component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on output optional the specific output of the component pid optional the process id of the vector instance utilization gauge a ratio from 0 to 1 of the load on a component a value of 0 would indicate a completely idle component that is simply waiting for input a value of 1 would indicate a that is never idle this value is updated every 5 seconds component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance how it works buffers and batches this component buffers batches data as shown in the diagram above you ll notice that vector treats these concepts differently instead of treating them as global concepts vector treats them as sink specific concepts this isolates sinks ensuring services disruptions are contained and delivery guarantees are honored batches are flushed when 1 of 2 conditions are met the batch age meets or exceeds the configured timeout_secs the batch size meets or exceeds the configured max_bytes or max_events buffers are controlled via the buffer options health checks health checks ensure that the downstream service is accessible and ready to accept data this check is performed upon sink initialization if the health check fails an error will be logged and vector will proceed to start require health checks if you d like to exit immediately upon a health check failure you can pass the require healthy flag vector config etc vector vector yaml require healthy disable health checks if you d like to disable health checks for this sink you can set the healthcheck option to false rate limits adaptive concurrency adaptive request concurrency arc adaptive request concurrency is a feature of vector that does away with static concurrency limits and automatically optimizes http concurrency based on downstream service responses the underlying mechanism is a feedback loop inspired by tcp congestion control algorithms checkout the announcement blog post we highly recommend enabling this feature as it improves performance and reliability of vector and the systems it communicates with as such we have made it the default and no further configuration is required static concurrency if adaptive request concurrency is not for you you can manually set static concurrency limits by specifying an integer for request concurrency sinks my sink request concurrency 10 rate limits in addition to limiting request concurrency you can also limit the overall request throughput via the request rate_limit_duration_secs and request rate_limit_num options sinks my sink request rate_limit_duration_secs 1 rate_limit_num 10 these will apply to both adaptive and fixed request concurrency values retry policy vector will retry failed requests status in ...
|
