Meta tags:
description=Stream observability data to Databricks Unity Catalog via Zerobus;
keywords=databricks,component,sink,logs;

Headings (most frequently used words):

request, proxy, adaptive_concurrency, buffer, auth, batch, stream_options, concurrency, acknowledgements, health, checks, enabled, healthcheck, rate, limits, adaptive, max_events, timeout_secs, deprecated, databricks, zerobus, requirements, configuration, input, types, telemetry, how, it, works, vector, site, footer, example, configurations, ingestion_endpoint, inputs, table_name, unity_catalog_endpoint, user_agent, logs, metrics, authentication, batching, buffers, and, batches, error, handling, retries, retry, policy, schema, state, about, components, setup, community, download, client_id, client_secret, strategy, max_bytes, max_size, type, when_full, http, https, no_proxy, rate_limit_duration_secs, rate_limit_num, retry_attempts, retry_initial_backoff_secs, retry_jitter_mode, retry_max_duration_secs, compression, flush_timeout_ms, server_lack_of_ack_timeout_ms, buffer_byte_size, buffer_discarded_events_total, buffer_events, buffer_received_bytes_total, buffer_received_events_total, buffer_sent_bytes_total, buffer_sent_events_total, buffer_size_bytes, buffer_size_events, component_discarded_events_total, component_errors_total, component_received_event_bytes_total, component_received_events_count, component_received_events_total, component_sent_bytes_total, component_sent_event_bytes_total, component_sent_events_total, utilization, require, disable, arc, static, decrease_ratio, ewma_alpha, initial_concurrency, max_concurrency_limit, rtt_deviation_scale,

Text of the page (most frequently used words):
the (403), #vector (121), optional (99), component (73), this (52), and (47), buffer (41), #request (36), for (35), default (34), from (33), data (32), zerobus (30), databricks (30), type (30), concurrency (29), events (28), that (27), are (25), which (25), proxy (24), https (24), logs (24), sink (22), batch (21), number (21), configuration (20), host (20), string (20), stage (20), com (20), system (19), hostname (19), can (18), process (18), instance (18), pid (18), running (18), component_type (18), kind (18), component_kind (18), component_id (18), originated (18), cloud (18), you (17), metrics (17), when (16), literal (16), acknowledgements (15), uint (15), sinks (14), not (14), service (14), http (13), aws (13), file (13), examples (13), enum (13), table (12), requests (12), options (12), adaptive (12), limit (12), used (12), source (12), end (12), counter (12), bytes (12), within (12), event (11), required (11), will (11), values (11), maximum (11), pipeline (11), value (10), retry (10), with (10), endpoint (10), uri (10), transform (10), identifier (10), datadog (9), behavior (9), enabled (9), client (9), before (9), mode (9), numbered (9), unique (9), buffer_id (9), object (9), schema (8), catalog (8), unity_catalog_endpoint (8), table_name (8), set (8), health (8), option (8), size (8), than (8), sent (8), name (8), dbc (8), main (8), use (8), error (7), unity (7), sdk (7), errors (7), see (7), example (7), oauth (7), a1b2c3d4 (7), e5f6 (7), time (7), rtt (7), inputs (7), 1234567890123456 (7), west (7), ingestion_endpoint (7), target (6), configured (6), matches (6), arrow (6), backoff (6), url (6), like (6), write (6), has (6), seconds (6), internal (6), metric (6), log (6), compression (6), description (6), full (6), adaptive_concurrency (6), more (6), model (6), databricks_zerobus (6), my_sink_id (6), gcp (6), github (5), new (5), must (5), may (5), using (5), limits (5), proxying (5), checks (5), stream (5), current (5), retries (5), batches (5), specific (5), float (5), been (5), gauge (5), your (5), documentation (5), valid (5), reference (5), memory (5), disk (5), prometheus (5), all (4), chat (4), sources (4), components (4), input (4), non (4), dropped (4), always (4), failed (4), rate (4), rate_limit_num (4), rate_limit_duration_secs (4), static (4), performance (4), does (4), based (4), false (4), only (4), hosts (4), supported (4), no_proxy (4), ingestion (4), healthcheck (4), require (4), check (4), retryable (4), connection (4), one (4), max_bytes (4), concepts (4), authentication (4), whether (4), support (4), server (4), how (4), output (4), were (4), origins (4), true (4), currently (4), deprecated (4), types (4), after (4), user_agent (4), should (4), stream_options (4), none (4), jitter (4), strategy (4), allowed (4), any (4), average (4), measurements (4), range (4), cause (4), less (4), traffic (4), auth (4), docs (4), architecture (4), rss (3), releases (3), download (3), updates (3), its (3), each (3), state (3), column (3), every (3), null (3), automatically (3), api (3), into (3), other (3), these (3), both (3), fixed (3), also (3), via (3), integer (3), feature (3), systems (3), downstream (3), blog (3), setting (3), grpc (3), own (3), invalid (3), issues (3), buffers (3), max_events (3), timeout_secs (3), flushed (3), them (3), global (3), delivery (3), guarantees (3), 10mb (3), smaller (3), but (3), encoded (3), larger (3), ingest (3), being (3), batching (3), credentials (3), permissions (3), controls (3), accepted (3), total (3), configurations (3), sanitized (3), pod (3), pod_name (3), pathname (3), peer_path (3), peer_addr (3), container (3), container_name (3), either (3), note (3), level (3), due (3), intentionally (3), filter (3), agent (3), find (3), workspace (3), create (3), timeout (3), amount (3), wait (3), delay (3), window (3), past (3), deviation (3), increases (3), scale (3), greater (3), bool (3), lost (3), buffering (3), observability (3), websocket (3), splunk (3), hec (3), azure (3), kinesis (3), going (2), administration (2), deployment (2), installation (2), setup (2), transforms (2), about (2), next (2), databend (2), stateless (2), across (2), columns (2), nullable (2), then (2), provide (2), fails (2), encoding (2), logged (2), fields (2), some (2), ensuring (2), format (2), status (2), 500 (2), responses (2), control (2), retry_attempts (2), policy (2), highly (2), enabling (2), such (2), have (2), arc (2), fallback (2), proxies (2), tls (2), channel (2), standard (2), environment (2), variables (2), disable (2), yaml (2), healthy (2), upon (2), failure (2), proceed (2), timeouts (2), controlled (2), manual (2), handling (2), meets (2), exceeds (2), age (2), conditions (2), above (2), treats (2), instead (2), against (2), while (2), applies (2), schemas (2), many (2), side (2), lower (2), uncompressed (2), batched (2), over (2), flight (2), client_secret (2), client_id (2), back (2), upstream (2), per (2), waits (2), acknowledgement (2), ratio (2), would (2), indicate (2), idle (2), waiting (2), emitted (2), region (2), protocol (2), absolute (2), path (2), destination (2), tagged (2), cumulatively (2), useful (2), histogram (2), topology (2), discarded (2), intentional (2), buffer_size_events (2), buffer_size_bytes (2), received (2), favor (2), telemetry (2), specified (2), header (2), appended (2), custom (2), metadata (2), 60000 (2), milliseconds (2), 30000 (2), defaults (2), recommends (2), below (2), pile (2), reduce (2), first (2), fibonacci (2), sequence (2), 223372036854776e (2), outbound (2), ignore (2), algorithm (2), takes (2), initial (2), high (2), down (2), parameters (2), typically (2), meta (2), settings (2), addresses (2), 127 (2), list (2), foo (2), bar (2), 3128 (2), proxied (2), block (2), acceptance (2), consumption (2), free (2), space (2), performant (2), durable (2), restarted (2), forcefully (2), crashes (2), buffered (2), synchronized (2), least (2), information (2), secret (2), connected (2), handled (2), open (2), home (2), statsd (2), socket (2), sematext (2), redis (2), pulsar (2), remote (2), opentelemetry (2), nats (2), mqtt (2), formerly (2), kafka (2), influxdb (2), humio (2), greptimedb (2), pubsub (2), stackdriver (2), storage (2), monitoring (2), sqs (2), firehose (2), cloudwatch (2), amqp (2), route (2), remap (2), windows (2), kubernetes (2), docker (2), linux (2), pipelines (2), guides (2), sidebar, 2026, inc, rights, reserved, community, prod, cookies, privacy, team, contact, site, footer, previous, thank, joining, our, newsletter, sign, receive, emails, latest, content, page, meaning, consistent, declares, otherwise, entire, naming, offending, prefer, absent, fetches, startup, additional, requires, encode, 408, 429, 501, retried, attempts, retry_backoff_secs, apply, addition, limiting, overall, throughput, manually, specifying, recommend, improves, reliability, communicates, made, further, away, optimizes, underlying, mechanism, feedback, loop, inspired, tcp, congestion, algorithms, checkout, announcement, post, disables, entirely, including, built, env, var, because, matching, bypass, uris, hop, handshake, trust, store, discovery, honor, itself, merged, https_proxy, http_proxy, config, etc, exit, immediately, pass, flag, ensure, accessible, ready, accept, performed, initialization, start, arguments, rejected, permanently, existing, kept, alive, failures, closed, discards, creates, fresh, attempt, ensures, recovery, transient, network, without, intervention, classifies, categories, met, shown, diagram, notice, differently, treating, isolates, services, disruptions, contained, honored, sizes, cap, most, comparable, numeric, heavy, form, right, boundary, exceed, call, fail, leave, headroom, 8mb, pre, serialization, single, enforced, recordbatch, authenticates, granted, confirmation, propagated, weaken, durability, guarantee, offset, considering, delivered, regardless, durably, reports, success, works, load, completely, simply, never, updated, utilization, component_sent_events_total, component_sent_event_bytes_total, send, excluding, query, raw, component_sent_bytes_total, component_received_events_total, separate, mostly, low, debugging, small, passed, component_received_events_count, component_received_event_bytes_total, occurred, error_type, encountered, component_errors_total, component_discarded_events_total, buffer_sent_events_total, buffer_sent_bytes_total, buffer_received_events_total, buffer_received_bytes_total, buffer_events, blocking, buffer_discarded_events_total, buffer_byte_size, link, following, lists, possible, aware, available, differ, codec, includes, version, user, f6e5d4c3, b2a1, vector_logs, identify, server_lack_of_ack_timeout_ms, flush, operations, flush_timeout_ms, zstandard, zstd, lz4, frame, lz4_frame, ipc, payloads, thin, wrapper, around, attributes, suitable, case, streamconfigurationoptions, could, orphaned, result, duplicate, take, aborted, between, retry_max_duration_secs, incorporating, greatly, likelihood, creating, accidental, denial, dos, clients, recovering, random, anywhere, calculated, retry_jitter_mode, select, future, backoffs, attempting, retry_initial_backoff_secs, make, outstanding, given, managed, positive, denotes, calculating, secondary, computed, indicates, variable, those, comparing, expected, factor, appropriate, equal, reasonable, deviations, considered, anomalous, rtt_deviation_scale, 200, bound, put, place, safeguard, max_concurrency_limit, seeing, long, ramp, restart, looking, adaptive_concurrency_limit, initial_concurrency, uses, exponentially, weighted, moving, ewma, compare, adjust, slowly, unusually, response, variability, weighting, compared, older, ewma_alpha, rounded, applying, rapidly, latency, fraction, decreasing, decrease_ratio, changes, incorrect, lead, stable, unstable, caution, follows, various, middleware, splat, 192, 168, blocks, cidr, subdomains, wildcard, domains, domain, names, match, pattern, multiple, patterns, avoid, enables, similar, common, convention, different, configure, through, making, external, prefix, array, info, wildcards, ids, 6543210987654321, east, starts, highest, priority, preferable, temporarily, lose, rather, slowdown, drops, drop_newest, backpressure, signalling, slow, means, edge, when_full, 500ms, 256, megabytes, 268435488, allocated, hold, max_size, relevant, individual, found, section, configures, they, serialized, compressed, processed, authenticate, secret123, databricks_client_secret, abc123, databricks_client_id, principal, grant, disabling, precedence, supports, acknowledged, acknowledging, json, toml, advanced, minimal, requirements, view, egress, yes, once, beta, versioning, security, glossary, cli, secrets, template, syntax, unit, tests, webhdfs, exporter, postgres, papertrail, relic, mezmo, logdna, loki, keep, honeycomb, chronicle, unstructured, elasticsearch, doris, traces, console, clickhouse, blackhole, monitor, blob, axiom, sns, streams, appsignal, trace, throttle, tag, cardinality, sample, lua, incremental, exclusive, dedupe, ec2, aggregate, vrl, syslog, stdin, scrape, pushgateway, postgresql, okta, nginx, mongodb, logstash, journald, heroku, logplex, fluent, descriptor, exec, eventstoredb, dnstap, demo, ecs, apache, expressions, functions, language, pgo, optimization, validating, management, runtime, sizing, capacity, planning, rollout, availability, hardening, architecting, unified, aggregator, architectures, production, topologies, roles, installer, archives, ubuntu, rhel, raspbian, nixos, macos, debian, centos, arch, amazon, operating, platforms, yum, rpm, pacman, nix, msi, homebrew, helm, dpkg, apt, package, managers, quickstart, introduction, navbar, dropdown, menu, search, toggle, dark,

Text of the page (random words):
ng acknowledgements at the sink level takes precedence over any global acknowledgements configuration auth required object databricks authentication configuration see the databricks zerobus documentation to create a service principal and grant it permissions to write to the target table auth client_id required string literal oauth 2 0 client id examples databricks_client_id abc123 auth client_secret required string literal oauth 2 0 client secret examples databricks_client_secret secret123 auth strategy required string literal enum the authentication strategy to use for databricks enum options option description oauth authenticate using oauth 2 0 client credentials examples oauth batch optional object event batching behavior batch max_bytes optional uint the maximum size of a batch that is processed by a sink this is based on the uncompressed size of the batched events before they are serialized or compressed default 1e 07 bytes batch max_events optional uint the maximum size of a batch before it is flushed batch timeout_secs optional float the maximum age of a batch before it is flushed default 1 seconds buffer optional object configures the buffering behavior for this sink more information about the individual buffer types and buffer behavior can be found in the buffering model section buffer max_events optional uint the maximum number of events allowed in the buffer relevant when type memory default 500 buffer max_size required uint the maximum allowed amount of allocated memory the buffer can hold if type disk then must be at least 256 megabytes 268435488 bytes buffer type optional string literal enum the type of buffer to use enum options option description disk events are buffered on disk this is less performant but more durable data that has been synchronized to disk will not be lost if vector is restarted forcefully or crashes data is synchronized to disk every 500ms memory events are buffered in memory this is more performant but less durable data will be lost if vector is restarted forcefully or crashes default memory buffer when_full optional string literal enum event handling behavior when a buffer is full enum options option description block wait for free space in the buffer this applies backpressure up the topology signalling that sources should slow down the acceptance consumption of events this means that while no data is lost data will pile up at the edge drop_newest drops the event instead of waiting for free space in buffer the event will be intentionally dropped this mode is typically used when performance is the highest priority and it is preferable to temporarily lose events rather than cause a slowdown in the acceptance consumption of events default block healthcheck optional object healthcheck configuration healthcheck enabled optional bool whether or not to check the health of the sink when vector starts up default true ingestion_endpoint required string literal the zerobus ingestion endpoint url this should be the full url to the zerobus ingestion service see the databricks zerobus documentation to find your workspace url and zerobus ingest endpoint examples https 1234567890123456 zerobus us west 2 cloud databricks com https 6543210987654321 zerobus us east 1 cloud databricks com inputs required string a list of upstream source or transform ids wildcards are supported see configuration for more info array string literal examples my source or transform id prefix proxy optional object proxy configuration configure to proxy traffic through an http s proxy when making external requests similar to common proxy configuration convention you can set different proxies to use based on the type of traffic being proxied you can also set specific hosts that should not be proxied proxy enabled optional bool enables proxying support default true proxy http optional string literal proxy endpoint to use when proxying http traffic must be a valid uri string examples http foo bar 3128 proxy https optional string literal proxy endpoint to use when proxying https traffic must be a valid uri string examples http foo bar 3128 proxy no_proxy optional string a list of hosts to avoid proxying multiple patterns are allowed pattern example match domain names example com matches requests to example com wildcard domains example com matches requests to example com and its subdomains ip addresses 127 0 0 1 matches requests to 127 0 0 1 cidr blocks 192 168 0 0 16 matches requests to any ip addresses in this range splat matches all hosts request optional object middleware settings for outbound requests various settings can be configured such as concurrency and rate limits timeouts and retry behavior note that the retry backoff policy follows the fibonacci sequence request adaptive_concurrency optional object configuration of adaptive concurrency parameters these parameters typically do not require changes from the default and incorrect values can lead to meta stable or unstable performance and sink behavior proceed with caution request adaptive_concurrency decrease_ratio optional float the fraction of the current value to set the new concurrency limit when decreasing the limit valid values are greater than 0 and less than 1 smaller values cause the algorithm to scale back rapidly when latency increases note the new limit is rounded down after applying this ratio default 0 9 request adaptive_concurrency ewma_alpha optional float the weighting of new measurements compared to older measurements valid values are greater than 0 and less than 1 arc uses an exponentially weighted moving average ewma of past rtt measurements as a reference to compare with the current rtt smaller values cause this reference to adjust more slowly which may be useful if a service has unusually high response variability default 0 4 request adaptive_concurrency initial_concurrency optional uint the initial concurrency limit to use if not specified the initial limit is 1 no concurrency datadog recommends setting this value to your service s average limit if you re seeing that it takes a long time to ramp up adaptive concurrency after a restart you can find this value by looking at the adaptive_concurrency_limit metric default 1 request adaptive_concurrency max_concurrency_limit optional uint the maximum concurrency limit the adaptive request concurrency limit does not go above this bound this is put in place as a safeguard default 200 request adaptive_concurrency rtt_deviation_scale optional float scale of rtt deviations which are not considered anomalous valid values are greater than or equal to 0 and reasonable values range from 1 0 to 3 0 when calculating the past rtt average a secondary deviation value is also computed that indicates how variable those values are that deviation is used when comparing the past rtt average to the current measurements so we can ignore increases in rtt that are within an expected range this factor is used to scale up the deviation to an appropriate range larger values cause the algorithm to ignore larger increases in the rtt default 2 5 request concurrency optional string literal enum uint configuration for outbound request concurrency this can be set either to one of the below enum values or to a positive integer which denotes a fixed concurrency limit enum options option description adaptive concurrency is managed by vector s adaptive request concurrency feature none a fixed concurrency of 1 only one request can be outstanding at any given time default adaptive request rate_limit_duration_secs optional uint the time window used for the rate_limit_num option default 1 seconds request rate_limit_num optional uint the maximum number of requests allowed within the rate_limit_duration_secs time window default 9 223372036854776e 18 requests request retry_attempts optional uint the maximum number of retries to make for failed requests default 9 223372036854776e 18 retries request retry_initial_backoff_secs optional uint the amount of time to wait before attempting the first retry for a failed request after the first retry has failed the fibonacci sequence is used to select future backoffs default 1 seconds request retry_jitter_mode optional string literal enum the jitter mode to use for retry backoff behavior enum options option description full full jitter the random delay is anywhere from 0 up to the maximum current delay calculated by the backoff strategy incorporating full jitter into your backoff strategy can greatly reduce the likelihood of creating accidental denial of service dos conditions against your own systems when many clients are recovering from a failure state none no jitter default full request retry_max_duration_secs optional uint the maximum amount of time to wait between retries default 30 seconds request timeout_secs optional uint the time a request can take before being aborted datadog highly recommends that you do not lower this value below the service s internal timeout as this could create orphaned requests pile on retries and result in duplicate data downstream default 60 seconds stream_options optional object zerobus stream configuration options this is a thin wrapper around the sdk s streamconfigurationoptions with vector specific configuration attributes and custom defaults suitable for vector s use case stream_options compression optional string literal enum arrow ipc compression for flight payloads defaults to no compression enum options option description lz4_frame lz4 frame compression none no compression zstd zstandard compression default none stream_options flush_timeout_ms optional uint timeout in milliseconds for flush operations examples 30000 default 30000 stream_options server_lack_of_ack_timeout_ms optional uint timeout in milliseconds for server acknowledgements examples 60000 default 60000 table_name required string literal the unity catalog table name to write to this should be in the format catalog schema table see the databricks zerobus documentation to create or identify the target table examples main default logs main default vector_logs unity_catalog_endpoint required string literal the unity catalog endpoint url this is used for authentication and table metadata see the databricks zerobus documentation to find your workspace url and zerobus ingest endpoint examples https dbc a1b2c3d4 e5f6 cloud databricks com https dbc f6e5d4c3 b2a1 cloud databricks com user_agent optional string literal custom identifier appended to the user agent header sent to databricks the header always includes vector version when set this value is appended after it e g my service 1 2 examples my service 1 2 input types the following table lists all telemetry data types supported by the component across possible configurations be aware that the available data types may differ based on the specified codec configuration logs log events are supported telemetry metrics link buffer_byte_size gauge the number of bytes currently in the buffer deprecated this metric has been deprecated in favor of buffer_size_bytes buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_discarded_events_total counter the number of events dropped by this non blocking buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on intentional true if the events were discarded intentionally like a filter transform or false if due to an error pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_events gauge the number of events currently in the buffer deprecated this metric has been deprecated in favor of buffer_size_events buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_received_bytes_total counter the number of bytes received by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_received_events_total counter the number of events received by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_sent_bytes_total counter the number of bytes sent by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_sent_events_total counter the number of events sent by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_size_bytes gauge the number of bytes currently in the buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_size_events gauge the number of events currently in the buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline component_discarded_events_total counter the number of events dropped by this component component_id the vector component id component_kind the vector component kind component_type the vector component type...