Meta tags:
description= Stream observability data to Databricks Unity Catalog via Zerobus;
keywords= databricks,component,sink,logs;
Headings (most frequently used words):
request, proxy, adaptive_concurrency, buffer, auth, batch, stream_options, concurrency, acknowledgements, health, checks, enabled, healthcheck, rate, limits, adaptive, max_events, timeout_secs, deprecated, databricks, zerobus, requirements, configuration, input, types, telemetry, how, it, works, vector, site, footer, example, configurations, ingestion_endpoint, inputs, table_name, unity_catalog_endpoint, user_agent, logs, metrics, authentication, batching, buffers, and, batches, error, handling, retries, retry, policy, schema, state, about, components, setup, community, download, client_id, client_secret, strategy, max_bytes, max_size, type, when_full, http, https, no_proxy, rate_limit_duration_secs, rate_limit_num, retry_attempts, retry_initial_backoff_secs, retry_jitter_mode, retry_max_duration_secs, compression, flush_timeout_ms, server_lack_of_ack_timeout_ms, buffer_byte_size, buffer_discarded_events_total, buffer_events, buffer_received_bytes_total, buffer_received_events_total, buffer_sent_bytes_total, buffer_sent_events_total, buffer_size_bytes, buffer_size_events, component_discarded_events_total, component_errors_total, component_received_event_bytes_total, component_received_events_count, component_received_events_total, component_sent_bytes_total, component_sent_event_bytes_total, component_sent_events_total, utilization, require, disable, arc, static, decrease_ratio, ewma_alpha, initial_concurrency, max_concurrency_limit, rtt_deviation_scale,
Text of the page (most frequently used words):
the (403), vector (121), optional (99), component (73), this (52), and (47), #buffer (41), #request (36), for (35), default (34), from (33), data (32), zerobus (30), databricks (30), type (30), concurrency (29), events (28), that (27), are (25), which (25), proxy (24), https (24), logs (24), sink (22), batch (21), number (21), configuration (20), host (20), string (20), stage (20), com (20), system (19), hostname (19), can (18), process (18), instance (18), pid (18), running (18), component_type (18), kind (18), component_kind (18), component_id (18), originated (18), cloud (18), you (17), metrics (17), when (16), literal (16), acknowledgements (15), uint (15), sinks (14), not (14), service (14), http (13), aws (13), file (13), examples (13), enum (13), table (12), requests (12), options (12), adaptive (12), limit (12), used (12), source (12), end (12), counter (12), bytes (12), within (12), event (11), required (11), will (11), values (11), maximum (11), pipeline (11), value (10), retry (10), with (10), endpoint (10), uri (10), transform (10), identifier (10), datadog (9), behavior (9), enabled (9), client (9), before (9), mode (9), numbered (9), unique (9), buffer_id (9), object (9), schema (8), catalog (8), unity_catalog_endpoint (8), table_name (8), set (8), health (8), option (8), size (8), than (8), sent (8), name (8), dbc (8), main (8), use (8), error (7), unity (7), sdk (7), errors (7), see (7), example (7), oauth (7), a1b2c3d4 (7), e5f6 (7), time (7), rtt (7), inputs (7), 1234567890123456 (7), west (7), ingestion_endpoint (7), target (6), configured (6), matches (6), arrow (6), backoff (6), url (6), like (6), write (6), has (6), seconds (6), internal (6), metric (6), log (6), compression (6), description (6), full (6), adaptive_concurrency (6), more (6), model (6), databricks_zerobus (6), my_sink_id (6), gcp (6), github (5), new (5), must (5), may (5), using (5), limits (5), proxying (5), checks (5), stream (5), current (5), retries (5), batches (5), specific (5), float (5), been (5), gauge (5), your (5), documentation (5), valid (5), reference (5), memory (5), disk (5), prometheus (5), all (4), chat (4), sources (4), components (4), input (4), non (4), dropped (4), always (4), failed (4), rate (4), rate_limit_num (4), rate_limit_duration_secs (4), static (4), performance (4), does (4), based (4), false (4), only (4), hosts (4), supported (4), no_proxy (4), ingestion (4), healthcheck (4), require (4), check (4), retryable (4), connection (4), one (4), max_bytes (4), concepts (4), authentication (4), whether (4), support (4), server (4), how (4), output (4), were (4), origins (4), true (4), currently (4), deprecated (4), types (4), after (4), user_agent (4), should (4), stream_options (4), none (4), jitter (4), strategy (4), allowed (4), any (4), average (4), measurements (4), range (4), cause (4), less (4), traffic (4), auth (4), docs (4), architecture (4), rss (3), releases (3), download (3), updates (3), its (3), each (3), state (3), column (3), every (3), null (3), automatically (3), api (3), into (3), other (3), these (3), both (3), fixed (3), also (3), via (3), integer (3), feature (3), systems (3), downstream (3), blog (3), setting (3), grpc (3), own (3), invalid (3), issues (3), buffers (3), max_events (3), timeout_secs (3), flushed (3), them (3), global (3), delivery (3), guarantees (3), 10mb (3), smaller (3), but (3), encoded (3), larger (3), ingest (3), being (3), batching (3), credentials (3), permissions (3), controls (3), accepted (3), total (3), configurations (3), sanitized (3), pod (3), pod_name (3), pathname (3), peer_path (3), peer_addr (3), container (3), container_name (3), either (3), note (3), level (3), due (3), intentionally (3), filter (3), agent (3), find (3), workspace (3), create (3), timeout (3), amount (3), wait (3), delay (3), window (3), past (3), deviation (3), increases (3), scale (3), greater (3), bool (3), lost (3), buffering (3), observability (3), websocket (3), splunk (3), hec (3), azure (3), kinesis (3), going (2), administration (2), deployment (2), installation (2), setup (2), transforms (2), about (2), next (2), databend (2), stateless (2), across (2), columns (2), nullable (2), then (2), provide (2), fails (2), encoding (2), logged (2), fields (2), some (2), ensuring (2), format (2), status (2), 500 (2), responses (2), control (2), retry_attempts (2), policy (2), highly (2), enabling (2), such (2), have (2), arc (2), fallback (2), proxies (2), tls (2), channel (2), standard (2), environment (2), variables (2), disable (2), yaml (2), healthy (2), upon (2), failure (2), proceed (2), timeouts (2), controlled (2), manual (2), handling (2), meets (2), exceeds (2), age (2), conditions (2), above (2), treats (2), instead (2), against (2), while (2), applies (2), schemas (2), many (2), side (2), lower (2), uncompressed (2), batched (2), over (2), flight (2), client_secret (2), client_id (2), back (2), upstream (2), per (2), waits (2), acknowledgement (2), ratio (2), would (2), indicate (2), idle (2), waiting (2), emitted (2), region (2), protocol (2), absolute (2), path (2), destination (2), tagged (2), cumulatively (2), useful (2), histogram (2), topology (2), discarded (2), intentional (2), buffer_size_events (2), buffer_size_bytes (2), received (2), favor (2), telemetry (2), specified (2), header (2), appended (2), custom (2), metadata (2), 60000 (2), milliseconds (2), 30000 (2), defaults (2), recommends (2), below (2), pile (2), reduce (2), first (2), fibonacci (2), sequence (2), 223372036854776e (2), outbound (2), ignore (2), algorithm (2), takes (2), initial (2), high (2), down (2), parameters (2), typically (2), meta (2), settings (2), addresses (2), 127 (2), list (2), foo (2), bar (2), 3128 (2), proxied (2), block (2), acceptance (2), consumption (2), free (2), space (2), performant (2), durable (2), restarted (2), forcefully (2), crashes (2), buffered (2), synchronized (2), least (2), information (2), secret (2), connected (2), handled (2), open (2), home (2), statsd (2), socket (2), sematext (2), redis (2), pulsar (2), remote (2), opentelemetry (2), nats (2), mqtt (2), formerly (2), kafka (2), influxdb (2), humio (2), greptimedb (2), pubsub (2), stackdriver (2), storage (2), monitoring (2), sqs (2), firehose (2), cloudwatch (2), amqp (2), route (2), remap (2), windows (2), kubernetes (2), docker (2), linux (2), pipelines (2), guides (2), sidebar, 2026, inc, rights, reserved, community, prod, cookies, privacy, team, contact, site, footer, previous, thank, joining, our, newsletter, sign, receive, emails, latest, content, page, meaning, consistent, declares, otherwise, entire, naming, offending, prefer, absent, fetches, startup, additional, requires, encode, 408, 429, 501, retried, attempts, retry_backoff_secs, apply, addition, limiting, overall, throughput, manually, specifying, recommend, improves, reliability, communicates, made, further, away, optimizes, underlying, mechanism, feedback, loop, inspired, tcp, congestion, algorithms, checkout, announcement, post, disables, entirely, including, built, env, var, because, matching, bypass, uris, hop, handshake, trust, store, discovery, honor, itself, merged, https_proxy, http_proxy, config, etc, exit, immediately, pass, flag, ensure, accessible, ready, accept, performed, initialization, start, arguments, rejected, permanently, existing, kept, alive, failures, closed, discards, creates, fresh, attempt, ensures, recovery, transient, network, without, intervention, classifies, categories, met, shown, diagram, notice, differently, treating, isolates, services, disruptions, contained, honored, sizes, cap, most, comparable, numeric, heavy, form, right, boundary, exceed, call, fail, leave, headroom, 8mb, pre, serialization, single, enforced, recordbatch, authenticates, granted, confirmation, propagated, weaken, durability, guarantee, offset, considering, delivered, regardless, durably, reports, success, works, load, completely, simply, never, updated, utilization, component_sent_events_total, component_sent_event_bytes_total, send, excluding, query, raw, component_sent_bytes_total, component_received_events_total, separate, mostly, low, debugging, small, passed, component_received_events_count, component_received_event_bytes_total, occurred, error_type, encountered, component_errors_total, component_discarded_events_total, buffer_sent_events_total, buffer_sent_bytes_total, buffer_received_events_total, buffer_received_bytes_total, buffer_events, blocking, buffer_discarded_events_total, buffer_byte_size, link, following, lists, possible, aware, available, differ, codec, includes, version, user, f6e5d4c3, b2a1, vector_logs, identify, server_lack_of_ack_timeout_ms, flush, operations, flush_timeout_ms, zstandard, zstd, lz4, frame, lz4_frame, ipc, payloads, thin, wrapper, around, attributes, suitable, case, streamconfigurationoptions, could, orphaned, result, duplicate, take, aborted, between, retry_max_duration_secs, incorporating, greatly, likelihood, creating, accidental, denial, dos, clients, recovering, random, anywhere, calculated, retry_jitter_mode, select, future, backoffs, attempting, retry_initial_backoff_secs, make, outstanding, given, managed, positive, denotes, calculating, secondary, computed, indicates, variable, those, comparing, expected, factor, appropriate, equal, reasonable, deviations, considered, anomalous, rtt_deviation_scale, 200, bound, put, place, safeguard, max_concurrency_limit, seeing, long, ramp, restart, looking, adaptive_concurrency_limit, initial_concurrency, uses, exponentially, weighted, moving, ewma, compare, adjust, slowly, unusually, response, variability, weighting, compared, older, ewma_alpha, rounded, applying, rapidly, latency, fraction, decreasing, decrease_ratio, changes, incorrect, lead, stable, unstable, caution, follows, various, middleware, splat, 192, 168, blocks, cidr, subdomains, wildcard, domains, domain, names, match, pattern, multiple, patterns, avoid, enables, similar, common, convention, different, configure, through, making, external, prefix, array, info, wildcards, ids, 6543210987654321, east, starts, highest, priority, preferable, temporarily, lose, rather, slowdown, drops, drop_newest, backpressure, signalling, slow, means, edge, when_full, 500ms, 256, megabytes, 268435488, allocated, hold, max_size, relevant, individual, found, section, configures, they, serialized, compressed, processed, authenticate, secret123, databricks_client_secret, abc123, databricks_client_id, principal, grant, disabling, precedence, supports, acknowledged, acknowledging, json, toml, advanced, minimal, requirements, view, egress, yes, once, beta, versioning, security, glossary, cli, secrets, template, syntax, unit, tests, webhdfs, exporter, postgres, papertrail, relic, mezmo, logdna, loki, keep, honeycomb, chronicle, unstructured, elasticsearch, doris, traces, console, clickhouse, blackhole, monitor, blob, axiom, sns, streams, appsignal, trace, throttle, tag, cardinality, sample, lua, incremental, exclusive, dedupe, ec2, aggregate, vrl, syslog, stdin, scrape, pushgateway, postgresql, okta, nginx, mongodb, logstash, journald, heroku, logplex, fluent, descriptor, exec, eventstoredb, dnstap, demo, ecs, apache, expressions, functions, language, pgo, optimization, validating, management, runtime, sizing, capacity, planning, rollout, availability, hardening, architecting, unified, aggregator, architectures, production, topologies, roles, installer, archives, ubuntu, rhel, raspbian, nixos, macos, debian, centos, arch, amazon, operating, platforms, yum, rpm, pacman, nix, msi, homebrew, helm, dpkg, apt, package, managers, quickstart, introduction, navbar, dropdown, menu, search, toggle, dark,
Text of the page (random words):
retry_initial_backoff_secs optional uint the amount of time to wait before attempting the first retry for a failed request after the first retry has failed the fibonacci sequence is used to select future backoffs default 1 seconds request retry_jitter_mode optional string literal enum the jitter mode to use for retry backoff behavior enum options option description full full jitter the random delay is anywhere from 0 up to the maximum current delay calculated by the backoff strategy incorporating full jitter into your backoff strategy can greatly reduce the likelihood of creating accidental denial of service dos conditions against your own systems when many clients are recovering from a failure state none no jitter default full request retry_max_duration_secs optional uint the maximum amount of time to wait between retries default 30 seconds request timeout_secs optional uint the time a request can take before being aborted datadog highly recommends that you do not lower this value below the service s internal timeout as this could create orphaned requests pile on retries and result in duplicate data downstream default 60 seconds stream_options optional object zerobus stream configuration options this is a thin wrapper around the sdk s streamconfigurationoptions with vector specific configuration attributes and custom defaults suitable for vector s use case stream_options compression optional string literal enum arrow ipc compression for flight payloads defaults to no compression enum options option description lz4_frame lz4 frame compression none no compression zstd zstandard compression default none stream_options flush_timeout_ms optional uint timeout in milliseconds for flush operations examples 30000 default 30000 stream_options server_lack_of_ack_timeout_ms optional uint timeout in milliseconds for server acknowledgements examples 60000 default 60000 table_name required string literal the unity catalog table name to write to this should be in the format catalog schema table see the databricks zerobus documentation to create or identify the target table examples main default logs main default vector_logs unity_catalog_endpoint required string literal the unity catalog endpoint url this is used for authentication and table metadata see the databricks zerobus documentation to find your workspace url and zerobus ingest endpoint examples https dbc a1b2c3d4 e5f6 cloud databricks com https dbc f6e5d4c3 b2a1 cloud databricks com user_agent optional string literal custom identifier appended to the user agent header sent to databricks the header always includes vector version when set this value is appended after it e g my service 1 2 examples my service 1 2 input types the following table lists all telemetry data types supported by the component across possible configurations be aware that the available data types may differ based on the specified codec configuration logs log events are supported telemetry metrics link buffer_byte_size gauge the number of bytes currently in the buffer deprecated this metric has been deprecated in favor of buffer_size_bytes buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_discarded_events_total counter the number of events dropped by this non blocking buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on intentional true if the events were discarded intentionally like a filter transform or false if due to an error pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_events gauge the number of events currently in the buffer deprecated this metric has been deprecated in favor of buffer_size_events buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_received_bytes_total counter the number of bytes received by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_received_events_total counter the number of events received by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_sent_bytes_total counter the number of bytes sent by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_sent_events_total counter the number of events sent by this buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_size_bytes gauge the number of bytes currently in the buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline buffer_size_events gauge the number of events currently in the buffer buffer_id the unique identifier of the buffer component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the numbered stage within the buffer pipeline component_discarded_events_total counter the number of events dropped by this component component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on intentional true if the events were discarded intentionally like a filter transform or false if due to an error pid optional the process id of the vector instance component_errors_total counter the total number of errors encountered by this component component_id the vector component id component_kind the vector component kind component_type the vector component type error_type the type of the error host optional the hostname of the system vector is running on pid optional the process id of the vector instance stage the stage within the component at which the error occurred component_received_event_bytes_total counter the number of event bytes accepted by this component either from tagged origins like file and uri or cumulatively from other origins component_id the vector component id component_kind the vector component kind component_type the vector component type container_name optional the name of the container from which the data originated file optional the file from which the data originated host optional the hostname of the system vector is running on mode optional the connection mode used by the component peer_addr optional the ip from which the data originated peer_path optional the pathname from which the data originated pid optional the process id of the vector instance pod_name optional the name of the pod from which the data originated uri optional the sanitized uri from which the data originated component_received_events_count histogram a histogram of the number of events passed in each internal batch in vector s internal topology note that this is separate than sink level batching it is mostly useful for low level debugging performance issues in vector due to small internal batches component_id the vector component id component_kind the vector component kind component_type the vector component type container_name optional the name of the container from which the data originated file optional the file from which the data originated host optional the hostname of the system vector is running on mode optional the connection mode used by the component peer_addr optional the ip from which the data originated peer_path optional the pathname from which the data originated pid optional the process id of the vector instance pod_name optional the name of the pod from which the data originated uri optional the sanitized uri from which the data originated component_received_events_total counter the number of events accepted by this component either from tagged origins like file and uri or cumulatively from other origins component_id the vector component id component_kind the vector component kind component_type the vector component type container_name optional the name of the container from which the data originated file optional the file from which the data originated host optional the hostname of the system vector is running on mode optional the connection mode used by the component peer_addr optional the ip from which the data originated peer_path optional the pathname from which the data originated pid optional the process id of the vector instance pod_name optional the name of the pod from which the data originated uri optional the sanitized uri from which the data originated component_sent_bytes_total counter the number of raw bytes sent by this component to destination sinks component_id the vector component id component_kind the vector component kind component_type the vector component type endpoint optional the endpoint to which the bytes were sent for http this will be the host and path only excluding the query string file optional the absolute path of the destination file host optional the hostname of the system vector is running on pid optional the process id of the vector instance protocol the protocol used to send the bytes region optional the aws region name to which the bytes were sent in some configurations this may be a literal hostname component_sent_event_bytes_total counter the total number of event bytes emitted by this component component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on output optional the specific output of the component pid optional the process id of the vector instance component_sent_events_total counter the total number of events emitted by this component component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on output optional the specific output of the component pid optional the process id of the vector instance utilization gauge a ratio from 0 to 1 of the load on a component a value of 0 would indicate a completely idle component that is simply waiting for input a value of 1 would indicate a that is never idle this value is updated every 5 seconds component_id the vector component id component_kind the vector component kind component_type the vector component type host optional the hostname of the system vector is running on pid optional the process id of the vector instance how it works acknowledgements the sink always waits for a per batch server side offset acknowledgement from zerobus before considering a batch delivered regardless of whether vector s end to end acknowledgements are enabled this guarantees that data has been durably accepted by the zerobus service before the sink reports success vector s acknowledgements enabled setting only controls whether that delivery confirmation is propagated back to upstream sources that support end to end acknowledgements it does not weaken the sink s own per batch durability guarantee authentication the databricks zerobus sink authenticates using oauth 2 0 client credentials you must provide a client_id and client_secret that have been granted permissions to write to the target unity catalog table batching events are batched before being sent to zerobus each batch is encoded as a single arrow recordbatch and sent over arrow flight as one request the maximum batch size is 10mb enforced by the zerobus sdk vector sizes batches against batch max_bytes using the uncompressed pre serialization event size while the sdk s 10mb cap applies to the encoded arrow size for most schemas the arrow encoding is smaller than or comparable to the source event but for numeric heavy schemas many integer or float fields the encoded form can be larger so a batch configured right at the 10mb boundary may exceed the sdk limit and the ingest call will fail if you see sdk side size errors lower batch max_bytes to leave headroom for example 8mb buffers and batches this component buffers batches data as shown in the diagram above you ll notice that vector treats these concepts differently instead of treating them as global concepts vector treats them as sink specific concepts this isolates sinks ensuring services disruptions are contained and delivery guarantees are honored batches are flushed when 1 of 2 conditions are met the batch age meets or exceeds the configured timeout_secs the batch size meets or exceeds the configured max_bytes or max_events buffers are controlled via the buffer options error handling retries the sink classifies errors from the zerobus sdk into retryable and non retryable categories retryable errors connection failures stream closed channel errors the sink automatically discards the current grpc stream and creates a fresh one on the next retry attempt this ensures recovery from transient network issues without manual intervention non retryable errors invalid table invalid endpoint invalid arguments events are rejected permanently and the existing stream is kept alive retry behavior backoff concurrency timeouts is controlled by the standard request configuration options health c...
|
