Meta tags:
description= Common errors and frequently asked questions;
Headings (most frequently used words):
how, ssh, or, why, is, the, borg, questions, append, only, borgbase, my, prune, repository, can, archives, mode, other, connections, to, repo, with, an, error, after, backup, get, data, repositories, storage, use, key, passphrase, and, from, in, do, tips, frequently, asked, troubleshooting, all, fail, immediately, connection, breaks, long, operation, integrity, when, accessing, process, so, slow, repair, damaged, still, delete, active, showing, same, deleting, prunging, what, difference, between, password, which, backend, are, you, using, migrate, existing, including, size, shown, web, interface, different, info, often, should, run, check, fully, remove, account, there, maintenance, window, support, have, any, email, us, frequent, keepalive, packets, enable, automatic, retries, borgmatic, external, retry, mechanism, debug, packet, loss, unstable, general, for, missing, segments, cache, issues,
Text of the page (most frequently used words):
the (147), borg (77), you (75), and (71), ssh (55), this (52), for (49), with (44), can (43), #borgbase (35), data (34), key (33), will (31), repo (30), are (29), your (27), backup (25), #repository (24), only (23), using (22), run (19), from (19), have (18), connection (17), check (16), use (16), account (15), when (15), backups (14), get (14), how (13), there (13), see (13), archives (13), keys (13), also (12), not (12), all (12), prune (12), delete (12), append (12), any (11), borgmatic (11), more (11), old (11), passphrase (11), time (10), that (10), server (10), our (9), first (9), com (9), remove (9), after (9), segments (9), mode (9), machine (9), issue (8), here (8), repositories (8), make (8), still (8), command (8), running (8), storage (8), full (8), files (8), why (8), add (8), doesn (8), same (8), client (8), error (8), questions (7), some (7), issues (7), but (7), them (7), then (7), archive (7), speed (7), access (7), usage (7), disk (7), different (7), need (7), which (7), used (7), password (7), side (7), config (7), cache (7), network (7), remote (7), support (6), maintenance (6), new (6), keep (6), may (6), create (6), should (6), free (6), space (6), compact (6), means (6), docs (6), size (6), details (6), process (6), repo_url (6), below (6), connections (6), retry (6), other (5), documentation (5), restore (5), during (5), done (5), security (5), while (5), set (5), repos (5), like (5), version (5), operation (5), deleted (5), one (5), case (5), well (5), copy (5), high (5), errors (5), xxx (5), restic (5), too (5), send (5), host (5), tracker (4), vorta (4), best (4), usually (4), source (4), few (4), day (4), window (4), necessary (4), servers (4), log (4), via (4), another (4), excessive (4), checks (4), those (4), default (4), slower (4), already (4), write (4), transaction (4), than (4), over (4), always (4), login (4), steps (4), configuration (4), where (4), work (4), part (4), encrypted (4), upload (4), cleaned (4), before (4), about (4), try (4), local (4), missing (4), long (4), compression (4), many (4), slow (4), packet (4), failed (4), closed (4), connect (3), system (3), response (3), general (3), open (3), related (3), hours (3), working (3), won (3), utc (3), region (3), apply (3), way (3), service (3), doing (3), into (3), sure (3), transfer (3), whole (3), found (3), ever (3), fully (3), contact (3), users (3), supports (3), once (3), every (3), higher (3), often (3), result (3), versions (3), actual (3), commands (3), never (3), what (3), file (3), reported (3), info (3), shown (3), public (3), both (3), based (3), redundancy (3), list (3), setup (3), hardware (3), multiple (3), centers (3), backend (3), encrypt (3), private (3), authentication (3), possible (3), control (3), panel (3), uses (3), added (3), back (3), deleting (3), enabled (3), level (3), operations (3), low (3), change (3), official (3), step (3), fix (3), repair (3), damaged (3), avoid (3), consistency (3), unstable (3), faq (3), bbr (3), latency (3), bandwidth (3), times (3), speeds (3), just (3), uncover (3), loss (3), minutes (3), sent (3), id_custom (3), permission (3), email (2), quick (2), help (2), out (2), projects (2), example (2), specific (2), question (2), receive (2), software (2), better (2), read (2), search (2), github (2), simply (2), generally (2), within (2), required (2), unless (2), urgent (2), action (2), needed (2), services (2), updates (2), each (2), choose (2), start (2), again (2), order (2), ansible (2), month (2), random (2), large (2), much (2), really (2), since (2), raid (2), verify (2), recommend (2), previous (2), they (2), variations (2), probably (2), includes (2), metadata (2), under (2), expected (2), web (2), interface (2), provide (2), internet (2), incoming (2), transfers (2), offer (2), follow (2), migrate (2), existing (2), including (2), xxxxxx (2), yyyyyy (2), two (2), supported (2), allows (2), adding (2), good (2), most (2), failures (2), purposes (2), backed (2), performance (2), hard (2), drives (2), site (2), authenticate (2), keypair (2), separate (2), replace (2), even (2), happens (2), automation (2), isn (2), prompted (2), don (2), generate (2), easily (2), difference (2), between (2), directly (2), transactions (2), changed (2), permissions (2), delay (2), section (2), associated (2), situations (2), removing (2), automatically (2), showing (2), prunging (2), older (2), edits (2), delayed (2), deletion (2), chunks (2), active (2), instead (2), own (2), repo_id (2), next (2), non (2), newer (2), checking (2), their (2), take (2), limit (2), ensure (2), compare (2), ctime (2), appropriate (2), zstd (2), tips (2), congestion (2), feature (2), sending (2), user (2), net (2), sysctl (2), etc (2), linux (2), would (2), limited (2), settings (2), want (2), being (2), busy (2), main (2), bottleneck (2), cpu (2), smaller (2), minimum (2), logs (2), integrity (2), accessing (2), iperf3 (2), xxxxx (2), test (2), retransmits (2), enable (2), mtr (2), 1000 (2), traceroute (2), routing (2), restricted (2), debug (2), shell (2), external (2), retries (2), alive (2), has (2), packets (2), been (2), broken (2), pipe (2), bytes (2), breaks (2), tried (2), xxxx (2), 0600 (2), per (2), identityfile (2), assigned (2), fingerprint (2), asked (2), fail (2), immediately (2), troubleshooting (2), link (2), require, additional, integration, relevant, maintainer, parent, company, offering, happy, around, setting, various, eco, others, effort, basis, speedy, feedback, project, genuine, bugs, links, regarding, otherwise, hello, downtime, restarts, sometimes, restart, such, schedule, minimize, disruption, follows, navigate, click, profile, saved, elsewhere, described, rsync, prefer, permanently, had, paid, subscription, invoicing, law, notice, interrupt, processes, guarantee, monthly, true, borgmatic_large_repo, role, cron, task, detect, cause, load, underlying, controller, does, weekly, scrubs, strongly, starting, perform, caluclated, now, match, closely, appended, nothing, removed, merely, tag, larger, deletes, thinks, displays, measured, index, slight, deduplicated, target, accessible, somehow, ftp, http, empty, migration, outgoing, currently, manually, these, location, geographic, regions, parallel, provides, durability, against, kinds, geographical, across, single, thus, recommended, archiving, deploy, mix, nvme, caches, enterprise, reliablity, replacement, ready, failovers, providers, gdpr, page, connecting, summarize, terms, encrypts, recover, somewhere, safe, off, something, encryption, addition, protected, interfere, preferred, without, totally, clean, suitable, trigger, table, explicitely, implicitely, grow, timing, conditions, depends, released, recently, ignore, requests, actually, rolled, segment, referencing, stay, reasons, preferably, download, sftp, timestamped, states, were, issued, going, effectively, recovered, pruning, quota, clear, pruned, knows, current, architecture, reject, right, away, know, called, prunes, deletions, undone, useful, shouldn, its, prevent, hacker, taking, instruction, roll, resolve, moving, folder, finally, potentially, ask, reveals, critical, fixing, independently, updated, object, rare, interrupted, incomplete, migrations, factors, symptoms, include, confirm, techniques, bitrot, strictly, purpose, consider, option, very, consuming, disabled, correctly, either, changes, inodes, inode, fast, medium, lz4, flags, algorithm, cubic, speedup, making, needs, works, clients, likely, improvement, lines, enabling, thanks, frédéric, sharing, tip, archlinux, debian, core, default_qdisc, ipv4, tcp_congestion_control, conf, apnic, bandwith, tcp, total, due, buffers, 100ms, 250, mbit, resolved, tuning, kernel, limits, coded, adjustments, calclate, product, congested, midnight, lead, hour, tends, enjoy, depending, watch, difficult, improve, uplink, tune, aware, trade, offs, maybe, initial, compressed, future, helps, understand, modern, tools, chunk, pieces, calculate, hashsums, compress, connected, 1gbit, located, professional, throttline, type, rarely, reports, encounter, find, verified, longer, borg_workarounds, ignore_invalid_archive_tam, upgrade, tam, trust, shipped, improved, verification, throw, created, release, depth, discussion, interruptions, stackexchange, 3988, 636, measure, takes, min, complete, suspect, certain, residential, come, tests, mechnism, script, adds, progress, preserved, mind, exit, code, encountered, warnings, checkpoint, mechanism, retry_wait, benefit, automatic, null, seconds, applied, clientaliveinterval, serveraliveinterval, serveralivecountmax, mostly, helpful, regular, ssh_config, frequent, keepalive, terminated, unable, solutions, client_loop, disconnect, remoterepository, received, messages, packet_write_wait, isps, period, inactivity, especially, nat, involved, terminate, print, potential, problems, end, worked, pty, allocation, request, channel, verbose, logging, chmod, important, trying, maximum, number, specify, above, maxauthtries, unlocked, suited, interactive, cases, increase, move, agent, differ, custom, name, id_, rsa, ed25519, ecdsa, keygen, id_ed25519, locally, show, sha256, opposed, almost, problem, double, following, further, denied, attempts, could, caused, scripts, blocked, solution, wait, aren, logins, refused, frequently, hosting, theme, jekyll, api, pricing, mount, browse, import, export, databases, truenas, synology, nas, containers, pika, line, strategy, home, copied, document, expand, menu, skip, content,
Text of the page (random words):
dy assigned a ssh key to the repo on borgbase com and is this the same key you are using locally borgbase will show the key s sha256 fingerprint in account ssh keys you can compare this to your local fingerprint like this ssh keygen lf ssh id_ed25519 is ssh using the key you assigned by default only ssh id_ rsa ed25519 ecdsa are used if you run borg from a different user account the ssh keys will differ too when using a custom key name you can add identityfile ssh id_custom to ssh config or to only use this key with borgbase host repo borgbase com identityfile ssh id_custom is the ssh key you are using encrypted encrypted keys need to be unlocked added to ssh agent before using them with vorta and you may get prompted for a password when using it with borg directly so not well suited for non interactive use for most use cases using an encrypted key doesn t increase security it s better to use one key per machine and not move it is ssh trying too many keys the maximum number of keys maxauthtries that can be tried per connection is 6 if you have more keys specify the key to use as shown above are the key permissions ok private keys and important config files if in use need to have a permission of 0600 to change the permission chmod 0600 ssh id_custom ssh config if you still get errors try to connect to your repo using the ssh command with verbose logging enabled ssh v xxxx xxxx repo borgbase com this will print a list of keys being tried and potential problems you won t get a shell at the end as borgbase only supports access via borg once you see remote key is restricted or pty allocation request failed on channel 0 then the login step still worked my ssh connection breaks after a long backup or prune operation if borg happens to be busy on the client or server side it may not send data over the ssh connection for a while in this case some isps will terminate the connection after a period of inactivity especially if a nat is involved you would then see an error like this remote packet_write_wait connection to xxx xxx xxx xxx broken pipe connection closed by remote host or remote client_loop send disconnect broken pipe remoterepository 2 61 kb bytes sent 1 01 mb bytes received 52 messages sent connection closed by remote host which means the ssh connection has been terminated and borg is unable to send data to the server side process some possible solutions are frequent keepalive packets mostly helpful for long borg prune or borg check operations running server side have the client send regular keep alive packets while no data is sent by borg on the client machine you can add the below configuration to ssh config or etc ssh ssh_config host repo borgbase com serveraliveinterval 10 serveralivecountmax 30 this configuration means that the client will send a null packet every 10 seconds to keep the connection alive if it doesn t get a response 30 times the connection will be closed borgbase already has the appropriate clientaliveinterval configuration applied server side enable automatic retries in borgmatic newer versions of borgmatic can automatically retry a failed command to benefit from this feature add this to your borgmatic config storage retries 5 retry_wait 5 use external retry mechanism you can also use your own retry mechnism in a shell script borg adds checkpoint archives every 30 minutes so your progress is preserved if a retry is needed keep in mind that exit code 1 means borg encountered warnings and you probably don t want to retry in this case but check the logs debug packet loss and unstable connections if you suspect routing issues or an unstable network connection certain residential internet connections come with restricted upload speed you can run the below network tests a mtr traceroute test to uncover packet loss or iperf3 for excessive retransmits traceroute to uncover routing issues and packet loss takes about 15 min to complete mtr s 1000 r c 1000 xxxxx repo borgbase com network performance test to uncover excessive retransmits and measure bandwidth contact support to enable this first iperf3 c xxxxx repo borgbase com for an in depth discussion on network interruptions also see borg issues 636 and 3988 or this and this stackexchange question i get a data integrity error when accessing a repository borg 1 2 6 shipped with improved archive verification and may throw this error when created with older borg versions full details in the release logs or this github issue if you trust the remote repository at a minimum you will need to run the below command borg_workarounds ignore_invalid_archive_tam borg upgrade archives tam repo_url after this all archives will be verified and you will no longer see this error why is my backup process so slow all our servers are connected with 1gbit connections at a minimum and located in professional data centers there is no throttline based on account type we rarely get reports of slow backup speeds if you do encounter slower than expected backups or slow upload speeds you can follow the steps below to find the bottleneck first it helps to understand that modern backup tools do more than just copy files they will also chunk them into smaller pieces calculate hashsums compress and encrypt each part see here for the full details so the upload speed is not always the main bottleneck depending on your setup you should also watch out for cpu usage or disk io it s usually difficult to improve uplink speed but if you are cpu or io limited there are a few settings you can tune just be aware of the trade offs maybe you are ok with a slower initial backup in order to have a smaller well compressed backup in the future so here some steps you can take to speed you backups general tips avoid congested times of the day many users set their backups to run at midnight utc which can lead to backups being slower at that time on the hour tends to be more busy too to enjoy the best speeds set your backups to run at random times high latency connection for backups using tcp which includes both borg and restic high latency can limit total bandwidth due to the way network buffers are set in linux e g for 100ms latency the speed would be limited to about 250 mbit s for restic this can be resolved by tuning the below linux kernel settings for borg the limits are hard coded in ssh and can t be changed easily if you want to make the adjustments for restic first calclate your connection s bandwidth delay product here and then see here on how to apply them use the bbr congestion algorithm instead of the default cubic users have reported a 6x speedup after making this change this feature needs to be enabled on the sending side and works best on low bandwith connections clients sending from data centers will likely not see an improvement as reported by apnic add the below lines to etc sysctl conf and apply with sysctl p net core default_qdisc fq net ipv4 tcp_congestion_control bbr more details on enabling bbr for network congestion for debian or archlinux thanks to our user frédéric for sharing this tip tips for borg see the official faq on steps to speed up backups using different borg flags make sure you choose an appropriate compression level for your data in general lz4 fast but low compression zstd 3 medium compression and zstd 8 high compression will work well ensure the local files cache is working correctly by default borg will compare ctime size inode and process the file if either changes if you have e g a network file system with unstable inodes try using files cache ctime size avoid excessive archive checking borg check can read all backup segments and confirm their consistency for large repos this can take a long time borgbase already uses different techniques to avoid bitrot in the storage backend so borg check is not strictly necessary for this purpose in borgmatic set checks to disabled in the consistency section if you still need consistency checks consider using the repository option to limit the check to the repository checking all archive metadata is done on the client and very time consuming see the official borg docs for details repair damaged repositories there are rare situations where borg repositories can be damaged by e g interrupted backups incomplete migrations hardware failures or other factors symptoms include object with key xxx not found in repository cache is newer than repository do you have multiple independently updated repos with same id borg is generally good at fixing those issues and missing data will be added during the next backup run missing data segments to fix errors related to missing data or files first run borg check repo_url this will first check the repository files and then one or more archives if the first step reveals errors you should make a local repo copy unless the data is non critical next run borg check repair repo_url to ask borg to fix any errors finally run borg create to do a full backup run and add potentially missing data cache issues to fix errors related to the cache from here first try to delete the local cache borg delete cache only repo_url if the first step doesn t resolve the issue try moving the security folder in config borg security repo_id where the repo_id can be found using borg config repo_url id run a repo check with borg check repo_url append only mode also called delayed deletion in this mode borg will never remove old segments and instead add a new transaction for any change in a transaction log the result is that no data is ever deleted and operations like archive prunes or deletions can be undone this is useful if the client machine shouldn t get full access to its own backups to e g prevent a hacker from deleting backups after taking over a client machine for full details and instruction on how to roll back see the official borg docs why can i still prune or delete archives with active append only mode the server side borg process doesn t know about the high level commands borg delete borg prune you run it only knows about adding chunks removing chunks and so on so with the current architecture it s not possible to reject e g a borg delete command right away as a result all high level operations still work but old data isn t deleted delayed deletion effectively this means that while running backups with append only ssh keys no disk space will be recovered in your borgbase repo with pruning but you can run a prune with an all access ssh key when your free quota is running low which will then clear pruned backups and free up disk space with append only mode enabled the repository will have a timestamped transaction log this allows going back to previous states even if prune or delete commands were issued by the backup client if you need to restore an older repo version you can use sftp mode to make the necessary edits or preferably download a backup copy of the whole repo before doing any edits why is borgbase showing the same storage use after deleting or prunging archives there are situations where removing an archive doesn t automatically delete the data associated with it this can have multiple reasons the same data segment is still associated with another archive e g if you have two archives referencing the same files and you delete one of them the data usage will stay the same you are using an append only key in this case the server side borg process will ignore all requests to actually delete any data and transactions can be rolled back see the whole append only section for more when using borg 1 1 x old data is cleaned up only during a write operation to the repository so if you changed key permissions recently there can be a delay before the data is cleaned up when using borg 1 2 x or higher old data is only deleted when running borg compact or running the compact repo action from the borgbase control panel before that no disk space is released how can i prune append only repositories when using append only mode old transactions and segments are never cleaned from the repo and the size can grow over time the timing and conditions of when old segments are cleaned up depends on the borg version in use borg 1 1 x in this version old segments are deleted implicitely when a write operation e g borg create borg delete is done with a full access key borg 1 2 x this version added a new borg compact command to explicitely clean up old segments at a suitable time you can trigger this command from our control panel under more compact repo from the repo table or run borg compact with a full access key directly other questions what s the difference between a ssh key ssh password ssh key passphrase and borg repository passphrase borgbase uses ssh keys for authentication this means to connect to a backup repository you need to upload the public part of a ssh keypair and then use the private part to make a connection the ssh key and the borg repository passphrase are totally separate ssh keys are only used for authentication to log into the borgbase repo and you can easily replace it in the control panel it s possible to create an encrypted ssh key that s protected with a passphrase we don t recommend this for backup purposes since it doesn t add much security and can interfere with automation it s preferred to generate a ssh key without passphrase and keep it on this one machine only if the same borg repository is used from another machine generate a new key there in addition ssh also supports login by password this doesn t work well with automation and isn t supported by borgbase if you are prompted for a ssh password there is an issue with your ssh key the borg repository passphrase is used for encryption and is separate from ssh it encrypts the backup data on your machine and we can t help you recover or replace it you should keep it somewhere safe off the source machine so you always have access to the backup even if something happens to the source machine to summarize the terms again ssh key the private part of a keypair used to authenticate with a ssh server used by borgbase for authentication when connecting to a borg repository ssh password another way to authenticate with a ssh server not used by borgbase ssh key passphrase to encrypt a ssh key borg repository passphrase to encrypt the data in a borg repository which storage backend are you using we deploy a mix of hardware and software raid 6 backed storage servers with nvme backed write caches for performance and enterprise hard drives for reliablity replacement drives are ready on site for quick failovers for a list of the providers we work with you can also see our gdpr page while this setup provides good durability against most kinds of hardware failures borgbase doesn t offer geographical redundancy or redundancy across multiple data centers for single repos it s thus not recommended to use our service for archiving purposes where there is no other copy of the data if you need geographic redundancy you can add two repos in different regions and do p...
|
