Meta tags:
Headings (most frequently used words):
pgbouncer, configuring, redis, for, mastodon, separate, sidekiq, web, to, scaling, up, your, server, managing, concurrency, transaction, pooling, with, cache, sentinel, high, availability, read, replicas, using, load, balancer, user, generated, content, retention, sponsored, by, puma, streaming, api, background, processing, why, you, might, need, installing, number, of, threads, queues, setting, password, userlist, txt, ini, debugging, that, it, all, works, talk, administering,
Text of the page (most frequently used words):
the (182), and (62), redis (59), you (58), mastodon (46), sidekiq (44), for (41), pgbouncer (41), your (38), #streaming (34), #server (32), can (32), this (29), that (29), with (28), instance (24), are (22), database (22), use (21), password (21), postgresql (20), will (18), admin (18), number (18), user (17), api (17), default (17), web (16), processes (16), connections (16), load (15), etc (15), using (15), set (14), more (14), threads (14), service (13), start (13), environment (13), process (13), also (12), how (12), not (11), but (11), different (11), replica (11), port (11), then (11), one (10), than (10), systemctl (10), from (9), following (9), tasks (9), new (9), users (8), setting (8), should (8), multiple (8), make (8), run (8), connect (8), would (8), same (8), separate (8), restart (8), queue (8), configuring (8), queues (8), sudo (8), md5 (8), values (7), background (7), puma (7), single (7), may (7), when (7), file (7), many (7), all (7), some (7), sentinel (7), used (7), which (7), var (7), transaction (7), just (7), running (7), retention (6), controls (6), cache (6), them (6), need (6), these (6), http (6), scaling (6), servers (6), data (6), have (6), based (6), their (6), master (6), requests (6), want (6), variables (6), here (6), 127 (6), log (6), systemd (6), first (6), userlist (6), txt (6), 4000 (6), available (5), health (5), traffic (5), configure (5), balancer (5), like (5), sure (5), loss (5), add (5), configuration (5), db_port (5), production (5), read (5), about (5), example (5), clients (5), specify (5), 6432 (5), 5432 (5), mastodon_production (5), pooling (5), other (5), processing (5), accounts (5), long (4), settings (4), under (4), content (4), both (4), monitor (4), jobs (4), replicas (4), name (4), local (4), host (4), db_host (4), url (4), prepared_statements (4), good (4), without (4), management (4), such (4), variable (4), has (4), guide (4), down (4), instances (4), current (4), line (4), lib (4), conf (4), memory (4), requires (4), installing (4), out (4), posts (4), each (4), nginx (4), metrics (4), profile (4), account (4), oauth (4), generated (3), own (3), media (3), scheduler (3), endpoints (3), monitoring (3), documentation (3), setup (3), config (3), yml (3), could (3), makara (3), see (3), db_user (3), db_name (3), role (3), true (3), false (3), edit (3), currently (3), talk (3), possible (3), however (3), well (3), list (3), high (3), ensure (3), change (3), copy (3), system (3), creating (3), individual (3), caching (3), implementing (3), components (3), application (3), important (3), reload (3), psql (3), bundle (3), exec (3), default_pool_size (3), defaults (3), pghero (3), connection (3), databases (3), ini (3), real (3), roles (3), might (3), through (3), time (3), command (3), handling (3), importance (3), note (3), pool (3), db_pool (3), per (3), worker (3), streaming_cluster_num (3), 4002 (3), 4001 (3), fail_timeout (3), endpoint (3), 000 (3), search (3), preferences (3), apps (3), reports (3), view (2), source (2), last (2), backups (2), remote (2), prevent (2), stored (2), review (2), task (2), via (2), does (2), cron (2), administration (2), checks (2), status (2), code (2), text (2), deployment (2), behind (2), virtual (2), help (2), provide (2), performance (2), consider (2), otherwise (2), network (2), public (2), address (2), failing (2), urls (2), point (2), correct (2), going (2), primary (2), db_password (2), follows (2), writes (2), while (2), issue (2), impact (2), versions (2), before (2), ways (2), optional (2), value (2), replica_prepared_statements (2), replica_db_tasks (2), built (2), support (2), starting (2), queries (2), only (2), rails (2), app (2), sidekiq_ (2), critical (2), goes (2), track (2), node (2), after (2), configured (2), availability (2), still (2), working (2), 6479 (2), sidekiq_redis_url (2), enable (2), now (2), create (2), directory (2), pidfile (2), pid (2), making (2), simple (2), parts (2), unspecified (2), prefix (2), its (2), disk (2), context (2), additionally (2), maximum (2), limit (2), key (2), capacity (2), handle (2), allocate (2), specifically (2), define (2), uses (2), others (2), feeds (2), lose (2), though (2), never (2), quit (2), work (2), rails_env (2), migrate (2), next (2), instead (2), since (2), logs (2), able (2), max_client_conn (2), 100 (2), fine (2), later (2), accept (2), correspond (2), pool_mode (2), mode (2), session (2), therefore (2), auth_type (2), assuming (2), listen_port (2), listen_addr (2), username (2), md5d75bb2be2d7086c6148944261a00f605 (2), passwordmastodon (2), md5sum (2), ubuntu (2), debian (2), where (2), query (2), stats (2), permissions (2), equal (2), total (2), find (2), common (2), why (2), execution (2), services (2), experience (2), lower (2), priority (2), delivery (2), mails (2), push (2), affect (2), varying (2), much (2), affects (2), sent (2), request (2), those (2), made (2), cpu (2), resources (2), consume (2), started (2), balance (2), three (2), prometheus (2), messages (2), channels (2), connected (2), client (2), websocket (2), throughput (2), streaming_api_base_url (2), lived (2), control (2), share (2), larger (2), maxes (2), managing (2), concurrency (2), notificationpolicy (2), filter (2), report (2), quote (2), domainblock (2), notifications (2), trends (2), collections (2), domain_blocks (2), tokens (2), security (2), machine (2), features (2), imprint, blog, join, sponsored, updated, november, 2025, 简体中文, improve, page, archive, period, kept, cached, runaway, growth, assets, appropriate, cleanup, runs, job, manual, triggering, third, party, alerting, utility, return, 200, result, purpose, provision, balancers, consistent, reducing, risk, become, overwhelmed, decrease, downtime, performing, maintenance, upgrades, consult, provider, backend, nodes, send, responsive, cloud, providers, digitalocean, aws, hetzner, offer, balancing, solutions, distribute, across, stock, avoid, locations, locally, installed, two, mastodon_replica, above, there, possibilities, information, slave, blacklist_duration, sticky, postgres, postgresql_makara, adapter, replace, section, supported, lead, driver, reads, let, straight, querying, very, often, anyway, small, once, done, seeing, against, override, any, schema, migrations, seeds, alternatively, replica_database_url, replica_db_pass, replica_db_user, replica_db_name, replica_db_port, replica_db_host, version, every, included, directed, lagging, few, seconds, stop, sending, until, catches, reduce, wish, hot, replication, learn, https, docs, latest, operate, oss_and_stack, related, prefixed, case, cache_, mentioned, part, operation, bring, entire, alleviate, automatically, direct, comma, delimited, combinations, sentinels, determine, select, minute, being, unreachable, redis_sentinel_master, redis_sentinels, wrong, update, chown, mkdir, dir, logfile, alias, readwritepaths, usr, bin, supervised, daemonize, execstart, relatively, fallback, sidekiq_redis_port, sidekiq_redis_host, keep, locks, although, general, gain, big, benefit, having, disable, saving, save, eviction, policy, details, configurations, refer, lru, volatile, concerned, cache_redis_port, cache_redis_host, cache_redis_url, widely, throughout, home, backed, wouldn, even, survived, unlike, easiest, way, reboot, administering, cannot, perform, easy, around, defining, together, calling, too, db_pass, localhost, prepared, statements, off, env, tail, check, debugging, works, don, forget, changes, always, increase, defines, itself, puts, opened, hood, reported, because, knowledge, needs, means, established, terminated, upon, completion, essential, admin_users, put, format, tell, dbname, listing, underlying, cases, md5a45753afaca0db833a6f7c7b2864b9d9, sample, beginning, macos, openbsd, echo, scheme, string, prepended, derive, hash, arbitrarily, passwords, reuse, credentials, simplicity, sake, alter, obviously, word, reset, firstly, apt, install, extensions, usage, link, generally, open, combined, devops, solution, document, describes, gotchas, necessary, efficiently, truly, parallel, arguments, checking, finds, none, subsequent, overfilled, delays, priorities, overridden, invocation, refreshing, trending, hashtags, cleaning, imports, resolving, deleting, forwarding, replies, pull, mailers, incoming, activities, ingress, payloads, defined, wasn, listed, order, descending, initiate, thread, large, size, managed, least, regulated, rather, argument, invoking, shown, responsiveness, end, allocated, quickly, delivered, author, anyone, else, soon, delegated, fast, aborts, affecting, configurable, handled, interacted, planning, difficult, especially, comes, cpus, cause, contention, software, another, was, misconfiguring, exhaust, opening, cluster, potentially, previous, had, clustering, starts, equivalent, distributed, files, commands, least_conn, upstream, route, consumed, likely, documented, below, routed, arbitrary, less, insightful, tracking, received, pubsub, intended, complement, directly, mastodon_streaming_redis_messages_received_total, mastodon_streaming_messages_sent_total, subscribed, higher, due, our, internal, mastodon_streaming_connected_channels, tagged, type, eventsource, mastodon_streaming_connected_clients, exposes, lot, understand, reasonably, isn, community, monthly, active, typically, average, concurrent, 800, 1200, subdomain, allows, applications, correctly, hosted, shared, listen, socket, bind, base, handles, receive, updates, terms, better, served, parent, they, write, ram, max_threads, web_concurrency, serves, short, most, types, webpushsubscription, translation, token, termsofservice, tag, suggestion, statussource, statusedit, shallowtag, shallowquote, scheduledstatus, rule, relationshipseveranceevent, relationship, reaction, quoteapproval, privacypolicy, previewcardauthor, previewcard, poll, notificationrequest, notificationfallback, notification, mediaattachment, marker, identityproof, filterstatus, filterresult, filterkeyword, featuredtag, featureapproval, familiarfollowers, extendeddescription, error, customemoji, conversation, collectionwithaccounts, collectionitem, collection, asyncrefresh, appeal, annualreport, announcement, measure, ipblock, emaildomainblock, domainallow, dimension, cohort, canonicalemailblock, accountwarning, entities, markers, lists, conversations, timelines, scheduled_statuses, polls, statuses, proofs, oembed, custom_emojis, announcements, grouped, async_refreshes, emails, annual, measures, ip_blocks, email_domain_blocks, domain_allows, dimensions, canonical_email_blocks, tags, suggestions, mutes, followed_tags, follow_requests, filters, featured_tags, favourites, endorsements, bookmarks, blocks, methods, rate, limits, scopes, guidelines, best, practices, datetime, formats, rest, bearcaps, microformats, webfinger, activitypub, spec, compliance, design, reference, themes, css, styling, state, frontend, issues, responsible, disclosure, routes, structure, dev, technical, overview, contributing, libraries, implementations, logging, obtaining, access, playing, getting, developing, webhooks, index, corruption, troubleshooting, errors, moderation, actions, migrating, backing, upgrading, release, cli, sign, captcha, onion, object, storage, full, preparing, official, ios, android, moving, leaving, externally, promoting, yourself, dealing, unwanted, quoting, posting, signing, what,
Text of the page (random words):
tance where importance is defined by how much it would impact the user experience of your server s local users if the queue wasn t working the queues are listed here in order of descending importance default all tasks that affect local users push delivery of payloads to other servers ingress incoming remote activities lower priority than the default queue so that local users still see their posts when the server is under load mailers delivery of e mails pull lower priority tasks such as handling imports backups resolving threads deleting users forwarding replies scheduler handling cron jobs such as refreshing trending hashtags and cleaning up logs the default queues and their priorities are stored in config sidekiq yml but can be overridden by the command line invocation of sidekiq e g bundle exec sidekiq q default this command will run just the default queue sidekiq processes queues by first checking for tasks in the first queue and if it finds none it then checks the subsequent queue therefore if the first queue is overfilled tasks in the other queues may experience delays it is possible to start different sidekiq processes for the queues to ensure truly parallel execution by e g creating multiple systemd services for sidekiq with different arguments you may run as many sidekiq processes with as many threads as necessary to efficiently process running jobs however the scheduler queue should never be run in more than one sidekiq process at a time transaction pooling with pgbouncer why you might need pgbouncer if you start running out of available postgresql connections the default is 100 then you may find pgbouncer to be a good solution this document describes some common gotchas as well as good configuration defaults for mastodon user roles with devops permissions in mastodon can monitor the current usage of postgresql connections through the pghero link in the administration view generally the number of connections open is equal to the total threads in puma sidekiq and the streaming api combined to enable query stats in pghero review the query stats guide and make sure your postgresql has the correct permissions and extensions configured installing pgbouncer on debian and ubuntu sudo apt install pgbouncer configuring pgbouncer setting a password firstly if your mastodon user in postgresql is set up without a password you will need to set a password here s how you might reset the password psql p 5432 u mastodon mastodon_production w then obviously use a different password than the word password alter user mastodon with password password then q to quit configuring userlist txt edit etc pgbouncer userlist txt as long as you specify a user password in pgbouncer ini later the values in userlist txt do not have to correspond to real postgresql roles you can arbitrarily define users and passwords but you can reuse the real credentials for simplicity s sake add the mastodon user to the userlist txt mastodon md5d75bb2be2d7086c6148944261a00f605 here we re using the md5 scheme where the md5 password is just the md5sum of password username with the string md5 prepended for instance to derive the hash for user mastodon with password password ubuntu debian etc echo n passwordmastodon md5sum macos openbsd etc md5 s passwordmastodon then just add md5 to the beginning of that you ll also want to create a pgbouncer admin user to log in to the pgbouncer admin database so here s a sample userlist txt mastodon md5d75bb2be2d7086c6148944261a00f605 pgbouncer md5a45753afaca0db833a6f7c7b2864b9d9 in both cases the password is just password configuring pgbouncer ini edit etc pgbouncer pgbouncer ini add a line under databases listing the postgresql databases you want to connect to here we ll just have pgbouncer use the same username password and database name to connect to the underlying postgresql database databases mastodon_production host 127 0 0 1 port 5432 dbname mastodon_production user mastodon password password the listen_addr and listen_port tell pgbouncer which address port to accept connections the defaults are fine listen_addr 127 0 0 1 listen_port 6432 put md5 as the auth_type assuming you re using the md5 format in userlist txt auth_type md5 make sure the pgbouncer user is an admin admin_users pgbouncer mastodon requires a different pooling mode than the default session based one specifically it needs a transaction based pooling mode this means that a postgresql connection is established at the start of a transaction and terminated upon its completion therefore it is essential to change the pool_mode setting from session to transaction pool_mode transaction next up max_client_conn defines how many connections pgbouncer itself will accept and default_pool_size puts a limit on how many postgresql connections will be opened under the hood in pghero the number of connections reported will correspond to default_pool_size because it has no knowledge of pgbouncer the defaults are fine to start and you can always increase them later max_client_conn 100 default_pool_size 20 don t forget to reload or restart pgbouncer after making your changes sudo systemctl reload pgbouncer debugging that it all works you should be able to connect to pgbouncer just like you would with postgresql psql p 6432 u mastodon mastodon_production then use your password to log in you can also check the pgbouncer logs like so tail f var log postgresql pgbouncer log configuring mastodon to talk to pgbouncer in your env production file first off make sure that this is set prepared_statements false since we re using transaction based pooling we can t use prepared statements next up configure mastodon to use port 6432 pgbouncer instead of 5432 postgresql and you should be good to go db_host localhost db_user mastodon db_name mastodon_production db_pass password db_port 6432 you cannot use pgbouncer to perform db migrate tasks but this is easy to work around if your postgresql and pgbouncer are on the same host it can be as simple as defining db_port 5432 together with rails_env production when calling the task for example rails_env production db_port 5432 bundle exec rails db migrate you can specify db_host too if it s different etc administering pgbouncer the easiest way to reboot is sudo systemctl restart pgbouncer but if you ve set up a pgbouncer admin user you can also connect as the admin psql p 6432 u pgbouncer pgbouncer and then do reload then use q to quit separate redis for cache redis is used widely throughout the application but some uses are more important than others home feeds list feeds and sidekiq queues as well as the streaming api are backed by redis and that s important data you wouldn t want to lose even though the loss can be survived unlike the loss of the postgresql database never lose that redis is also used for volatile caching if you are scaling up and you are concerned about redis s capacity to handle the load you can allocate a separate redis database specifically for caching to do this set cache_redis_url in the environment or define individual components such as cache_redis_host cache_redis_port etc unspecified components will default to their values without the cache prefix when configuring the redis database for caching it is possible to disable background saving to disk as data loss on restart is not critical in this context and this can save some disk i o additionally consider setting a maximum memory limit and implementing a key eviction policy for more details on these configurations refer to this guide using redis as an lru cache separate redis for sidekiq redis is used in sidekiq to keep track of its locks and queue although in general the performance gain is not that big some instances may benefit from having a separate redis instance for sidekiq in the environment file you can specify sidekiq_redis_url or individual parts like sidekiq_redis_host sidekiq_redis_port etc unspecified parts fallback to the same values as without the sidekiq_ prefix creating a separate redis instance for sidekiq is relatively simple start by making a copy of the default redis systemd service cp etc systemd system redis service etc systemd system redis sidekiq service in the redis sidekiq service file change the following values execstart usr bin redis server etc redis redis sidekiq conf supervised systemd daemonize no pidfile run redis redis server sidekiq pid readwritepaths var lib redis sidekiq alias redis sidekiq service make a copy of the redis configuration file for the new sidekiq redis instance cp etc redis redis conf etc redis redis sidekiq conf in this redis sidekiq conf file change the following values port 6479 pidfile var run redis redis server sidekiq pid logfile var log redis redis server sidekiq log dir var lib redis sidekiq before starting the new redis instance create a data directory mkdir var lib redis sidekiq chown redis var lib redis sidekiq start the new redis instance systemctl enable now redis sidekiq update your environment and add the following line sidekiq_redis_url redis 127 0 0 1 6479 restart mastodon to use the new redis instance ensure that you restart both web and sidekiq otherwise one of them will still be working from the wrong instance systemctl restart mastodon web service systemctl restart redis sidekiq service redis sentinel for high availability as mentioned redis is a critical part of a mastodon instance s operation by default your deployment will use a single redis instance or multiple if you ve setup a cache however if that instance goes down it can bring the entire mastodon instance down as well to alleviate this redis sentinel can be used to track your redis instances and automatically direct clients to a new primary if one goes down you can specify redis_sentinels which is a comma delimited list of the ip port combinations of sentinels that mastodon can talk with to determine the current master redis node you also need to specify the name of the master you would like to connect to in redis_sentinel_master by default sentinel will set an instance as down and select a new master after a minute of the current master being unreachable but this can be configured based on your setup all sentinel related variables are also available prefixed with cache_ and sidekiq_ in case you use more than one redis instance learn more about redis sentinel here https redis io docs latest operate oss_and_stack management sentinel read replicas to reduce the load on your postgresql server you may wish to set up hot streaming replication read replica see this guide for an example mastodon 4 2 mastodon has built in replica support starting with version 4 2 you can use the same configuration for every service sidekiq included and some queries will be directed to your read only replica when possible using rails s built in replica support if your replica is lagging behind for more than a few seconds then the app will stop sending it queries until it catches up to configure it use the following environment variables replica_db_host replica_db_port replica_db_name replica_db_user replica_db_pass replica_prepared_statements replica_db_tasks alternatively you can also use replica_database_url if you want to configure them all using the same variable replica_db_tasks false will connect to an replica database without any database management tasks such as schema management migrations seeds etc by default it is set to true replica_prepared_statements is an optional override for the prepared_statements value by default it is set to true if prepared_statements is not set once done this is all good and you should start seeing requests against your replica server mastodon 4 1 for mastodon versions before 4 2 you can make use of the replica in mastodon in these ways the streaming api server does not issue writes at all so you can connect it straight to the replica it is not querying the database very often anyway so the impact of this is small use the makara driver in the web and sidekiq processes so that writes go to the master database while reads go to the replica let s talk about that read replicas are currently not supported for the sidekiq processes and using them will lead to failing jobs and data loss you will have to use a separate config database yml file for the web processes and edit it to replace the production section as follows production default adapter postgresql_makara prepared_statements false makara id postgres sticky true connections role master blacklist_duration 0 url postgresql db_user db_password db_host db_port db_name role slave url postgresql db_user db_password db_host db_port db_name make sure that the urls point to the correct locations for your postgresql servers you can add multiple replicas you could have a locally installed pgbouncer with a configuration to connect to two different servers based on the database name e g mastodon going to the primary mastodon_replica going to the replica so in the file above both urls would point to the local pgbouncer with the same user password host and port but different database name there are many possibilities for how this could be set up for more information on makara see their documentation make sure the sidekiq processes run with the stock config database yml to avoid failing jobs and data loss using a web load balancer cloud providers like digitalocean aws hetzner etc offer virtual load balancing solutions that distribute network traffic across multiple servers but provide a single public ip address scaling your deployment to provision multiple web puma servers behind one of these virtual load balancers can help provide more consistent performance by reducing the risk that a single server may become overwhelmed by user traffic and decrease downtime when performing maintenance or upgrades you should consult your provider documentation on how to setup and configure a load balancer but consider that you need to configure your load balancer to monitor the health of the backend web puma nodes otherwise you may send traffic to a service that is not responsive the following endpoints are available to monitor for this purpose web puma health streaming api api v1 streaming health these endpoints should both return an http status code of 200 and the text ok as a result you can also use these endpoints for health checks with a third party monitoring alerting utility user generated content retention to prevent runaway growth of stored assets review the settings under administration content retention and set them to values appropriate for your server and users the cleanup task runs via the background job scheduler and does not need manual triggering or cron settings the media cache retention setting controls how long media from remote users is cached on your server the user archive retention period controls how long backups generated by your own users are kept last updated november 23 2025 improve this page also available in 简体中文 sponsored by join mastodon blog view source cc by ...
|
