Meta tags:
Headings (most frequently used words):
pgbouncer, configuring, redis, for, mastodon, separate, sidekiq, web, to, scaling, up, your, server, managing, concurrency, transaction, pooling, with, cache, sentinel, high, availability, read, replicas, using, load, balancer, user, generated, content, retention, sponsored, by, puma, streaming, api, background, processing, why, you, might, need, installing, number, of, threads, queues, setting, password, userlist, txt, ini, debugging, that, it, all, works, talk, administering,
Text of the page (most frequently used words):
the (182), and (62), redis (59), you (58), mastodon (46), #sidekiq (44), for (41), pgbouncer (41), your (38), streaming (34), #server (32), can (32), this (29), that (29), with (28), instance (24), are (22), database (22), use (21), password (21), postgresql (20), will (18), admin (18), number (18), user (17), api (17), default (17), web (16), processes (16), connections (16), load (15), etc (15), using (15), set (14), more (14), threads (14), service (13), start (13), environment (13), process (13), also (12), how (12), not (11), but (11), different (11), replica (11), port (11), then (11), one (10), than (10), systemctl (10), from (9), following (9), tasks (9), new (9), users (8), setting (8), should (8), multiple (8), make (8), run (8), connect (8), would (8), same (8), separate (8), restart (8), queue (8), configuring (8), queues (8), sudo (8), md5 (8), values (7), background (7), puma (7), single (7), may (7), when (7), file (7), many (7), all (7), some (7), sentinel (7), used (7), which (7), var (7), transaction (7), just (7), running (7), retention (6), controls (6), cache (6), them (6), need (6), these (6), http (6), scaling (6), servers (6), data (6), have (6), based (6), their (6), master (6), requests (6), want (6), variables (6), here (6), 127 (6), log (6), systemd (6), first (6), userlist (6), txt (6), 4000 (6), available (5), health (5), traffic (5), configure (5), balancer (5), like (5), sure (5), loss (5), add (5), configuration (5), db_port (5), production (5), read (5), about (5), example (5), clients (5), specify (5), 6432 (5), 5432 (5), mastodon_production (5), pooling (5), other (5), processing (5), accounts (5), long (4), settings (4), under (4), content (4), both (4), monitor (4), jobs (4), replicas (4), name (4), local (4), host (4), db_host (4), url (4), prepared_statements (4), good (4), without (4), management (4), such (4), variable (4), has (4), guide (4), down (4), instances (4), current (4), line (4), lib (4), conf (4), memory (4), requires (4), installing (4), out (4), posts (4), each (4), nginx (4), metrics (4), profile (4), account (4), oauth (4), generated (3), own (3), media (3), scheduler (3), endpoints (3), monitoring (3), documentation (3), setup (3), config (3), yml (3), could (3), makara (3), see (3), db_user (3), db_name (3), role (3), true (3), false (3), edit (3), currently (3), talk (3), possible (3), however (3), well (3), list (3), high (3), ensure (3), change (3), copy (3), system (3), creating (3), individual (3), caching (3), implementing (3), components (3), application (3), important (3), reload (3), psql (3), bundle (3), exec (3), default_pool_size (3), defaults (3), pghero (3), connection (3), databases (3), ini (3), real (3), roles (3), might (3), through (3), time (3), command (3), handling (3), importance (3), note (3), pool (3), db_pool (3), per (3), worker (3), streaming_cluster_num (3), 4002 (3), 4001 (3), fail_timeout (3), endpoint (3), 000 (3), search (3), preferences (3), apps (3), reports (3), view (2), source (2), last (2), backups (2), remote (2), prevent (2), stored (2), review (2), task (2), via (2), does (2), cron (2), administration (2), checks (2), status (2), code (2), text (2), deployment (2), behind (2), virtual (2), help (2), provide (2), performance (2), consider (2), otherwise (2), network (2), public (2), address (2), failing (2), urls (2), point (2), correct (2), going (2), primary (2), db_password (2), follows (2), writes (2), while (2), issue (2), impact (2), versions (2), before (2), ways (2), optional (2), value (2), replica_prepared_statements (2), replica_db_tasks (2), built (2), support (2), starting (2), queries (2), only (2), rails (2), app (2), sidekiq_ (2), critical (2), goes (2), track (2), node (2), after (2), configured (2), availability (2), still (2), working (2), 6479 (2), sidekiq_redis_url (2), enable (2), now (2), create (2), directory (2), pidfile (2), pid (2), making (2), simple (2), parts (2), unspecified (2), prefix (2), its (2), disk (2), context (2), additionally (2), maximum (2), limit (2), key (2), capacity (2), handle (2), allocate (2), specifically (2), define (2), uses (2), others (2), feeds (2), lose (2), though (2), never (2), quit (2), work (2), rails_env (2), migrate (2), next (2), instead (2), since (2), logs (2), able (2), max_client_conn (2), 100 (2), fine (2), later (2), accept (2), correspond (2), pool_mode (2), mode (2), session (2), therefore (2), auth_type (2), assuming (2), listen_port (2), listen_addr (2), username (2), md5d75bb2be2d7086c6148944261a00f605 (2), passwordmastodon (2), md5sum (2), ubuntu (2), debian (2), where (2), query (2), stats (2), permissions (2), equal (2), total (2), find (2), common (2), why (2), execution (2), services (2), experience (2), lower (2), priority (2), delivery (2), mails (2), push (2), affect (2), varying (2), much (2), affects (2), sent (2), request (2), those (2), made (2), cpu (2), resources (2), consume (2), started (2), balance (2), three (2), prometheus (2), messages (2), channels (2), connected (2), client (2), websocket (2), throughput (2), streaming_api_base_url (2), lived (2), control (2), share (2), larger (2), maxes (2), managing (2), concurrency (2), notificationpolicy (2), filter (2), report (2), quote (2), domainblock (2), notifications (2), trends (2), collections (2), domain_blocks (2), tokens (2), security (2), machine (2), features (2), imprint, blog, join, sponsored, updated, november, 2025, 简体中文, improve, page, archive, period, kept, cached, runaway, growth, assets, appropriate, cleanup, runs, job, manual, triggering, third, party, alerting, utility, return, 200, result, purpose, provision, balancers, consistent, reducing, risk, become, overwhelmed, decrease, downtime, performing, maintenance, upgrades, consult, provider, backend, nodes, send, responsive, cloud, providers, digitalocean, aws, hetzner, offer, balancing, solutions, distribute, across, stock, avoid, locations, locally, installed, two, mastodon_replica, above, there, possibilities, information, slave, blacklist_duration, sticky, postgres, postgresql_makara, adapter, replace, section, supported, lead, driver, reads, let, straight, querying, very, often, anyway, small, once, done, seeing, against, override, any, schema, migrations, seeds, alternatively, replica_database_url, replica_db_pass, replica_db_user, replica_db_name, replica_db_port, replica_db_host, version, every, included, directed, lagging, few, seconds, stop, sending, until, catches, reduce, wish, hot, replication, learn, https, docs, latest, operate, oss_and_stack, related, prefixed, case, cache_, mentioned, part, operation, bring, entire, alleviate, automatically, direct, comma, delimited, combinations, sentinels, determine, select, minute, being, unreachable, redis_sentinel_master, redis_sentinels, wrong, update, chown, mkdir, dir, logfile, alias, readwritepaths, usr, bin, supervised, daemonize, execstart, relatively, fallback, sidekiq_redis_port, sidekiq_redis_host, keep, locks, although, general, gain, big, benefit, having, disable, saving, save, eviction, policy, details, configurations, refer, lru, volatile, concerned, cache_redis_port, cache_redis_host, cache_redis_url, widely, throughout, home, backed, wouldn, even, survived, unlike, easiest, way, reboot, administering, cannot, perform, easy, around, defining, together, calling, too, db_pass, localhost, prepared, statements, off, env, tail, check, debugging, works, don, forget, changes, always, increase, defines, itself, puts, opened, hood, reported, because, knowledge, needs, means, established, terminated, upon, completion, essential, admin_users, put, format, tell, dbname, listing, underlying, cases, md5a45753afaca0db833a6f7c7b2864b9d9, sample, beginning, macos, openbsd, echo, scheme, string, prepended, derive, hash, arbitrarily, passwords, reuse, credentials, simplicity, sake, alter, obviously, word, reset, firstly, apt, install, extensions, usage, link, generally, open, combined, devops, solution, document, describes, gotchas, necessary, efficiently, truly, parallel, arguments, checking, finds, none, subsequent, overfilled, delays, priorities, overridden, invocation, refreshing, trending, hashtags, cleaning, imports, resolving, deleting, forwarding, replies, pull, mailers, incoming, activities, ingress, payloads, defined, wasn, listed, order, descending, initiate, thread, large, size, managed, least, regulated, rather, argument, invoking, shown, responsiveness, end, allocated, quickly, delivered, author, anyone, else, soon, delegated, fast, aborts, affecting, configurable, handled, interacted, planning, difficult, especially, comes, cpus, cause, contention, software, another, was, misconfiguring, exhaust, opening, cluster, potentially, previous, had, clustering, starts, equivalent, distributed, files, commands, least_conn, upstream, route, consumed, likely, documented, below, routed, arbitrary, less, insightful, tracking, received, pubsub, intended, complement, directly, mastodon_streaming_redis_messages_received_total, mastodon_streaming_messages_sent_total, subscribed, higher, due, our, internal, mastodon_streaming_connected_channels, tagged, type, eventsource, mastodon_streaming_connected_clients, exposes, lot, understand, reasonably, isn, community, monthly, active, typically, average, concurrent, 800, 1200, subdomain, allows, applications, correctly, hosted, shared, listen, socket, bind, base, handles, receive, updates, terms, better, served, parent, they, write, ram, max_threads, web_concurrency, serves, short, most, types, webpushsubscription, translation, token, termsofservice, tag, suggestion, statussource, statusedit, shallowtag, shallowquote, scheduledstatus, rule, relationshipseveranceevent, relationship, reaction, quoteapproval, privacypolicy, previewcardauthor, previewcard, poll, notificationrequest, notificationfallback, notification, mediaattachment, marker, identityproof, filterstatus, filterresult, filterkeyword, featuredtag, featureapproval, familiarfollowers, extendeddescription, error, customemoji, conversation, collectionwithaccounts, collectionitem, collection, asyncrefresh, appeal, annualreport, announcement, measure, ipblock, emaildomainblock, domainallow, dimension, cohort, canonicalemailblock, accountwarning, entities, markers, lists, conversations, timelines, scheduled_statuses, polls, statuses, proofs, oembed, custom_emojis, announcements, grouped, async_refreshes, emails, annual, measures, ip_blocks, email_domain_blocks, domain_allows, dimensions, canonical_email_blocks, tags, suggestions, mutes, followed_tags, follow_requests, filters, featured_tags, favourites, endorsements, bookmarks, blocks, methods, rate, limits, scopes, guidelines, best, practices, datetime, formats, rest, bearcaps, microformats, webfinger, activitypub, spec, compliance, design, reference, themes, css, styling, state, frontend, issues, responsible, disclosure, routes, structure, dev, technical, overview, contributing, libraries, implementations, logging, obtaining, access, playing, getting, developing, webhooks, index, corruption, troubleshooting, errors, moderation, actions, migrating, backing, upgrading, release, cli, sign, captcha, onion, object, storage, full, preparing, official, ios, android, moving, leaving, externally, promoting, yourself, dealing, unwanted, quoting, posting, signing, what,
Text of the page (random words):
time updates the following environment variables control it streaming_api_base_url controls the base url of the streaming api port controls the port the streaming server will listen on by default 4000 the bind and socket environment variables are also able to be used additionally the shared database and redis environment variables are used the streaming api can use a different subdomain if you want to by setting streaming_api_base_url this allows you to have one load balancer for streaming and one for web api requests however this also requires applications to correctly request the streaming url from the instance endpoint instead of assuming that it s hosted on the same host as the web api one process of the streaming server can handle a reasonably high number of connections and throughput but if you find that a single process isn t handling your instance s load you can run multiple processes by varying the port number of each and then using nginx to load balance traffic to each of those instances for example a community of about 50 000 accounts with 10 000 20 000 monthly active accounts you ll typically have an average concurrent load of about 800 1200 streaming connections the streaming server also exposes a prometheus endpoint on metrics with a lot of metrics to help you understand the current load on your mastodon streaming server some key metrics are mastodon_streaming_connected_clients this is the number of connected clients tagged by client type websocket or eventsource mastodon_streaming_connected_channels this is the number of channels that are currently subscribed note that this is much higher than connected clients due to how our internal system channels currently work mastodon_streaming_messages_sent_total this is the total number of messages sent to clients since last restart mastodon_streaming_redis_messages_received_total this is the number of messages received from redis pubsub and intended to complement monitoring redis directly note that for prometheus based monitoring you should monitor each individual streaming server instance not the nginx endpoint as requests to nginx will be routed to an arbitrary streaming server instance and make the metrics less insightful than the per instance tracking the more streaming server processes that you run the more database connections will be consumed on postgresql so you ll likely want to use pgbouncer as documented below an example nginx configuration to route traffic to three different processes on port 4000 4001 and 4002 is as follows upstream streaming least_conn server 127 0 0 1 4000 fail_timeout 0 server 127 0 0 1 4001 fail_timeout 0 server 127 0 0 1 4002 fail_timeout 0 if you re using the distributed systemd files then you can start up multiple streaming servers with the following commands sudo systemctl start mastodon streaming 4000 service sudo systemctl start mastodon streaming 4001 service sudo systemctl start mastodon streaming 4002 service by default sudo systemctl start mastodon streaming starts just one process on port 4000 equivalent to running sudo systemctl start mastodon streaming 4000 service previous versions of mastodon had a streaming_cluster_num environment variable that made the streaming server use clustering which started multiple worker processes and used node js to load balance them this interacted with the other settings in ways which made capacity planning difficult especially when it comes to database connections and cpu resources by default the streaming server would consume resources on all available cpus which could cause contention with other software running on that server another common issue was that misconfiguring the streaming_cluster_num would exhaust your database connections by opening up a connection pool per cluster worker process so a streaming_cluster_num of 5 and db_pool of 10 would potentially consume 50 database connections now a single streaming server process will only use at maximum db_pool postgresql connections and scaling is handled by running more instances of the streaming server background processing sidekiq many tasks in mastodon are delegated to background processing to ensure the http requests are fast and to prevent http request aborts from affecting the execution of those tasks sidekiq is a single process with a configurable number of threads number of threads while the number of threads in the web process affects the responsiveness of the mastodon instance to the end user the number of threads allocated to background processing affects how quickly posts can be delivered from the author to anyone else how soon e mails are sent out etc the number of threads is not regulated by an environment variable but rather through a command line argument when invoking sidekiq as shown in the following example bundle exec sidekiq c 15 this would initiate the sidekiq process with 15 threads it is important to note that each thread requires a database connection so this requires a large database pool the size of this pool is managed by the db_pool environment variable which should be set to a value at least equal to the number of threads queues sidekiq uses different queues for tasks of varying importance where importance is defined by how much it would impact the user experience of your server s local users if the queue wasn t working the queues are listed here in order of descending importance default all tasks that affect local users push delivery of payloads to other servers ingress incoming remote activities lower priority than the default queue so that local users still see their posts when the server is under load mailers delivery of e mails pull lower priority tasks such as handling imports backups resolving threads deleting users forwarding replies scheduler handling cron jobs such as refreshing trending hashtags and cleaning up logs the default queues and their priorities are stored in config sidekiq yml but can be overridden by the command line invocation of sidekiq e g bundle exec sidekiq q default this command will run just the default queue sidekiq processes queues by first checking for tasks in the first queue and if it finds none it then checks the subsequent queue therefore if the first queue is overfilled tasks in the other queues may experience delays it is possible to start different sidekiq processes for the queues to ensure truly parallel execution by e g creating multiple systemd services for sidekiq with different arguments you may run as many sidekiq processes with as many threads as necessary to efficiently process running jobs however the scheduler queue should never be run in more than one sidekiq process at a time transaction pooling with pgbouncer why you might need pgbouncer if you start running out of available postgresql connections the default is 100 then you may find pgbouncer to be a good solution this document describes some common gotchas as well as good configuration defaults for mastodon user roles with devops permissions in mastodon can monitor the current usage of postgresql connections through the pghero link in the administration view generally the number of connections open is equal to the total threads in puma sidekiq and the streaming api combined to enable query stats in pghero review the query stats guide and make sure your postgresql has the correct permissions and extensions configured installing pgbouncer on debian and ubuntu sudo apt install pgbouncer configuring pgbouncer setting a password firstly if your mastodon user in postgresql is set up without a password you will need to set a password here s how you might reset the password psql p 5432 u mastodon mastodon_production w then obviously use a different password than the word password alter user mastodon with password password then q to quit configuring userlist txt edit etc pgbouncer userlist txt as long as you specify a user password in pgbouncer ini later the values in userlist txt do not have to correspond to real postgresql roles you can arbitrarily define users and passwords but you can reuse the real credentials for simplicity s sake add the mastodon user to the userlist txt mastodon md5d75bb2be2d7086c6148944261a00f605 here we re using the md5 scheme where the md5 password is just the md5sum of password username with the string md5 prepended for instance to derive the hash for user mastodon with password password ubuntu debian etc echo n passwordmastodon md5sum macos openbsd etc md5 s passwordmastodon then just add md5 to the beginning of that you ll also want to create a pgbouncer admin user to log in to the pgbouncer admin database so here s a sample userlist txt mastodon md5d75bb2be2d7086c6148944261a00f605 pgbouncer md5a45753afaca0db833a6f7c7b2864b9d9 in both cases the password is just password configuring pgbouncer ini edit etc pgbouncer pgbouncer ini add a line under databases listing the postgresql databases you want to connect to here we ll just have pgbouncer use the same username password and database name to connect to the underlying postgresql database databases mastodon_production host 127 0 0 1 port 5432 dbname mastodon_production user mastodon password password the listen_addr and listen_port tell pgbouncer which address port to accept connections the defaults are fine listen_addr 127 0 0 1 listen_port 6432 put md5 as the auth_type assuming you re using the md5 format in userlist txt auth_type md5 make sure the pgbouncer user is an admin admin_users pgbouncer mastodon requires a different pooling mode than the default session based one specifically it needs a transaction based pooling mode this means that a postgresql connection is established at the start of a transaction and terminated upon its completion therefore it is essential to change the pool_mode setting from session to transaction pool_mode transaction next up max_client_conn defines how many connections pgbouncer itself will accept and default_pool_size puts a limit on how many postgresql connections will be opened under the hood in pghero the number of connections reported will correspond to default_pool_size because it has no knowledge of pgbouncer the defaults are fine to start and you can always increase them later max_client_conn 100 default_pool_size 20 don t forget to reload or restart pgbouncer after making your changes sudo systemctl reload pgbouncer debugging that it all works you should be able to connect to pgbouncer just like you would with postgresql psql p 6432 u mastodon mastodon_production then use your password to log in you can also check the pgbouncer logs like so tail f var log postgresql pgbouncer log configuring mastodon to talk to pgbouncer in your env production file first off make sure that this is set prepared_statements false since we re using transaction based pooling we can t use prepared statements next up configure mastodon to use port 6432 pgbouncer instead of 5432 postgresql and you should be good to go db_host localhost db_user mastodon db_name mastodon_production db_pass password db_port 6432 you cannot use pgbouncer to perform db migrate tasks but this is easy to work around if your postgresql and pgbouncer are on the same host it can be as simple as defining db_port 5432 together with rails_env production when calling the task for example rails_env production db_port 5432 bundle exec rails db migrate you can specify db_host too if it s different etc administering pgbouncer the easiest way to reboot is sudo systemctl restart pgbouncer but if you ve set up a pgbouncer admin user you can also connect as the admin psql p 6432 u pgbouncer pgbouncer and then do reload then use q to quit separate redis for cache redis is used widely throughout the application but some uses are more important than others home feeds list feeds and sidekiq queues as well as the streaming api are backed by redis and that s important data you wouldn t want to lose even though the loss can be survived unlike the loss of the postgresql database never lose that redis is also used for volatile caching if you are scaling up and you are concerned about redis s capacity to handle the load you can allocate a separate redis database specifically for caching to do this set cache_redis_url in the environment or define individual components such as cache_redis_host cache_redis_port etc unspecified components will default to their values without the cache prefix when configuring the redis database for caching it is possible to disable background saving to disk as data loss on restart is not critical in this context and this can save some disk i o additionally consider setting a maximum memory limit and implementing a key eviction policy for more details on these configurations refer to this guide using redis as an lru cache separate redis for sidekiq redis is used in sidekiq to keep track of its locks and queue although in general the performance gain is not that big some instances may benefit from having a separate redis instance for sidekiq in the environment file you can specify sidekiq_redis_url or individual parts like sidekiq_redis_host sidekiq_redis_port etc unspecified parts fallback to the same values as without the sidekiq_ prefix creating a separate redis instance for sidekiq is relatively simple start by making a copy of the default redis systemd service cp etc systemd system redis service etc systemd system redis sidekiq service in the redis sidekiq service file change the following values execstart usr bin redis server etc redis redis sidekiq conf supervised systemd daemonize no pidfile run redis redis server sidekiq pid readwritepaths var lib redis sidekiq alias redis sidekiq service make a copy of the redis configuration file for the new sidekiq redis instance cp etc redis redis conf etc redis redis sidekiq conf in this redis sidekiq conf file change the following values port 6479 pidfile var run redis redis server sidekiq pid logfile var log redis redis server sidekiq log dir var lib redis sidekiq before starting the new redis instance create a data directory mkdir var lib redis sidekiq chown redis var lib redis sidekiq start the new redis instance systemctl enable now redis sidekiq update your environment and add the following line sidekiq_redis_url redis 127 0 0 1 6479 restart mastodon to use the new redis instance ensure that you restart both web and sidekiq otherwise one of them will still be working from the wrong instance systemctl restart mastodon web service systemctl restart redis sidekiq service redis sentinel for high availability as mentioned redis is a critical part of a mastodon instance s operation by default your deployment will use a single redis instance or multiple if you ve setup a cache however if that instance goes down it can bring the entire mastodon instance down as well to alleviate this redis sentinel can be used to track your redis instances and automatically direct clients to a new primary if one goes down you can specify redis_sentinels which is a comma delimited list of the...
|
