Meta tags:

Headings (most frequently used words):

pgbouncer, configuring, redis, for, mastodon, separate, sidekiq, web, to, scaling, up, your, server, managing, concurrency, transaction, pooling, with, cache, sentinel, high, availability, read, replicas, using, load, balancer, user, generated, content, retention, sponsored, by, puma, streaming, api, background, processing, why, you, might, need, installing, number, of, threads, queues, setting, password, userlist, txt, ini, debugging, that, it, all, works, talk, administering,

Text of the page (most frequently used words):
the (182), and (62), redis (59), you (58), #mastodon (46), sidekiq (44), for (41), pgbouncer (41), your (38), streaming (34), #server (32), can (32), this (29), that (29), with (28), instance (24), are (22), database (22), use (21), password (21), postgresql (20), will (18), admin (18), number (18), user (17), api (17), default (17), web (16), processes (16), connections (16), load (15), etc (15), using (15), set (14), more (14), threads (14), service (13), start (13), environment (13), process (13), also (12), how (12), not (11), but (11), different (11), replica (11), port (11), then (11), one (10), than (10), systemctl (10), from (9), following (9), tasks (9), new (9), users (8), setting (8), should (8), multiple (8), make (8), run (8), connect (8), would (8), same (8), separate (8), restart (8), queue (8), configuring (8), queues (8), sudo (8), md5 (8), values (7), background (7), puma (7), single (7), may (7), when (7), file (7), many (7), all (7), some (7), sentinel (7), used (7), which (7), var (7), transaction (7), just (7), running (7), retention (6), controls (6), cache (6), them (6), need (6), these (6), http (6), scaling (6), servers (6), data (6), have (6), based (6), their (6), master (6), requests (6), want (6), variables (6), here (6), 127 (6), log (6), systemd (6), first (6), userlist (6), txt (6), 4000 (6), available (5), health (5), traffic (5), configure (5), balancer (5), like (5), sure (5), loss (5), add (5), configuration (5), db_port (5), production (5), read (5), about (5), example (5), clients (5), specify (5), 6432 (5), 5432 (5), mastodon_production (5), pooling (5), other (5), processing (5), accounts (5), long (4), settings (4), under (4), content (4), both (4), monitor (4), jobs (4), replicas (4), name (4), local (4), host (4), db_host (4), url (4), prepared_statements (4), good (4), without (4), management (4), such (4), variable (4), has (4), guide (4), down (4), instances (4), current (4), line (4), lib (4), conf (4), memory (4), requires (4), installing (4), out (4), posts (4), each (4), nginx (4), metrics (4), profile (4), account (4), oauth (4), generated (3), own (3), media (3), scheduler (3), endpoints (3), monitoring (3), documentation (3), setup (3), config (3), yml (3), could (3), makara (3), see (3), db_user (3), db_name (3), role (3), true (3), false (3), edit (3), currently (3), talk (3), possible (3), however (3), well (3), list (3), high (3), ensure (3), change (3), copy (3), system (3), creating (3), individual (3), caching (3), implementing (3), components (3), application (3), important (3), reload (3), psql (3), bundle (3), exec (3), default_pool_size (3), defaults (3), pghero (3), connection (3), databases (3), ini (3), real (3), roles (3), might (3), through (3), time (3), command (3), handling (3), importance (3), note (3), pool (3), db_pool (3), per (3), worker (3), streaming_cluster_num (3), 4002 (3), 4001 (3), fail_timeout (3), endpoint (3), 000 (3), search (3), preferences (3), apps (3), reports (3), view (2), source (2), last (2), backups (2), remote (2), prevent (2), stored (2), review (2), task (2), via (2), does (2), cron (2), administration (2), checks (2), status (2), code (2), text (2), deployment (2), behind (2), virtual (2), help (2), provide (2), performance (2), consider (2), otherwise (2), network (2), public (2), address (2), failing (2), urls (2), point (2), correct (2), going (2), primary (2), db_password (2), follows (2), writes (2), while (2), issue (2), impact (2), versions (2), before (2), ways (2), optional (2), value (2), replica_prepared_statements (2), replica_db_tasks (2), built (2), support (2), starting (2), queries (2), only (2), rails (2), app (2), sidekiq_ (2), critical (2), goes (2), track (2), node (2), after (2), configured (2), availability (2), still (2), working (2), 6479 (2), sidekiq_redis_url (2), enable (2), now (2), create (2), directory (2), pidfile (2), pid (2), making (2), simple (2), parts (2), unspecified (2), prefix (2), its (2), disk (2), context (2), additionally (2), maximum (2), limit (2), key (2), capacity (2), handle (2), allocate (2), specifically (2), define (2), uses (2), others (2), feeds (2), lose (2), though (2), never (2), quit (2), work (2), rails_env (2), migrate (2), next (2), instead (2), since (2), logs (2), able (2), max_client_conn (2), 100 (2), fine (2), later (2), accept (2), correspond (2), pool_mode (2), mode (2), session (2), therefore (2), auth_type (2), assuming (2), listen_port (2), listen_addr (2), username (2), md5d75bb2be2d7086c6148944261a00f605 (2), passwordmastodon (2), md5sum (2), ubuntu (2), debian (2), where (2), query (2), stats (2), permissions (2), equal (2), total (2), find (2), common (2), why (2), execution (2), services (2), experience (2), lower (2), priority (2), delivery (2), mails (2), push (2), affect (2), varying (2), much (2), affects (2), sent (2), request (2), those (2), made (2), cpu (2), resources (2), consume (2), started (2), balance (2), three (2), prometheus (2), messages (2), channels (2), connected (2), client (2), websocket (2), throughput (2), streaming_api_base_url (2), lived (2), control (2), share (2), larger (2), maxes (2), managing (2), concurrency (2), notificationpolicy (2), filter (2), report (2), quote (2), domainblock (2), notifications (2), trends (2), collections (2), domain_blocks (2), tokens (2), security (2), machine (2), features (2), imprint, blog, join, sponsored, updated, november, 2025, 简体中文, improve, page, archive, period, kept, cached, runaway, growth, assets, appropriate, cleanup, runs, job, manual, triggering, third, party, alerting, utility, return, 200, result, purpose, provision, balancers, consistent, reducing, risk, become, overwhelmed, decrease, downtime, performing, maintenance, upgrades, consult, provider, backend, nodes, send, responsive, cloud, providers, digitalocean, aws, hetzner, offer, balancing, solutions, distribute, across, stock, avoid, locations, locally, installed, two, mastodon_replica, above, there, possibilities, information, slave, blacklist_duration, sticky, postgres, postgresql_makara, adapter, replace, section, supported, lead, driver, reads, let, straight, querying, very, often, anyway, small, once, done, seeing, against, override, any, schema, migrations, seeds, alternatively, replica_database_url, replica_db_pass, replica_db_user, replica_db_name, replica_db_port, replica_db_host, version, every, included, directed, lagging, few, seconds, stop, sending, until, catches, reduce, wish, hot, replication, learn, https, docs, latest, operate, oss_and_stack, related, prefixed, case, cache_, mentioned, part, operation, bring, entire, alleviate, automatically, direct, comma, delimited, combinations, sentinels, determine, select, minute, being, unreachable, redis_sentinel_master, redis_sentinels, wrong, update, chown, mkdir, dir, logfile, alias, readwritepaths, usr, bin, supervised, daemonize, execstart, relatively, fallback, sidekiq_redis_port, sidekiq_redis_host, keep, locks, although, general, gain, big, benefit, having, disable, saving, save, eviction, policy, details, configurations, refer, lru, volatile, concerned, cache_redis_port, cache_redis_host, cache_redis_url, widely, throughout, home, backed, wouldn, even, survived, unlike, easiest, way, reboot, administering, cannot, perform, easy, around, defining, together, calling, too, db_pass, localhost, prepared, statements, off, env, tail, check, debugging, works, don, forget, changes, always, increase, defines, itself, puts, opened, hood, reported, because, knowledge, needs, means, established, terminated, upon, completion, essential, admin_users, put, format, tell, dbname, listing, underlying, cases, md5a45753afaca0db833a6f7c7b2864b9d9, sample, beginning, macos, openbsd, echo, scheme, string, prepended, derive, hash, arbitrarily, passwords, reuse, credentials, simplicity, sake, alter, obviously, word, reset, firstly, apt, install, extensions, usage, link, generally, open, combined, devops, solution, document, describes, gotchas, necessary, efficiently, truly, parallel, arguments, checking, finds, none, subsequent, overfilled, delays, priorities, overridden, invocation, refreshing, trending, hashtags, cleaning, imports, resolving, deleting, forwarding, replies, pull, mailers, incoming, activities, ingress, payloads, defined, wasn, listed, order, descending, initiate, thread, large, size, managed, least, regulated, rather, argument, invoking, shown, responsiveness, end, allocated, quickly, delivered, author, anyone, else, soon, delegated, fast, aborts, affecting, configurable, handled, interacted, planning, difficult, especially, comes, cpus, cause, contention, software, another, was, misconfiguring, exhaust, opening, cluster, potentially, previous, had, clustering, starts, equivalent, distributed, files, commands, least_conn, upstream, route, consumed, likely, documented, below, routed, arbitrary, less, insightful, tracking, received, pubsub, intended, complement, directly, mastodon_streaming_redis_messages_received_total, mastodon_streaming_messages_sent_total, subscribed, higher, due, our, internal, mastodon_streaming_connected_channels, tagged, type, eventsource, mastodon_streaming_connected_clients, exposes, lot, understand, reasonably, isn, community, monthly, active, typically, average, concurrent, 800, 1200, subdomain, allows, applications, correctly, hosted, shared, listen, socket, bind, base, handles, receive, updates, terms, better, served, parent, they, write, ram, max_threads, web_concurrency, serves, short, most, types, webpushsubscription, translation, token, termsofservice, tag, suggestion, statussource, statusedit, shallowtag, shallowquote, scheduledstatus, rule, relationshipseveranceevent, relationship, reaction, quoteapproval, privacypolicy, previewcardauthor, previewcard, poll, notificationrequest, notificationfallback, notification, mediaattachment, marker, identityproof, filterstatus, filterresult, filterkeyword, featuredtag, featureapproval, familiarfollowers, extendeddescription, error, customemoji, conversation, collectionwithaccounts, collectionitem, collection, asyncrefresh, appeal, annualreport, announcement, measure, ipblock, emaildomainblock, domainallow, dimension, cohort, canonicalemailblock, accountwarning, entities, markers, lists, conversations, timelines, scheduled_statuses, polls, statuses, proofs, oembed, custom_emojis, announcements, grouped, async_refreshes, emails, annual, measures, ip_blocks, email_domain_blocks, domain_allows, dimensions, canonical_email_blocks, tags, suggestions, mutes, followed_tags, follow_requests, filters, featured_tags, favourites, endorsements, bookmarks, blocks, methods, rate, limits, scopes, guidelines, best, practices, datetime, formats, rest, bearcaps, microformats, webfinger, activitypub, spec, compliance, design, reference, themes, css, styling, state, frontend, issues, responsible, disclosure, routes, structure, dev, technical, overview, contributing, libraries, implementations, logging, obtaining, access, playing, getting, developing, webhooks, index, corruption, troubleshooting, errors, moderation, actions, migrating, backing, upgrading, release, cli, sign, captcha, onion, object, storage, full, preparing, official, ios, android, moving, leaving, externally, promoting, yourself, dealing, unwanted, quoting, posting, signing, what,

Text of the page (random words):
unning your own server running mastodon preparing your machine installing from source configuring your environment configuring full text search installing optional features object storage onion services captcha single sign on setting up your new instance using the admin cli upgrading to a new release backing up your server migrating to a new machine scaling up your server moderation actions troubleshooting errors database index corruption webhooks roles developing mastodon apps getting started with the api playing with public data obtaining client app access logging in with an account libraries and implementations implementing quote posts implementing collections contributing to mastodon technical overview setting up a dev environment code structure routes security issues and responsible disclosure frontend guide components state management css and styling creating themes design tokens reference spec compliance activitypub webfinger security microformats oauth bearcaps rest api datetime formats guidelines and best practices oauth tokens oauth scopes rate limits api methods accounts blocks bookmarks domain_blocks endorsements favourites featured_tags filters follow_requests followed_tags mutes preferences reports suggestions tags admin accounts canonical_email_blocks dimensions domain_allows domain_blocks email_domain_blocks ip_blocks measures reports retention trends annual reports apps emails oauth async_refreshes collections grouped notifications health instance announcements custom_emojis directory trends notifications push oembed profile proofs search statuses media polls scheduled_statuses timelines conversations lists markers streaming api entities account accountwarning admin account admin canonicalemailblock admin cohort admin dimension admin domainallow admin domainblock admin emaildomainblock admin ip admin ipblock admin measure admin report announcement annualreport appeal application asyncrefresh collection collectionitem collectionwithaccounts context conversation customemoji domainblock error extendeddescription familiarfollowers featureapproval featuredtag filter filterkeyword filterresult filterstatus identityproof instance list marker mediaattachment notification notificationfallback notificationpolicy notificationrequest poll preferences previewcard previewcardauthor privacypolicy profile quote quoteapproval reaction relationship relationshipseveranceevent report role rule scheduledstatus search shallowquote shallowtag status statusedit statussource suggestion tag termsofservice token translation v1 filter v1 instance v1 notificationpolicy webpushsubscription scaling up your server managing concurrency web puma streaming api background processing sidekiq transaction pooling with pgbouncer why you might need pgbouncer installing pgbouncer configuring pgbouncer separate redis for cache separate redis for sidekiq redis sentinel for high availability read replicas mastodon 4 2 mastodon 4 1 using a web load balancer user generated content retention managing concurrency mastodon has three types of processes web puma streaming api background processing sidekiq web puma the web process serves short lived http requests for most of the application the following environment variables control it web_concurrency controls the number of worker processes max_threads controls the number of threads per process threads share the memory of their parent process different processes allocate their own memory though they share some memory via copy on write a larger number of threads maxes out your cpu first and a larger number of processes maxes out your ram first these values affect how many http requests can be served at the same time in terms of throughput more processes are better than more threads streaming api the streaming api handles long lived http and websocket connections through which clients receive real time updates the following environment variables control it streaming_api_base_url controls the base url of the streaming api port controls the port the streaming server will listen on by default 4000 the bind and socket environment variables are also able to be used additionally the shared database and redis environment variables are used the streaming api can use a different subdomain if you want to by setting streaming_api_base_url this allows you to have one load balancer for streaming and one for web api requests however this also requires applications to correctly request the streaming url from the instance endpoint instead of assuming that it s hosted on the same host as the web api one process of the streaming server can handle a reasonably high number of connections and throughput but if you find that a single process isn t handling your instance s load you can run multiple processes by varying the port number of each and then using nginx to load balance traffic to each of those instances for example a community of about 50 000 accounts with 10 000 20 000 monthly active accounts you ll typically have an average concurrent load of about 800 1200 streaming connections the streaming server also exposes a prometheus endpoint on metrics with a lot of metrics to help you understand the current load on your mastodon streaming server some key metrics are mastodon_streaming_connected_clients this is the number of connected clients tagged by client type websocket or eventsource mastodon_streaming_connected_channels this is the number of channels that are currently subscribed note that this is much higher than connected clients due to how our internal system channels currently work mastodon_streaming_messages_sent_total this is the total number of messages sent to clients since last restart mastodon_streaming_redis_messages_received_total this is the number of messages received from redis pubsub and intended to complement monitoring redis directly note that for prometheus based monitoring you should monitor each individual streaming server instance not the nginx endpoint as requests to nginx will be routed to an arbitrary streaming server instance and make the metrics less insightful than the per instance tracking the more streaming server processes that you run the more database connections will be consumed on postgresql so you ll likely want to use pgbouncer as documented below an example nginx configuration to route traffic to three different processes on port 4000 4001 and 4002 is as follows upstream streaming least_conn server 127 0 0 1 4000 fail_timeout 0 server 127 0 0 1 4001 fail_timeout 0 server 127 0 0 1 4002 fail_timeout 0 if you re using the distributed systemd files then you can start up multiple streaming servers with the following commands sudo systemctl start mastodon streaming 4000 service sudo systemctl start mastodon streaming 4001 service sudo systemctl start mastodon streaming 4002 service by default sudo systemctl start mastodon streaming starts just one process on port 4000 equivalent to running sudo systemctl start mastodon streaming 4000 service previous versions of mastodon had a streaming_cluster_num environment variable that made the streaming server use clustering which started multiple worker processes and used node js to load balance them this interacted with the other settings in ways which made capacity planning difficult especially when it comes to database connections and cpu resources by default the streaming server would consume resources on all available cpus which could cause contention with other software running on that server another common issue was that misconfiguring the streaming_cluster_num would exhaust your database connections by opening up a connection pool per cluster worker process so a streaming_cluster_num of 5 and db_pool of 10 would potentially consume 50 database connections now a single streaming server process will only use at maximum db_pool postgresql connections and scaling is handled by running more instances of the streaming server background processing sidekiq many tasks in mastodon are delegated to background processing to ensure the http requests are fast and to prevent http request aborts from affecting the execution of those tasks sidekiq is a single process with a configurable number of threads number of threads while the number of threads in the web process affects the responsiveness of the mastodon instance to the end user the number of threads allocated to background processing affects how quickly posts can be delivered from the author to anyone else how soon e mails are sent out etc the number of threads is not regulated by an environment variable but rather through a command line argument when invoking sidekiq as shown in the following example bundle exec sidekiq c 15 this would initiate the sidekiq process with 15 threads it is important to note that each thread requires a database connection so this requires a large database pool the size of this pool is managed by the db_pool environment variable which should be set to a value at least equal to the number of threads queues sidekiq uses different queues for tasks of varying importance where importance is defined by how much it would impact the user experience of your server s local users if the queue wasn t working the queues are listed here in order of descending importance default all tasks that affect local users push delivery of payloads to other servers ingress incoming remote activities lower priority than the default queue so that local users still see their posts when the server is under load mailers delivery of e mails pull lower priority tasks such as handling imports backups resolving threads deleting users forwarding replies scheduler handling cron jobs such as refreshing trending hashtags and cleaning up logs the default queues and their priorities are stored in config sidekiq yml but can be overridden by the command line invocation of sidekiq e g bundle exec sidekiq q default this command will run just the default queue sidekiq processes queues by first checking for tasks in the first queue and if it finds none it then checks the subsequent queue therefore if the first queue is overfilled tasks in the other queues may experience delays it is possible to start different sidekiq processes for the queues to ensure truly parallel execution by e g creating multiple systemd services for sidekiq with different arguments you may run as many sidekiq processes with as many threads as necessary to efficiently process running jobs however the scheduler queue should never be run in more than one sidekiq process at a time transaction pooling with pgbouncer why you might need pgbouncer if you start running out of available postgresql connections the default is 100 then you may find pgbouncer to be a good solution this document describes some common gotchas as well as good configuration defaults for mastodon user roles with devops permissions in mastodon can monitor the current usage of postgresql connections through the pghero link in the administration view generally the number of connections open is equal to the total threads in puma sidekiq and the streaming api combined to enable query stats in pghero review the query stats guide and make sure your postgresql has the correct permissions and extensions configured installing pgbouncer on debian and ubuntu sudo apt install pgbouncer configuring pgbouncer setting a password firstly if your mastodon user in postgresql is set up without a password you will need to set a password here s how you might reset the password psql p 5432 u mastodon mastodon_production w then obviously use a different password than the word password alter user mastodon with password password then q to quit configuring userlist txt edit etc pgbouncer userlist txt as long as you specify a user password in pgbouncer ini later the values in userlist txt do not have to correspond to real postgresql roles you can arbitrarily define users and passwords but you can reuse the real credentials for simplicity s sake add the mastodon user to the userlist txt mastodon md5d75bb2be2d7086c6148944261a00f605 here we re using the md5 scheme where the md5 password is just the md5sum of password username with the string md5 prepended for instance to derive the hash for user mastodon with password password ubuntu debian etc echo n passwordmastodon md5sum macos openbsd etc md5 s passwordmastodon then just add md5 to the beginning of that you ll also want to create a pgbouncer admin user to log in to the pgbouncer admin database so here s a sample userlist txt mastodon md5d75bb2be2d7086c6148944261a00f605 pgbouncer md5a45753afaca0db833a6f7c7b2864b9d9 in both cases the password is just password configuring pgbouncer ini edit etc pgbouncer pgbouncer ini add a line under databases listing the postgresql databases you want to connect to here we ll just have pgbouncer use the same username password and database name to connect to the underlying postgresql database databases mastodon_production host 127 0 0 1 port 5432 dbname mastodon_production user mastodon password password the listen_addr and listen_port tell pgbouncer which address port to accept connections the defaults are fine listen_addr 127 0 0 1 listen_port 6432 put md5 as the auth_type assuming you re using the md5 format in userlist txt auth_type md5 make sure the pgbouncer user is an admin admin_users pgbouncer mastodon requires a different pooling mode than the default session based one specifically it needs a transaction based pooling mode this means that a postgresql connection is established at the start of a transaction and terminated upon its completion therefore it is essential to change the pool_mode setting from session to transaction pool_mode transaction next up max_client_conn defines how many connections pgbouncer itself will accept and default_pool_size puts a limit on how many postgresql connections will be opened under the hood in pghero the number of connections reported will correspond to default_pool_size because it has no knowledge of pgbouncer the defaults are fine to start and you can always increase them later max_client_conn 100 default_pool_size 20 don t forget to reload or restart pgbouncer after making your changes sudo systemctl reload pgbouncer debugging that it all works you should be able to connect to pgbouncer just like you would with postgresql psql p 6432 u mastodon mastodon_production then use your password to log in you can also check the pgbouncer logs like so tail f var log postgresql pgbouncer log configuring mastodon to talk to pgbouncer in your env production file first off make sure that this is set prepared_statements false since we re using transaction based pooling we can t use prepared statements next up configure mastodon to use port 6432 pgbouncer instead of 5432 postgresql and you should be good to go db_host localhost db_user mastodon db_name mastodon_production db_pass password db_port 6432 you cannot use pgbouncer to perform db migrate tasks but this is easy to work around...