Meta tags:
description=Service status updates for the Student-Run Computing Facility;

Headings (most frequently used words):

2026, server, upgrades, 12, webserver, srcf, net, 06, downtime, service, issues, sinkhole, aka, pip, 2025, services, student, run, computing, facility, status, updates, what, happening, actions, required, preview, environment, 10, 09, shell, login, on, and, doom, upcoming, software, full, outage, 26, to, 28, mysql, 01, 02, 22, 2024, 18, membership, about,

Text of the page (most frequently used words):
the (68), and (39), srcf (26), #upgrade (26), sinkhole (20), this (19), you (18), net (16), with (16), user (14), 2026 (14), #server (13), upgrades (13), out (12), some (12), all (12), after (12), our (12), drupal (11), service (10), update (10), pip (10), for (10), run (10), services (9), software (9), but (9), version (9), have (8), webserver (8), begins (8), running (8), systemd (8), php (8), can (8), web (7), account (7), aka (7), rebooted (7), disk (7), that (7), quicksand (7), postgresql (7), ubuntu (7), was (6), should (6), shell (6), are (6), system (6), mysql (6), reboot (6), https (6), need (6), here (6), your (6), latest (6), packages (6), email (6), control (5), panel (5), back (5), from (5), updates (5), space (5), machines (5), not (5), well (5), machine (5), which (5), first (5), servers (5), new (5), python (5), will (5), now (5), changes (5), updated (5), complete (5), more (5), status (4), there (4), failed (4), around (4), downtime (4), how (4), like (4), available (4), doom (4), been (4), 2025 (4), performance (4), initiated (4), any (4), since (4), files (4), use (4), installed (4), site (4), versions (4), while (4), time (4), done (4), details (4), going (4), upcoming (4), create (3), though (3), start (3), these (3), were (3), 30am (3), hosting (3), attempts (3), seems (3), caused (3), shared (3), rebooting (3), looks (3), facing (3), has (3), completes (3), restart (3), sysadmins (3), room (3), installation (3), conditionhost (3), already (3), both (3), websites (3), may (3), still (3), continue (3), environment (3), wordpress (3), make (3), sure (3), support (3), final (3), release (3), required (3), hopefully (3), weekend (3), please (3), student (3), computing (3), facility (3), guidance (3), publishing (3), issues (3), tumblr (3), reverted (3), suffers (3), pid (3), segfault (3), problematic (3), sep (3), grub (3), upgrader (3), contact (2), mailing (2), list (2), join (2), older (2), itself (2), online (2), where (2), normal (2), morning (2), taking (2), access (2), just (2), 2024 (2), overnight (2), lock (2), other (2), group (2), similar (2), processes (2), investigation (2), startup (2), two (2), hours (2), jun (2), keep (2), fire (2), making (2), bringing (2), systems (2), full (2), bst (2), docs (2), reference (2), scheduled (2), between (2), preview (2), then (2), ssh (2), usual (2), delay (2), conditions (2), follows (2), missing (2), know (2), set (2), see (2), works (2), under (2), sites (2), those (2), live (2), default (2), only (2), don (2), work (2), python3 (2), venv (2), virtual (2), newer (2), old (2), installations (2), one (2), actions (2), improve (2), continuing (2), receive (2), security (2), them (2), take (2), place (2), sat (2), sun (2), september (2), much (2), their (2), long (2), supposed (2), aug (2), without (2), resolving (2), turbulence (2), nis (2), ran (2), current (2), planned (2), obtains (2), until (2), boot (2), finished (2), sleep (2), migration (2), blocked (2), process (2), memory (2), faqs, addresses, minutes, constitution, committee, about, timeout, terminal, lists, webmail, privacy, statement, terms, donate, society, membership, archive, stay, warm, holiday, season, 9am, timed, caught, retried, 10am, resumed, experienced, freeze, remote, noticed, 8am, little, later, thaw, dec, 20pm, fully, restored, 00pm, exactly, remains, unclear, turn, locking, several, fixed, problem, 45am, issue, home, directories, experiencing, unwedge, diagnose, seemed, ongoing, feb, seemingly, levels, improving, yes, showing, poor, impacting, applications, squirrel, reading, indication, actual, damage, arrangements, physical, visit, evaluate, situation, further, hopes, soon, possible, learnt, fault, suppression, triggered, loss, power, equipment, main, unreachable, offline, outage, tasks, www, org, project, releases, posting, progress, irc, discord, during, interested, getting, help, minute, when, creating, applies, app, enable, assuming, want, parallel, replace, condition, triggering, rather, than, databases, static, cgi, modules, libraries, disabled, let, something, depend, install, temporary, found, note, custom, domains, aren, supported, point, sample, soc, spqr2, unix, socket, connection, stop, working, future, aim, split, across, multiple, ident, authentication, provide, password, host, postgres, connections, used, based, recreating, against, environments, supports, date, looking, active, its, maintainers, website, last, ideally, 103, minimum, also, aware, does, perform, heavy, load, many, noticeably, slow, step, towards, modernising, unlikely, who, previously, brought, attention, age, outdatedness, particularly, infrastructure, apologies, patches, secure, asking, self, case, lot, play, nice, offering, becomes, difficult, each, won, whole, expect, significant, outages, they, 12th, 13th, 5th, 6th, imminent, node, cover, four, years, feature, centrally, closer, notable, intended, such, game, hosted, preparing, what, happening, affect, capacity, read, including, section, gotten, temporarily, misplaced, within, university, enjoy, instead, sorry, kept, most, important, beginning, resolved, second, updater, crashing, expected, network, interface, renames, dns, cut, off, disabling, resolvers, attached, root, filesystem, very, good, estimating, needs, front, tinkering, free, build, downgraded, timeline, applied, resolve, moment, things, experience, managing, logging, over, typical, prepare, causing, repeat, crashes, login, success, welcome, couple, reinstalling, migrated, legacy, didn, properly, jumping, rescue, image, applying, post, config, resumes, drawbridge, immediately, get, pausing, tomorrow, too, survived, safety, precautions, manual, tidy, syncing, being, held, quite, bit, due, unresponsive, deadly, signal, segv, strikes, again, actually, false, decided, core, exist, dependencies, job, runner, currently, picking, jobs, attempt, continues, cluster, pending, early, hiccup, left, delisted, halfway, big, carrying, program, starting, break, cycle, address, excessively, high, usage, hitting, repeatedly, few, signs, recovery, oct, nagios, twitter, rss,

Text of the page (random words):
srcf status student run computing facility control panel services contact join service status updates rss twitter nagios 12 oct 2026 22 01 2026 10 12 webserver issues the webserver machine sinkhole has been hitting out of memory conditions repeatedly for a few hours without signs of recovery 22 00 we re rebooting the machine to break the cycle then can address any excessively high memory usage 23 00 sinkhole is rebooted and we re in the process of starting the system and user services that were running 12 sep 2026 19 53 2026 09 12 server upgrades sinkhole aka webserver srcf net this is the final big user facing server upgrade we re carrying out more details on the upgrade program here upcoming server software upgrades changes since publishing 2026 09 01 updated drupal guidance to upgrade to latest 9 x this email is going out to all with a user account on tumblr 12 09 20 00 bst upgrade process begins 22 25 upgrade of packages to 22 04 is complete rebooting for the first time 22 45 we are on ubuntu 22 04 halfway there 22 55 24 04 upgrade blocked after an early 22 04 hiccup left delisted packages still installed 23 35 24 04 upgrade begins 13 09 00 40 upgrade of packages to 24 04 complete pending reboot 02 00 postgresql cluster upgrade blocked on missing packages but sysadmins need some sleep 09 00 postgresql investigation continues 10 20 postgresql migration begins 10 35 failed as we ran out of disk space 10 45 attempt 2 with more disk 11 00 postgresql migration complete 11 45 srcf control panel updated to work with new dependencies but job runner currently not picking up new jobs 12 00 srcf control panel back online 5 sep 2026 16 41 server upgrades pip aka shell srcf net more details on the current and planned upgrades here upcoming server software upgrades changes since publishing 2026 09 01 updated drupal guidance to upgrade to latest 9 x this email is going out to all with a user account on tumblr 05 09 16 45 20 04 22 04 upgrade begins 16 55 false start as the upgrader decided the core files don t exist 17 45 upgrade actually begins 19 55 deadly signal segv strikes again though looks like the upgrade can continue 22 55 upgrader finished running after being held up quite a bit due to an unresponsive systemd 23 05 some manual tidy up done around nis user account syncing 23 20 after some safety precautions reboot initiated 23 25 we have survived a reboot and run on 22 04 06 09 01 10 we re now continuing to upgrade to 24 04 03 55 pausing until tomorrow sysadmins need sleep too 08 20 upgrade resumes though the drawbridge is up and we can t immediately get back in 08 30 we re in 09 55 upgrader finished running now applying post upgrade config changes 10 10 reboot initiated 10 15 looks like grub didn t update properly so we can t boot jumping in to rescue image 11 00 after a couple of no op attempts at reinstalling grub we ve now migrated from grub legacy to 2 11 05 boot success welcome to ubuntu 24 04 4 sep 2026 19 39 service login issues on pip and sinkhole typical just as we prepare to upgrade to the newer os version we receive a security update on the old os which seems to be causing repeat crashes you may experience some turbulence managing your systemd services or logging in over ssh up until the scheduled upgrades we ve reverted the problematic update for the moment to see if things improve ubuntu have now reverted their release and we ve applied the update which should hopefully resolve this timeline 02 09 06 30 sinkhole obtains the problematic update 20 15 sinkhole suffers a systemd pid 1 segfault 22 40 sinkhole rebooted 03 09 06 30 pip obtains the problematic update 04 09 09 25 pip suffers a systemd pid 1 segfault 16 00 sinkhole suffers a systemd pid 1 segfault 18 20 pip rebooted 19 10 sinkhole rebooted 19 30 systemd downgraded on pip and sinkhole 09 09 16 00 sinkhole updated with ubuntu s reverted build and rebooted 30 aug 2026 20 10 server upgrades doom srcf net more details on the current and planned upgrades here upcoming server software upgrades this email is going out to all with a user account on the student run computing facility srcf well this was supposed to be an email but tumblr 20 10 20 04 22 04 upgrade begins 21 00 ran out of disk space ubuntu is not very good at estimating how much it needs up front so some tinkering required to free up more space 21 30 first upgrade complete after resolving the usual turbulence with nis 21 45 22 04 24 04 upgrade begins this time with a new disk attached to the root filesystem 22 15 second upgrade complete without the updater crashing for the first time in a while after resolving expected network interface renames but dns cut itself off by disabling all resolvers 22 25 all issues hopefully resolved we re live on 24 04 30 aug 2026 13 41 upcoming server software upgrades changes since publishing 2026 09 01 updated drupal guidance to upgrade to latest 9 x this email is going out to all with a user account on the student run computing facility srcf well this was supposed to be an email but it seems to have gotten temporarily misplaced within the university mailing list system so please enjoy it here instead it s a long one sorry but we ve kept the most important details at the beginning this will affect you if you use our services in any capacity so please make sure you read up to and including the actions required section what s happening we are preparing to upgrade our user facing servers from ubuntu 20 04 to ubuntu 24 04 these servers are pip aka shell srcf net sinkhole aka webserver srcf net where all srcf hosted websites run doom intended for long running processes such as game servers this will cover four years of feature updates to all the software packages we have centrally installed bringing them much closer to their latest versions notable changes php 7 4 8 3 python 3 8 3 12 node js 10 18 postgresql 12 16 upgrades will take place as follows doom imminent done pip weekend of sat 5th sun 6th september done sinkhole weekend of sat 12th sun 13th september done hopefully each upgrade won t take the whole weekend but please expect significant outages while they re taking place for those of you who have previously brought to our attention the age and outdatedness of some of our software packages particularly our web infrastructure and php apologies for the delay while we ve been continuing to receive security patches which keep our systems secure 1 we know that asking you to run older versions of self installed software like wordpress or drupal in the case of php is a lot and that making them play nice with our offering becomes difficult with time we re also well aware that our web server does not perform well under heavy load and many websites run noticeably slow this upgrade is the first step towards modernising our web hosting system but it s unlikely that performance will improve for now actions required drupal installations if you run a drupal website you should be running the final release of drupal 9 version 9 5 11 the last version to support php 7 4 so ideally make sure your site is running this version the minimum version to support both php 7 4 and 8 3 is the final release of drupal 7 version 7 103 after the upgrade you should be looking to update to drupal 11 4 the latest version and the only one with active support from its maintainers wordpress installations the latest version of wordpress supports both old and new php versions so make sure your site installation is up to date python virtual environments if you ve used python3 m venv or similar to create a python virtual environment based on the system installation these will need recreating against the newer version of python python 3 12 is available on all machines already but is not the default you can use python3 12 m venv to create an environment which should continue to work after the upgrades postgresql connections set host postgres as the default unix socket connection only works on sinkhole and may stop working in future as we aim to split out web hosting across multiple machines ident authentication is available so you still don t need to provide your postgresql password preview environment we ve set up a temporary 24 04 web server quicksand srcf net so you can see how your site works under the new software versions user sites can be found at e g https spqr2 user quicksand srcf net while group account sites use e g https sample soc quicksand srcf net note that custom domains aren t supported here those continue to point to your live site on sinkhole files and databases are shared between quicksand and our other servers static php and cgi websites should already be up and running some php modules and installed libraries may be missing or disabled let us know if there s something you depend on and we can install it if it s still available if you use a systemd service to run an app you ll need to enable it for quicksand assuming you already have conditionhost sinkhole 3 and want to run it on both servers in parallel replace the condition with two triggering conditions rather than as follows conditionhost sinkhole conditionhost quicksand then ssh in to quicksand and start the service the usual 20 minute delay when first creating a service applies here we ll be posting progress updates on https status srcf net and we ll be around on irc and discord between and during the upgrades if you re interested in how we re getting on or need any help with the preview 1 https docs srcf net reference shell and files software and installation 2 https www drupal org project drupal releases 9 5 11 3 https docs srcf net reference shell and files scheduled tasks systemd 26 jun 2026 08 52 full outage 2026 06 26 to 2026 06 28 all srcf equipment in our main machine room has been unreachable offline since 06 56 bst this morning at 08 44 we learnt that this was caused by a fault in the machine room s fire suppression system which triggered a full loss of power there is no indication of an actual fire or of any damage to the srcf s machines the sysadmins are making arrangements for a physical visit to the machine room to evaluate the situation further with hopes of bringing systems back to running as soon as possible keep reading 1 jun 2025 13 00 mysql downtime 2025 06 01 our mysql server squirrel was showing some poor performance impacting user applications as well as the control panel 13 00 mysql service restart initiated 15 00 mysql service restart completes yes two hours to restart 16 25 reboot initiated after performance not improving 16 35 reboot completes mysql begins startup 17 25 mysql service startup completes 18 15 performance seemingly back to normal levels 22 feb 2025 09 45 services downtime 2025 02 22 overnight system updates seemed to have caused some disk access and processes on sinkhole aka webserver srcf net to lock up from around 6 30am investigation ongoing update 10 30am after some failed attempts to unwedge or diagnose it sinkhole has been rebooted update 10 45am looks like this is an issue with shared disk space i e user group home directories available on all user facing machines shell srcf net and doom srcf net are experiencing similar downtime update 3 20pm services were fully restored at 2 00pm it seems overnight updates caused pip aka shell srcf net to lock up the shared disk space exactly how remains unclear in turn locking up several other machines rebooting pip fixed the problem 18 dec 2024 10 00 webserver downtime 2024 12 18 the srcf webserver sinkhole aka webserver srcf net experienced a freeze from around 4 30am this morning taking out all web hosting and remote access this was noticed just after 8am and rebooted a little later after some failed attempts to thaw it the server itself was back online by 9am though some user services timed out and failed to start these were caught and retried by 10am where service should have resumed as normal stay warm out there this holiday season 1 2 3 4 5 older archive membership join the srcf create society account list of services donate terms of service privacy statement services control panel webmail mailing lists web terminal timeout server status about committee constitution minutes contact addresses faqs