Meta tags:
description=In November, we experienced two incidents that resulted in degraded performance across GitHub services. This report also sheds light into an incident that impacted Codespaces in October.;
author=Jakub Oleksy;

Headings (most frequently used words):

github, availability, report, 2022, the, insider, more, to, from, november, newsletter, on, interested, in, bringing, enterprise, your, organization, related, posts, explore, subscribe, product, platform, support, company, october, september, august, advanced, security, customers, can, now, push, protect, their, custom, patterns, creating, an, accessible, search, experience, with, querybuilder, component, highlights, git, 39, engineering, readme, project, actions, work, at,

Text of the page (most frequently used words):
#github (50), the (46), and (27), incident (19), that (17), report (16), for (14), codespaces (14), this (14), 2022 (13), our (13), actions (10), utc (10), more (9), experienced (9), impacted (9), availability (9), also (8), november (8), engineering (7), were (7), region (7), security (6), from (6), into (6), with (6), customers (6), performance (6), services (6), october (6), webhooks (6), blog (5), about (5), status (5), enterprise (5), learn (5), are (5), impact (5), investigating (5), sheds (5), light (5), across (5), which (5), statused (5), east (5), search (4), was (4), one (4), your (4), month (4), jakub (4), oleksy (4), august (4), will (4), resulted (4), degraded (4), queue (4), database (4), workflow (4), runs (4), minutes (4), community (3), api (3), product (3), subscribe (3), out (3), git (3), what (3), several (3), can (3), their (3), start (3), increase (3), trial (3), still (3), september (3), incidents (3), run (3), delays (3), caused (3), during (3), alerting (3), systems (3), detected (3), lasting (3), regional (3), third (3), party (3), outage (3), organization (2), company (2), contact (2), training (2), developer (2), stories (2), newsletter (2), insider (2), check (2), posts (2), team (2), new (2), highlights (2), have (2), component (2), advanced (2), protect (2), custom (2), patterns (2), push (2), other (2), days (2), after (2), free (2), resulting (2), significant (2), include (2), next (2), follow (2), time (2), updates (2), working (2), inefficient (2), query (2), repository (2), advisory (2), access (2), limits (2), they (2), potential (2), future (2), longer (2), background (2), jobs (2), first (2), found (2), webhook (2), high (2), backup (2), while (2), receiving (2), repositories (2), its (2), initially (2), red (2), yellow (2), hour (2), replication (2), mitigate (2), data (2), service (2), unable (2), not (2), healthy (2), create (2), could (2), manually (2), resumes (2), success (2), affected (2), hours (2), due (2), january (2), two (2), web (2), dec (2), privacy, terms, inc, linkedin, tiktok, twitch, youtube, facebook, twitter, shop, press, careers, forum, docs, support, desktop, electron, atom, partners, platform, resources, pricing, customer, features, developers, covering, techniques, technical, guides, latest, innovations, coming, current, job, openings, work, native, alongside, code, hosted, voices, readme, project, straight, explore, taylor, blau, another, release, here, end, year, take, look, some, lindsey, wild, inputs, complex, accessibility, considerations, let, dive, those, how, addressed, them, talk, standalone, reusable, ultimately, built, creating, accessible, experience, querybuilder, zain, malik, mariam, sulakian, just, click, admins, organizations, now, related, curious, plans, collaboration, per, user, expires, interested, bringing, july, four, get, best, once, directly, inbox, tags, please, real, changes, page, following, investigation, fixed, reviewed, rate, particular, endpoint, well, area, ensure, line, expectations, finally, increased, default, throttling, avoid, backups, term, improvement, resiliency, options, reduce, workflows, continue, game, conduct, enhanced, responders, better, assess, determine, appropriate, level, statusing, moving, forward, checking, triggered, volume, poorly, optimized, calls, running, why, multiple, addition, listens, trigger, scaled, accelerated, processing, mitigation, fully, drained, back, green, latency, deliveries, starting, progress, error, responses, pull, requests, peak, majority, attempting, than, five, took, assessing, identified, ways, improve, resilience, outages, internal, discussions, implementing, failover, type, quickly, uses, store, provider, experiencing, immediately, communicate, because, does, currently, any, returned, soon, engineers, able, engage, help, resume, select, alternate, but, decrease, rates, creates, only, persisted, provided, guidance, change, location, nearby, prolonged, using, packages, pages, recency, contributing, factors, provide, detailed, update, cause, remediation, publish, wednesday, december, author, keyword, sales, policy, education, changelog, open, source, wayback, machine, http, archive, org, 20221219202146, https, timestamps, capture, fail, 2023, 2021, jan, nov, may, 2024, captures,

Text of the page (random words):
github availability report november 2022 the github blog 22 captures 07 dec 2022 22 may 2024 nov dec jan 19 2021 2022 2023 success fail about this capture timestamps the wayback machine http web archive org web 20221219202146 https github blog 2022 12 07 github availability report november 2022 blog engineering product security open source enterprise changelog community education company policy free trial contact sales search by keyword search engineering enterprise github availability report november 2022 in november we experienced two incidents that resulted in degraded performance across github services this report also sheds light into an incident that impacted codespaces in october author jakub oleksy december 7 2022 in november we experienced two incidents that resulted in degraded performance across github services this report also sheds light into an incident that impacted github codespaces in october november 25 16 34 utc lasting 1 hour and 56 minutes our alerting systems detected an incident that impacted customers using github packages and pages due to the recency of this incident we are still investigating the contributing factors and will provide a more detailed update on cause and remediation in the january availability report which we will publish the first wednesday of january october 26 00 47 utc lasting 3 hours and 47 minutes on october 22 2022 at 00 47 utc our alerting systems detected a decrease in success rates for creates and resumes of codespaces in the east us region we initially statused yellow as the incident affected only the east us region as the incident persisted for several hours we provided guidance to customers in the affected region to manually change their location to a nearby healthy region at 01 55 utc and statused red at 2 34 utc due to the prolonged outage during this incident customers were unable to create or resume codespaces in the east us region customers could manually select an alternate region in which to create codespaces but could not do so for resumes codespaces uses a third party database to store data for the service and the provider was experiencing an outage which impacted codespaces performance we were unable to immediately communicate with our east us database because our service does not currently have any replication of its regional data our services in the east us region returned to healthy status as soon as codespaces engineers were able to engage with the third party to help mitigate the outage we identified several ways to improve our database resilience to regional outages while working with the third party during this incident and in follow up internal discussions we are implementing regional replication and failover so that we can mitigate this type of incident more quickly in the future november 3 16 10 utc lasting 1 hour and 2 minutes on november 3 2022 at 16 10 utc our alerting systems detected an increase in the time it took github actions workflow runs to start we initially statused github actions to red and after assessing impact we statused to yellow at 16 11 utc during this incident customers experienced high latency in receiving webhook deliveries starting github actions workflow runs and receiving status updates for in progress runs they also experienced an increase in error responses from repositories pull requests codespaces and the github api at its peak a majority of repositories attempting to run a github actions workflow experienced delays longer than five minutes github actions listens to webhooks to trigger workflow runs and while investigating we found that the run start delays were caused by a backup in the webhooks queue at 16 29 utc we scaled out and accelerated processing of the webhooks queue as a mitigation by 17 12 utc the webhooks queue was fully drained and we statused back to green we found that the webhook delays were caused by an inefficient database query for checking repository security advisory access which was triggered by a high volume of poorly optimized api calls this caused a backup in background jobs running across github which is why multiple services were impacted in addition to webhooks and github actions following our investigation we fixed the inefficient query for the repository security advisory access we also reviewed the rate limits for this particular endpoint as well as limits in this area to ensure they were in line with our performance expectations finally we increased the default throttling of the webhooks queue to avoid potential backups in the future as a longer term improvement to our resiliency we are investigating options to reduce the potential for other background jobs to impact github actions workflows we ll continue to run game days and conduct enhanced training for first responders to better assess impact for github actions and determine the appropriate level of statusing moving forward please follow our status page for real time updates on status changes to learn more about what we re working on check out the github engineering blog tags github availability report the github insider newsletter get the best of github once a month directly to your inbox subscribe more on github availability report github availability report october 2022 in october we experienced four incidents that resulted in degraded performance across github services this report also sheds light into an incident that impacted codespaces in september jakub oleksy github availability report september 2022 in september we experienced one incident that resulted in degraded performance across github services we also experienced one incident resulting in significant impact to codespaces we are still investigating that incident and will include it in next month s report this report also sheds light into an incident that impacted codespaces in august and an incident that impacted actions in august jakub oleksy github availability report august 2022 in august we experienced one incident resulting in significant impact to codespaces we re still investigating that incident and will include it in next month s report this report also sheds light into an incident that impacted codespaces in july jakub oleksy interested in bringing github enterprise to your organization start your free trial for 30 days and increase your team s collaboration 21 per user month after trial expires curious about other plans related posts enterprise github advanced security customers can now push protect their custom patterns with just one click admins in github advanced security organizations can protect their custom patterns on push mariam sulakian zain malik engineering creating an accessible search experience with the querybuilder component github s search inputs have several complex accessibility considerations let s dive into what those are how we addressed them and talk about the standalone reusable component that was ultimately built lindsey wild engineering highlights from git 2 39 another new release of git is here to end the year take a look at some of our highlights on what s new in git 2 39 taylor blau explore more from github engineering posts straight from the github engineering team learn more the readme project stories and voices from the developer community learn more github actions native ci cd alongside code hosted in github learn more work at github check out our current job openings learn more subscribe to the github insider a newsletter for developers covering techniques technical guides and the latest product innovations coming from github subscribe product features security enterprise customer stories pricing resources platform developer api partners atom electron github desktop support docs community forum training status contact company about blog careers press shop github on twitter github on facebook github on youtube github on twitch github on tiktok github on linkedin github s organization on github 2022 github inc terms privacy