Meta tags:
Headings (most frequently used words):
in, operational, slowness, portal, site, and, environment, us, api, eu, kb, down, base, service, knowledge, authentication, hub, intermittent, timeouts, issue, time, disruption, knowledgebase, document360, user, false, alert, instance, is, incidents, canada, rendering, timeout, high, response, out, private, projects, outage, azure, interruption, all, systems, subscribe, to, updates, status, maintenance, europe, incident, we, are, experiencing, outs, application, for, sites, cdn, version, apihub, v2, degradation, facing, got, knowlege, home, page, analytics, services, customer, website, downtime, certificate, dns, issues, public, affected, components,
Text of the page (most frequently used words):
the (277), and (155), uptime (135), 2026 (132), 100 (120), 000 (120), nov (115), gmt (93), are (86), issue (85), our (77), resolved (69), have (50), this (49), #slowness (47), service (41), portal (41), for (39), monitoring (38), environment (37), was (35), services (34), site (33), history (31), now (31), has (31), show (30), incident (28), identified (27), operational (27), caused (25), been (25), azure (24), 2025 (24), timeouts (23), api (22), cause (22), base (21), document360 (20), investigating (20), knowledge (20), 2024 (20), currently (19), team (19), that (19), disruption (18), application (18), customers (18), situation (18), all (17), system (17), with (16), status (16), experienced (16), some (16), jul (16), over (16), authentication (15), actively (15), database (15), past (15), days (15), causing (14), you (14), root (14), intermittent (14), down (13), will (13), open (13), observed (13), during (13), scaling (13), your (12), outage (12), due (12), impact (12), 2023 (12), any (12), support (12), time (12), monitor (12), working (11), feb (11), update (11), alert (11), mitigation (11), out (11), systems (11), hub (11), sep (11), page (10), response (10), may (10), closed (10), false (10), region (10), ensure (10), requests (10), high (10), server (10), from (9), normal (9), knowledgebase (9), which (9), resolution (9), traffic (9), please (8), aug (8), affecting (8), back (8), being (8), instance (8), mar (8), measures (8), further (8), analysis (8), not (8), user (8), availability (8), connection (8), incidents (8), jun (8), cloudflare (8), times (7), experiencing (7), change (7), utc (7), stable (7), fully (7), home (7), had (7), surge (7), understanding (7), performance (7), future (7), private (7), degradation (7), redis (7), applied (7), address (6), 2019 (6), issues (6), apr (6), facing (6), servers (6), functioning (6), sql (6), more (6), connectivity (6), taken (6), investigation (6), analytics (6), cluster (6), node (6), load (6), continued (6), should (6), increased (6), potential (6), identity (6), timeout (6), mitigated (6), operating (6), custom (6), monitors (6), updates (5), https (5), there (5), oct (5), infrastructure (5), one (5), dns (5), restored (5), accessing (5), sites (5), however (5), once (5), latency (5), resources (5), after (5), apologize (5), reach (5), impacting (5), volume (5), resulted (5), process (5), thank (5), inquiries (5), additional (5), information (5), hesitate (5), steps (5), periods (5), february (5), minutes (5), machines (5), closing (5), event (5), applications (5), health (5), stabilized (5), normally (5), returned (5), fixed (4), inconvenience (4), delayed (4), dec (4), investigated (4), progress (4), 2020 (4), customer (4), closely (4), related (4), users (4), 2022 (4), downtime (4), platform (4), europe (4), provide (4), when (4), recent (4), underlying (4), stability (4), ping (4), url (4), period (4), triggered (4), data (4), other (4), such (4), pages (4), next (4), operations (4), prevent (4), require (4), affected (4), also (4), usage (4), sudden (4), resulting (4), projects (4), ist (4), peak (4), horizontal (4), were (4), implemented (4), resource (4), fix (4), expected (4), virtual (4), continuing (4), corrective (4), actions (4), css (4), baseline (4), maintenance (3), higher (3), apis (3), problem (3), while (3), keep (3), here (3), jan (3), shortly (3), 2021 (3), cloud (3), com (3), microsoft (3), certificate (3), resolving (3), resolve (3), configuration (3), operation (3), summary (3), business (3), treated (3), state (3), automatic (3), recovery (3), seamless (3), comprehensive (3), enhance (3), reliability (3), auto (3), these (3), loads (3), disruptions (3), who (3), mode (3), approximately (3), increase (3), connections (3), mitigating (3), engaged (3), reliable (3), experience (3), even (3), appreciate (3), patience (3), dedicated (3), subsequent (3), determined (3), characterized (3), its (3), temporary (3), production (3), multiple (3), using (3), mechanisms (3), manage (3), threshold (3), share (3), since (3), apihub (3), version (3), result (3), scale (3), spike (3), current (3), central (3), backend (3), request (3), styles (3), correctly (3), capacity (3), cache (3), metrics (3), levels (3), canada (3), email (2), subscribe (2), aware (2), public (2), moment (2), auth0 (2), login (2), outages (2), technical (2), posted (2), instances (2), hosted (2), everything (2), rendering (2), link (2), website (2), running (2), encountering (2), waiting (2), same (2), interruption (2), network (2), rolled (2), level (2), conducting (2), earlier (2), today (2), positive (2), specified (2), confusion (2), questions (2), concerns (2), insights (2), feedback (2), improve (2), processes (2), beginning (2), november (2), highest (2), priority (2), rca (2), report (2), nodes (2), unhealthy (2), incoming (2), processing (2), leading (2), configured (2), articles (2), tier (2), ensured (2), conclusion (2), implementing (2), necessary (2), got (2), restoring (2), configurations (2), worked (2), patterns (2), taking (2), contact (2), set (2), inaccessibility (2), would (2), consistently (2), seconds (2), unusually (2), influx (2), turn (2), revealed (2), thread (2), attributable (2), pool (2), starvation (2), led (2), elevated (2), initiated (2), measure (2), demand (2), identify (2), areas (2), improvement (2), similar (2), resilience (2), continuously (2), enhancing (2), between (2), encountered (2), limit (2), write (2), delay (2), occurred (2), disrupting (2), duration (2), persisted (2), 185 (2), total (2), interruptions (2), activated (2), reached (2), critical (2), exacerbated (2), challenge (2), solution (2), vertically (2), core (2), capable (2), handling (2), adjustment (2), ensures (2), improved (2), thereby (2), limitations (2), improving (2), provided (2), monitored (2), failover (2), lasted (2), few (2), recycled (2), make (2), mitigate (2), originated (2), based (2), cpu (2), consumption (2), sustained (2), waf (2), rate (2), residual (2), several (2), incorrectly (2), impacted (2), www (2), cloudflarestatus (2), online (2), avoid (2), within (2), access (2), restarted (2), unexpectedly (2), compute (2), automatically (2), continue (2), slower (2), than (2), usual (2), recovered (2), errors (2), safe (2), html (2), valid (2), deployment (2), upgrade (2), something, went, wrong, try, again, thanks, subscribing, get, notifications, about, reported, 13th, august, websites, apologies, partner, logging, function, engineering, updating, particular, short, ssl, blocking, 05th, april, both, malfunction, send, notification, notified, earliest, seamlessly, west, affects, engineers, metadata, drop, overall, networking, manifesting, attempting, connect, wan, additionally, determine, flagged, thorough, concluded, fact, genuine, feel, free, invaluable, work, tuesday, restore, overwhelming, consequently, synchronization, secondary, new, failure, widget, display, sections, docs, remained, unaffected, responded, initiating, available, premium, transitioned, healthy, restoration, ultimately, deeply, regret, committed, occurrences, knowlege, friday, select, functionality, unusual, directed, towards, overwhelmed, partial, manager, hosting, includes, strengthening, better, handle, demands, challenges, diligently, only, resilient, but, efficient, responding, varying, upon, found, significant, marked, number, notable, directly, coupled, inability, establish, immediate, heal, setup, efficiently, reactive, could, adapt, severity, accessibility, aim, implement, proactive, responsiveness, ensuring, robust, unexpected, offline, attributing, meticulous, indeed, findings, affirm, throughout, authentic, sincerely, erroneous, pertaining, don, great, value, strive, continually, 02pm, substantial, reaching, maximum, concurrent, defined, start, details, made, code, faced, scheduled, limited, tags, received, alarm, endpoint, components, entering, addressed, through, changes, targeted, fixes, abnormal, fail, apps, seem, upgrades, temporarily, force, performing, forced, edge, cases, involving, constraints, bring, post, low, malicious, payloads, evaded, detection, existing, dos, layers, threat, actor, exploited, gaps, rule, sensitivity, bypassing, anomaly, scoring, involved, reinforcing, heuristics, adaptive, limiting, where, displayed, though, they, remain, acknowledge, most, face, occasional, steadily, 8gmgl950y3h7, confirmed, without, hence, cdn, noticed, brought, statement, starting, failures, trying, teams, management, workflow, deployed, blocked, subset, storage, clusters, restarting, lost, disks, applying, see, signs, applies, across, events, warrant, specific, pattern, caching, slowdown, keeping, close, watch, sure, stays, smoothly, brief, slow, delays, window, normalized, validated, recently, isolated, strictly, outs, pressure, prepared, traced, guardrails, unsafe, prevented, those, updated, validation, logic, allow, maintaining, intended, security, protections, completed, exceptions, reboot, responses, changed, primary, quick, succession, transitions, unable, obtain, repeatedly, extended, revisited, ceiling, headroom, inicdent, upstream, p95, p99, round, trip, saturation, surfacing, downstream, exceeded, client, thresholds, workload, side, end, contributed, remediated, fault, pools, error, around, silent, non, disruptive, refreshed,
Text of the page (random words):
ng to access some virtual machines hosted in the region these virtual machines may have also restarted unexpectedly current status we are aware of this issue and have engaged multiple teams we have determined the underlying cause a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of azure storage clusters and compute resources in the central us region this resulted in the compute resources automatically restarting when connectivity was lost to virtual disks we are currently applying mitigation customers should see signs of recovery at this time as mitigation applies across resources in the region the next update will be provided in 60 minutes or as events warrant open jul 19 2024 00 25 gmt 01 00 we are currently facing downtime in the portal and user authentication applications within our us environment caused by a disruption in azure services in the central us region our team is actively investigating this issue high response time for kb sites in eu environment closed feb 24 2026 00 00 gmt 00 00 the issue was the result of a db cluster change which resulted in temporary connection issue we will be taking steps to improve the connection resilience to avoid such incidents in the future show history investigating feb 17 2026 10 11 gmt 00 00 we have noticed an incident in eu knowledge base site the service has been stabilized now we are currently investigating the issue cdn issue resolved nov 18 2025 21 00 gmt 00 00 cloudflare has confirmed the issue has resolved and all our services have been working without any issue hence closing this issue show history monitoring nov 18 2025 12 55 gmt 00 00 most services are now online some may face occasional slowness traffic performance is steadily improving as cloudflare services are restored cloudflare status https www cloudflarestatus com incidents 8gmgl950y3h7 open nov 18 2025 11 54 gmt 00 00 all our services are impacted due to the cloudflare incident we are monitoring the situation cloudflare has acknowledge the issue and is working on mitigation cloudflare status link https www cloudflarestatus com open nov 18 2025 11 46 gmt 00 00 we are currently facing an issue with cloudflare where several services are incorrectly displayed as down even though they remain operational this is a cloudflare related problem and our team is actively monitoring the situation slowness in eu environment resolved nov 03 2025 11 30 gmt 00 00 post incident analysis revealed the cpu spike was triggered by a sustained influx of low threshold malicious payloads that evaded detection by the existing waf and dos mitigation layers the threat actor exploited gaps in rule sensitivity bypassing anomaly scoring mechanisms mitigation involved reinforcing waf heuristics implementing adaptive rate limiting current status all systems have been stabilized and are operating normally with no residual impact show history monitoring nov 03 2025 07 35 gmt 00 00 the knowledge base site version 2 has observed a slowness in and spike in response time based on our analysis this is due high cpu consumption we have taken mitigation measure to scale up and bring down resource consumption we are currently monitoring the situation intermittent slowness us environment resolved oct 28 2025 00 00 gmt 00 00 the issue was resolved by performing a forced scale up we have also implemented measures to ensure that scale up operations are triggered even in edge cases involving resource constraints show history monitoring oct 16 2025 19 51 gmt 01 00 the services are stable in us environment now and we are monitoring the issue open oct 16 2025 19 47 gmt 01 00 we encountered an issue with our database in us environment and have temporarily mitigated it by force scaling the server the situation is currently being monitored closely slowness and timeout in us environment resolved aug 24 2025 06 48 gmt 01 00 this incident was caused due to redis connectivity issue as a result of node upgrades this issue is now resolved and services are fully operational show history monitoring aug 22 2025 23 34 gmt 01 00 all the services are operational now and we are monitoring the situation open aug 22 2025 23 11 gmt 01 00 we are experience slowness and timeouts for applications in us environment the issue seem to be originated due to slowness from redis slowness and time out in eu kb site version 2 resolved jul 08 2025 04 02 gmt 01 00 the application health is back to normal and we are closing this incident show history monitoring jul 07 2025 20 05 gmt 01 00 we have observed some intermittent slowness in eu region due to redis fail over event the apps are stable now we are monitoring the situation monitoring jul 07 2025 12 13 gmt 01 00 some of our customers in the eu region have experienced slowness and timeouts on the kb site version 2 the issue has been mitigated and we are currently monitoring the situation to ensure stability slowness and time out in kb site resolved jul 04 2025 15 48 gmt 01 00 the issue causing intermittent slowness and timeouts for some customers accessing knowledge base sites in eu region has been resolved it was addressed through make changes to the auto scaling configurations to manage traffic load and targeted fixes to mitigate the impact of abnormal traffic patterns show history monitoring jun 30 2025 13 47 gmt 01 00 some of our customers experienced slowness and timeouts on the knowledge base site due to one of our service instances entering an unhealthy state the affected instance was recycled and the service has since been restored to normal operation we are actively monitoring the situation to ensure continued stability slowness in apihub resolved jun 30 2025 13 46 gmt 01 00 the api hub is now stable closing this issue after monitoring show history monitoring jun 23 2025 09 03 gmt 01 00 we experienced some slowness in api hub the issue has been mitigated for now and we are investigating the root cause service disruption in eu knowledge base site v2 closed jun 23 2025 09 04 gmt 01 00 the service disruption was caused by a sudden surge in application usage we have identified the potential cause and fixed the issue slowness in kb site and api hub resolved mar 18 2025 13 27 gmt 00 00 we observed a failover event in redis that lasted for a few minutes causing slowness in kb and apihub all applications are functioning as expected now affected components europe show history investigating mar 18 2025 12 20 gmt 00 00 we experienced some slowness in knowledge base site and api hub the issue has been mitigated for now and we are actively investigating the root cause false alert in us knowledge base resolved aug 08 2024 10 11 gmt 01 00 closing this issue since the root cause has been identified as a false alert show history open aug 08 2024 10 10 gmt 01 00 the service degradation alert received in the us environment is a false alarm caused by a dns issue with the monitored endpoint service degradation in kb resolved jul 29 2024 13 02 gmt 01 00 the issue is now resolved show history resolved jun 29 2024 06 06 gmt 01 00 we have observed the resolution provided have mitigated the service disruption and system is stable now monitoring jun 29 2024 05 00 gmt 01 00 we have identified the root cause and fixed the issue the service degradation was limited to articles with tags we are currently monitoring the situation identified jun 29 2024 04 30 gmt 01 00 we faced service degradation in eu and us environment in during scheduled maintenance intermittent slowness in portal api resolved jul 29 2024 11 00 gmt 01 00 the slowness was caused by a recent change made for improving the code stability we have identified the root cause and fixed the issue show history open jul 29 2024 09 56 gmt 01 00 we are facing intermittent slowness in portal api we have identified the root cause and working on a fix service disruption in portal resolved may 22 2024 17 17 gmt 01 00 start time 2024 05 22 16 23 00 we are experiencing slowness and downtime in portal site due to disruption in azure service we re actively working on to resolve this issue we will share more updates once the issue is resolved thank you for your understanding and continued support should you have any further inquiries or require additional information please do not hesitate to reach out to our support team resolved time 2024 05 22 16 50 00 this issue has been resolved now we ll share more details shortly facing slowness in authentication resolved feb 26 2024 10 40 gmt 00 00 incident during february 20 09 21 to 10 41 pm ist and february 21 07 26 to 09 11 pm ist a production outage occurred in the identity server disrupting service for multiple customers duration the slowness persisted for 185 minutes in total causing interruptions in service availability and connectivity impact customers using document360 portal and private projects had intermittent timeouts cause during peak traffic periods the system experienced performance degradation characterized by delayed response times and subsequent timeouts automatic horizontal scaling mechanisms were activated to manage the increased load on our servers however as the horizontal scaling reached a critical threshold it exacerbated connection issues with the sql database resulting in timeouts resolution to address this challenge we implemented a solution by vertically scaling our server infrastructure to higher core machines capable of handling increased loads with high availability this adjustment ensures improved performance and reliability during peak usage periods thereby mitigating potential disruptions caused by resource limitations thank you for your understanding and continued support should you have any further inquiries or require additional information please do not hesitate to reach out to our support team show history monitoring feb 22 2024 07 01 gmt 00 00 we have taken mitigation measures for the identity service slowness the application is working normal now and we are monitoring the situation investigating feb 21 2024 15 25 gmt 00 00 we are facing slowness in our identity authentication servers we are currently investigating the issue service disruption in private projects kb site and portal resolved feb 26 2024 10 40 gmt 00 00 incident during february 20 09 21 to 10 41 pm ist and february 21 07 26 to 09 11 pm ist a production outage occurred in the identity server disrupting service for multiple customers duration the slowness persisted for 185 minutes in total causing interruptions in service availability and connectivity impact customers using document360 portal and private projects had intermittent timeouts cause during peak traffic periods the system experienced performance degradation characterized by delayed response times and subsequent timeouts automatic horizontal scaling mechanisms were activated to manage the increased load on our servers however as the horizontal scaling reached a critical threshold it exacerbated connection issues with the sql database resulting in timeouts resolution to address this challenge we implemented a solution by vertically scaling our server infrastructure to higher core machines capable of handling increased loads with high availability this adjustment ensures improved performance and reliability during peak usage periods thereby mitigating potential disruptions caused by resource limitations thank you for your understanding and continued support should you have any further inquiries or require additional information please do not hesitate to reach out to our support team show history identified feb 20 2024 04 02 gmt 00 00 between 04 02pm and 04 57 pm utc on 20 february 2024 customers who had set up private mode experienced inaccessibility to the knowledgebase and portal site s pages causing a disruption in service requests for the home page or portal would consistently time out after approximately 30 seconds root cause after conducting an investigation it was determined that the identity server encountered a sudden and substantial increase in traffic characterized by an unusually high volume of requests this surge led to the database system reaching its maximum concurrent requests limit resulting in data write issues mitigation the delay in the data write process resulted in temporary system slowness which in turn caused a delay in the scaling up process as defined in the system next steps we are currently engaged in a comprehensive analysis to identify potential areas for improvement to prevent similar incidents in the future we appreciate the understanding and patience of our customers during this incident and we are dedicated to continuously enhancing our systems to provide a seamless and reliable service false alert knowledgebase instance 3 is down resolved dec 26 2023 23 22 gmt 00 00 earlier today our monitoring system identified a ping url as being offline attributing it to a dns issue subsequent to a meticulous investigation we have determined that this alert was indeed a false positive our findings affirm that the ping url was fully operational throughout the specified period and there was no authentic disruption in service we sincerely apologize for any confusion or inconvenience this erroneous alert may have caused should you have further inquiries concerns or insights pertaining to this incident please don t hesitate to contact our dedicated support team your feedback is of great value to us as we strive to enhance our systems and processes continually service disruption in private projects portal and kb site resolved dec 14 2023 15 24 gmt 00 00 impact customers who had set up private mode experienced inaccessibility to the knowledgebase and portal site s pages causing a disruption in service requests for the home page or portal would consistently time out after approximately 30 seconds cause upon investigation it was found that the identity server experienced a sudden and significant surge marked by an unusually high number of requests this influx in turn triggered a timeout situation in the sql server our analysis revealed a notable increase in thread connections directly attributable to the surge in requests this surge coupled with connection pool starvation led to the inability to establish necessary connections with the database resulting in the observed timeouts mitigation to address the immediate impact our auto heal setup and process efficiently identified the elevated server load and initiated the scaling out of resources this reactive measure ensured that the system could adapt to the increased demand mitigating the severity of the issue and restoring accessibility to the knowledgebase and portal pages for users in private mode next steps we are currently engaged in a comprehensive analysis to identify potential areas for improvement to prevent similar incidents in the future our aim is to implement proactive measures that will enhance the system s resilience and responsiveness ensuring a more robu...
|
