Meta tags:
description= Resolve issues with GPU VMs;
Headings (most frequently used words):
gpu, troubleshoot, vms, using, errors, and, nvidia, issues, support, xid, messages, nccl, memory, stay, organized, with, collections, save, categorize, content, based, on, your, preferences, by, dcgm, performance, for, a3, open, case, review, reset, gpus, what, next, diagnose, identify, faulty, node, run, benchmark, test, analyze, logs, monitor, thermal, throttling, smi, dcgmi, gsp, illegal, access, other, common, error, products, pricing, resources, engage,
Text of the page (most frequently used words):
the (181), and (135), create (119), vms (115), for (101), with (87), #troubleshoot (67), gpu (62), about (61), instance (50), manage (46), using (44), disk (43), server (39), view (38), disks (35), use (34), that (34), your (34), nvidia (33), performance (33), windows (33), mig (33), cloud (32), gpus (32), configure (32), overview (31), xid (30), compute (30), set (29), you (28), google (27), reservation (27), machine (26), engine (26), from (25), access (24), workloads (22), error (21), monitor (21), hyperdisk (21), sql (21), node (20), images (20), linux (19), logs (19), network (19), this (18), errors (18), run (18), following (18), cluster (18), resources (17), delete (17), reset (17), memory (17), workload (17), image (17), policies (16), host (16), reservations (16), instances (16), attached (15), stop (15), load (15), snapshots (15), groups (15), ssh (15), recommendations (15), configuration (15), issues (14), management (14), creation (14), virtual (14), apply (14), custom (14), connect (14), are (13), data (13), storage (13), benchmark (13), bulk (13), microsoft (13), all (12), type (12), sole (12), availability (12), persistent (12), best (12), regional (12), nodes (11), can (11), application (11), high (11), based (11), practices (11), stateful (11), install (10), metadata (10), migs (10), machines (10), migrate (10), add (10), zones (10), page (9), types (9), then (9), file (9), dcgm (9), ecc (9), recovery (9), files (9), output (9), nccl (9), across (9), number (9), licenses (9), managed (9), tenant (9), multiple (9), openshift (9), shutdown (9), time (9), start (9), hpc (9), license (8), review (8), issue (8), message (8), messages (8), dcgmi (8), manager (8), applications (8), enable (8), group (8), scale (8), service (8), import (8), cpu (8), resource (8), login (8), optimize (8), internal (8), choose (8), disaster (8), agent (8), existing (8), update (8), guest (8), boot (8), keys (8), h4d (8), flexibility (8), dns (8), optimized (8), tpu (8), events (7), thumb (7), information (7), case (7), directory (7), smi (7), example (7), driver (7), hardware (7), test (7), report (7), default (7), available (7), drivers (7), command (7), list (7), transfer (7), operations (7), multi (7), networking (7), commitments (7), startup (7), balancing (7), manually (7), idle (7), addresses (7), request (7), deploy (7), back (7), mysql (7), zonal (7), remove (7), maintenance (7), templates (7), snapshot (7), backup (7), center (6), code (6), support (6), see (6), down (6), more (6), details (6), send (6), recreate (6), customer (6), name (6), identify (6), single (6), highly (6), operating (6), log (6), environment (6), diagnostic (6), scripts (6), non (6), thermal (6), series (6), export (6), log_directory (6), nodeset (6), build (6), monitoring (6), usage (6), security (6), migration (6), updates (6), instant (6), nested (6), mode (6), demand (6), pools (6), failover (6), extension (6), change (6), new (6), customized (6), spot (6), samples (5), system (5), understand (5), after (5), care (5), 0x00000000 (5), common (5), illegal (5), resolve (5), gsp (5), bug (5), row (5), process (5), get (5), failure (5), h100 (5), section (5), local (5), health (5), active (5), analyze (5), faulty (5), bandwidth (5), tools (5), databases (5), accounts (5), future (5), virtualization (5), placement (5), modify (5), capacity (5), schedules (5), audit (5), symphony (5), options (5), email (5), between (5), systems (5), enhanced (5), capabilities (5), tenancy (5), restore (5), regions (5), control (5), workstation (5), faq (5), flex (5), português (4), español (4), other (4), need (4), persists (4), move (4), specify (4), have (4), reboot (4), some (4), complete (4), such (4), has (4), diag (4), debug (4), bit (4), retirement (4), together (4), event (4), enabled (4), these (4), gather (4), troubleshooting (4), project (4), during (4), throttling (4), value (4), query (4), check (4), increase (4), hostname (4), where (4), replace (4), net (4), plan (4), running (4), hostfile (4), per (4), perform (4), how (4), pro (4), distributed (4), authentication (4), ubuntu (4), console (4), tcp (4), pmu (4), ssd (4), share (4), projects (4), upgrade (4), live (4), autoscaling (4), cross (4), region (4), requests (4), calendar (4), metrics (4), clusters (4), containers (4), container (4), database (4), uses (4), postgresql (4), extensions (4), work (4), zone (4), suspend (4), location (4), resize (4), graceful (4), locations (4), plans (4), replication (4), bound (4), action (3), terms (3), status (3), content (3), under (3), apache (3), next (3), repair (3), tool (3), only (3), target (3), located (3), kubernetes (3), most (3), one (3), indicates (3), bus (3), resolution (3), returned (3), caused (3), out (3), when (3), timeout (3), remap (3), pending (3), contains (3), documentation (3), must (3), changes (3), were (3), blackwell (3), through (3), names (3), format (3), logging (3), count (3), setting (3), each (3), mpirun (3), nccl_debug_file (3), info (3), method (3), all_reduce_perf (3), full (3), error_severity (3), recommended (3), latest (3), versions (3), guide (3), guides (3), infrastructure (3), observability (3), analytics (3), automatic (3), encryption (3), serial (3), configurations (3), patterns (3), higher (3), discounts (3), cuds (3), manual (3), scalable (3), resilient (3), autoscale (3), balancer (3), prevent (3), policy (3), testing (3), ibm (3), spectrum (3), deploying (3), prepare (3), deployment (3), rhel (3), protect (3), certificates (3), state (3), disable (3), properties (3), account (3), schedule (3), lifecycle (3), design (3), size (3), pool (3), creating (3), workstations (3), tpus (3), product (3), 한국어 (2), 日本語 (2), עברית (2), brasil (2), italiano (2), indonesia (2), français (2), américa (2), latina (2), deutsch (2), english (2), sign (2), site (2), youtube (2), training (2), started (2), github (2), release (2), pricing (2), products (2), last (2), updated (2), 2026 (2), utc (2), licensed (2), its (2), feedback (2), into (2), resetting (2), restart (2), gke (2), automate (2), requires (2), executable (2), home (2), bin (2), var (2), than (2), might (2), require (2), steps (2), pci (2), 0x02a (2), 149 (2), means (2), off (2), cases (2), degradation (2), different (2), cuda (2), bounds (2), level (2), processor (2), corrects (2), affect (2), reserved (2), recording (2), double (2), a100 (2), dynamic (2), which (2), detects (2), reports (2), dbe (2), guidance (2), including (2), kernel (2), device (2), preceding (2), generate (2), checks (2), settings (2), interleaved (2), suspect (2), open (2), violation (2), likely (2), fail (2), warn (2), suite (2), includes (2), temperatures (2), clocks_throttle_reasons (2), hw_slowdown (2), temperature (2), either (2), job (2), collective (2), pid (2), stored (2), grep (2), opcount (2), nccl_log (2), find (2), fastrak (2), variables (2), unique (2), nccl_debug (2), nccl_debug_subsys (2), want (2), pinpoint (2), divide (2), results (2), shows (2), benchmarks (2), num_gpus_per_node (2), total_processes (2), hostfile_name (2), slots (2), slurm (2), operation (2), describes (2), h200 (2), ultra (2), severity (2), retirements (2), drain (2), help (2), error_id (2), diagnose (2), ensure (2), version (2), sdk (2), languages (2), frameworks (2), costs (2), industry (2), solutions (2), hybrid (2), multicloud (2), pipelines (2), hosting (2), development (2), quota (2), commitment (2), consumption (2), standard (2), registration (2), sles (2), port (2), collecting (2), arm (2), core (2), api (2), rdp (2), general (2), app (2), latency (2), visible (2), cores (2), automatically (2), committed (2), generation (2), underutilized (2), insights (2), utilization (2), globally (2), web (2), autohealing (2), iis (2), external (2), shared (2), reserve (2), autoscaler (2), autoscalers (2), activity (2), provider (2), place (2), services (2), redis (2), cloning (2), writer (2), always (2), alwayson (2), volumes (2), authenticate (2), synchronization (2), global (2), switch (2), byol (2), payg (2), byos (2), families (2), securing (2), simulate (2), topology (2), repairs (2), failures (2), template (2), suspended (2), stopped (2), distribution (2), physical (2), cancel (2), once (2), limit (2), interfaces (2), ipv6 (2), static (2), decrease (2), basic (2), consistent (2), scoped (2), copy (2), appliances (2), asynchronous (2), throughput (2), encrypt (2), mount (2), extreme (2), balanced (2), credentials (2), organization (2), user (2), advanced (2), rdma (2), created (2), specific (2), preemptible (2), provisioning (2), models (2), rtx (2), vws (2), constraints (2), accelerator (2), apis (2), reference (2), technology (2), areas (2), close (2), subscribe, newsletter, our, third, decade, climate, join, cookies, privacy, tech, twitter, blog, engage, certification, architecture, getting, notes, community, forums, contact, sales, marketplace, easy, easytounderstand, solved, problem, solvedmyproblem, otherup, hard, hardtounderstand, incorrect, sample, incorrectinformationorsamplecode, missing, missingtheinformationsamplesineed, otherdown, tell, except, otherwise, noted, java, registered, trademark, oracle, affiliates, developers, creative, commons, attribution, what, stage, persist, alternatively, also, whenever, lib, sudo, less, mentions, known, affecting, firmware, b200, 0000, netir_link_evt, fatal, xc0, link, 485c6, not, able, communicate, fallen, nvlink, very, rare, cause, levels, coverage, duration, gdb, memcheck, typically, trying, already, freed, dereferencing, invalid, pointer, array, fault, graphics, exception, xids, 120, collect, 119, rpc, microcontroller, runs, handles, low, functions, uncontained, contained, correctable, shouldn, informational, needed, rate, allows, heal, least, two, rows, bank, remapped, remapper, remapping, their, suggested, resolutions, prior, was, supported, later, releases, introduced, recommends, keeping, used, temporary, protected, correction, sbe, provides, handling, grouped, potential, causes, printed, placed, however, sometimes, return, any, recent, made, follow, installations, usr, relevant, failing, exact, identified, ids, unable, result, occurred, accompanied, clock, overheating, further, investigation, violations, column, being, throttled, due, timestamp, bus_id, csv, current, experience, they, consistently, reach, greater, before, aborts, temporarily, nccl_fastrak_data_transfer_timeout_ms, 3600000, straggler, completes, fewer, entries, ranks, first, earliest, sed, sort, head, shell, session, creates, propagate, flag, store, init, coll, doesn, detailed, sub, half, narrow, binary, search, until, individual, misbehaving, hangs, significantly, lower, particular, busbw, execute, against, individually, total, mpi, processes, launch, hosts, ld_library_path, hostfiles, separate, listing, hostnames, depends, highgpu, nodesets, logical, sets, partitions, causing, systematically, subsets, like, large, jobs, responding, poorly, often, occurs, because, underperform, slow, entire, analyzing, edge, mega, values, dcgm_errors, dcgmerrorseverity_enum, take, note, corresponds, resolves, dcgm_error_reset, tip, snippet, recoverable, provided, advice, debugging, instructions, category, tests, gpu_id, allocated, 23376170169, bytes, warnings, error_category, warning, detected, reported, include, taking, actionable, installed, installation, model, managing, environments, bottlenecks, save, categorize, preferences, stay, organized, collections, concurrent, renewal, nvme, resizing, pay, viewing, sudoers, screenshots, soft, lockups, locks, rescue, inaccessible, dumps, panic, fstab, unresponsive, suspension, reboots, shutdowns, connectivity, tips, nics, dpdk, improve, resiliency, communication, reduce, compact, idpf, interface, irdma, nic, customize, threads, sustained, merge, split, extend, renew, without, purchase, cost, savings, overcommit, cpus, floating, reliable, udp, backends, routing, https, scaling, consuming, consume, combine, cud, allow, sharing, decisions, signals, serving, predictions, organize, labels, replica, states, observe, tensorflow, inference, tensorrt5, learning, monte, carlo, methods, spark, forwarding, others, integrate, clustering, exchange, 2016, sharepoint, strategies, passive, inactive, setups, built, integrations, validation, deployed, transition, hammerdb, bucket, aws, ec2, pacemaker, s2d, block, netapp, hot, standby, drbd, architectures, client, private, address, mailjet, mailgun, sendgrid, sending, blue, green, deployments, lamp, joomla, asp, interactive, mongodb, flask, terraform, servers, over, mtls, random, generator, virtio, rng, accurate, protocol, ntp, append, els, restrictions, packages, deprecate, family, trusted, red, hat, knowledgebase, functionality, risks, vpc, controls, confidential, kek, secure, expiration, shielded, attributes, predefined, handle, notices, unmanaged, preserved, turn, alternate, fails, repairing, maintain, click, upgrades, override, selectively, applying, accelerate, outage, rebalance, reenable, proactive, redistribution, shape, distribute, define, accidental, deletion, spread, edit, rename, resume, referrers, source, uuid, recover, corrupted, duplicate, clones, alerts, backups, vss, regionally, protection, considerations, synchronous, consistency, failback, replicate, make, provisioned, iops, limits, evaluate, kms, supplied, symbolic, links, practice, ram, attach, additional, exapools, verify, identity, ptr, record, rates, tags, restrict, password, powershell, sac, securely, apps, root, vpn, bastion, iap, connection, browser, connections, networks, within, iso, imported, prerequisites, importing, exporting, bring, own, path, detach, reattach, constraint, accounting, provision, examples, team, slice, slices, adds, scenarios, deterministic, base, requirements, ready, aci, configured, preempted, historical, alternative, similar, displays, ops, subnet, public, gpudirect, minimum, platform, quickstarts, iam, roles, permissions, office, licensing, premium, strategy, gemini, bare, metal, accelerators, platforms, scores, purpose, discover, free, skip, main,
Text of the page (random words):
ty logs monitor resources monitor vm and sole tenant node usage observe and monitor vms monitor gpu performance monitor gpu performance on linux vms monitor gpu performance on windows vms monitor disks monitor disk health monitor the replica states of regional disks monitor disks list of metrics for pools monitor pools monitor reservations organize resources using labels scale autoscale groups of vms about autoscaling groups of vms create and manage autoscalers scale based on cpu utilization scale based on predictions scale based on load balancing serving capacity scale based on monitoring metrics scale based on schedules use an autoscaling policy with multiple signals manage autoscalers understand autoscaler decisions view autoscaler logs autoscale node groups reserve vm capacity choose a reservation type sharing reservations best practices for shared reservations allow a project to share reservations on demand reservations about on demand reservations create an on demand reservation for a single project for multiple projects combine an on demand reservation with a cud modify an on demand reservation delete an on demand reservation future reservations about future reservations create a reservation request for a single project for multiple projects modify a reservation request delete a reservation request future reservations in calendar mode about future reservations in calendar mode create a reservation request in calendar mode delete a reservation request in calendar mode view reservations or reservation requests consume a reservation prevent vms from consuming reservations load balancing about load balancing and scaling add an instance group to a load balancer request routing to a multi region external https load balancer cross region load balancing for microsoft iis backends set up internal tcp udp load balancing build reliable and scalable applications use autohealing for highly available applications use load balancing for highly available applications use autoscaling for highly scalable applications globally autoscale a web service on compute engine patterns for scalable and resilient applications patterns for using floating ip addresses on compute engine optimize resource utilization use recommendations to manage resources apply machine type recommendations to vms configure machine type recommendations apply machine type recommendations to migs view and apply idle resources recommendations view and understand vm insights view and understand mig insights manage idle vm recommendations idle vm recommendations overview view and apply idle vm recommendations configure idle vm recommendations manage reservation recommendations reservation recommendations overview view and apply idle reservation recommendations view and apply underutilized reservation recommendations configure idle reservation recommendations configure underutilized reservation recommendations overcommit cpus on sole tenant vms manual live migration about manual live migration manually live migrate vms share sole tenant node groups next generation dynamic resource management cost savings get discounts for committed usage about commitments and committed use discounts cuds resource based cuds purchase resource based commitments without attached reservations with attached reservations for os licenses manage resource based commitments renew commitments automatically extend commitment terms merge and split commitments upgrade commitments share resource based cuds across projects get discounts for sustained usage disk performance optimize hyperdisk performance optimize persistent disk performance optimize local ssd performance workload performance set the number of threads per core customize the number of visible cpu cores analyze the cpu performance using the pmu pmu overview enable the pmu in vms manage the pmu in vms network performance network bandwidth use google virtual nic use irdma network driver use idpf network interface configure a vm with higher bandwidth reduce latency by using compact placement policies optimize tcp network communication optimize tcp network performance optimize tcp network resiliency benchmark higher bandwidth vms optimize app latency with load balancing use dpdk to improve network performance network performance and gpu vms networking and gpu machines use higher network bandwidth patterns for using multiple host nics troubleshoot general tips troubleshoot connectivity troubleshoot rdp troubleshoot ssh troubleshoot os login troubleshoot vms troubleshoot vm operations troubleshoot vm creation troubleshoot resource availability errors troubleshoot bulk api vm creation troubleshoot vm reboots and shutdowns troubleshoot vm suspension troubleshoot vm updates troubleshoot unresponsive vms troubleshoot vm startup troubleshoot fstab errors troubleshoot kernel panic collecting core dumps rescue an inaccessible vm troubleshoot cpu bus locks troubleshoot cpu soft lockups troubleshoot vm configurations troubleshoot arm vms troubleshoot gpu vms troubleshoot nvidia gpu errors generate a nvidia bug report for blackwell gpus troubleshoot nested virtualization troubleshoot using vm screenshots troubleshoot sole tenant nodes troubleshoot vm performance issues troubleshoot sudoers files troubleshoot windows vms troubleshoot windows vms collecting diagnostic information troubleshoot using the serial console troubleshoot using the serial console viewing serial port output troubleshoot instance groups troubleshoot managed instance groups migs troubleshoot os management troubleshoot licenses troubleshoot image import and export troubleshooting sles pay as you go registration troubleshooting ubuntu pro registration troubleshoot metadata server troubleshoot metadata server troubleshoot networking issues troubleshoot common networking issues troubleshoot network drivers troubleshoot vm performance issues troubleshoot storage troubleshoot disk creation troubleshoot full disks and disk resizing troubleshoot disk encryption troubleshoot nvme disks troubleshoot instant snapshots troubleshoot standard snapshots troubleshoot reservations and commitments troubleshoot reservation creation troubleshoot reservation consumption troubleshooting reservation monitoring troubleshoot reservation updates troubleshoot future reservation creation and updates troubleshoot automatic commitment renewal troubleshoot quota errors troubleshoot concurrent operation quota errors troubleshoot workload authentication troubleshoot default service accounts troubleshoot workload to workload authentication ai and ml application development application hosting compute data analytics and pipelines databases distributed hybrid and multicloud industry solutions migration networking observability and monitoring security storage access and resources management costs and usage management infrastructure as code sdk languages frameworks and tools home documentation compute compute engine guides send feedback troubleshoot gpu vms stay organized with collections save and categorize content based on your preferences this guide describes how to diagnose and resolve common issues with compute engine vms that have attached gpus including hardware errors and performance bottlenecks troubleshoot gpu vms by using nvidia dcgm nvidia data center gpu manager dcgm is a suite of tools for managing and monitoring nvidia data center gpus in cluster environments if you want to use dcgm to troubleshoot issues in your gpu environment then complete the following ensure that you are using the latest recommended nvidia driver for the gpu model that is attached to your vm to review driver versions see recommended nvidia driver versions ensure that you installed the latest version of dcgm to install the latest version see dcgm installation diagnose issues when you run a dcgmi diagnostic command the issues reported by the diagnostic tool include next steps for taking action on the issue the following example shows the actionable output from the dcgmi diag r memory j command category hardware tests name gpu memory results gpu_id 0 info gpu 0 allocated 23376170169 bytes 98 3 status fail warnings warning pending page retirements together with a dbe were detected on gpu 0 drain the gpu and reset it or reboot the node to resolve this issue error_id 83 error_category 10 error_severity 6 from the preceding output snippet you can see that gpu 0 has pending page retirements that are caused by a non recoverable error the output provided the unique error_id and advice on debugging the issue for this example output it is recommended that you drain the gpu and reboot the vm in most cases following the instructions in this section of the output can help to resolve the issue pro tip take note of the error severity in the example output a value of error_severity 6 corresponds to a dcgm_error_reset which means that a reset resolves issues with this severity value for a full list of error_severity values review the dcgmerrorseverity_enum section on the dcgm_errors github file troubleshoot gpu performance issues for a3 vms the a3 machine series is available with nvidia h200 or h100 gpus attached this series includes a3 ultra h200 a3 mega h100 a3 high h100 and a3 edge h100 machine types identify a faulty node large scale training or benchmark jobs on a multi node gpu cluster can stop responding or perform poorly this often occurs because one or more nodes underperform and slow down the entire operation this section describes how to identify a faulty node or host machine by either running an nccl benchmark test or analyzing nccl logs run nccl benchmark test to identify the group of nodes causing the failure systematically test subsets of your cluster by using nccl benchmarks like all_reduce_perf to identify your nodesets group your nodes into logical sets for example partitions in slurm to create hostfiles create a separate hostfile for each nodeset listing hostnames and the number of gpus per node the number of slots you specify depends on the gpu count of your a3 vm type for example a3 highgpu 8g vms have 8 gpus so you must specify slots 8 to run benchmarks execute the all_reduce_perf benchmark against each nodeset individually mpirun x ld_library_path hostfile hostfile_name n total_processes build all_reduce_perf b 1g e 8g f 2 g num_gpus_per_node replace the following hostfile_name the name of the hostfile that contains the list of nodes and the number of gpus per node for the nodeset total_processes the total number of mpi processes to launch across all hosts in the nodeset num_gpus_per_node the number of gpus per node for all a3 machine types this value is 8 to analyze results if a job hangs or shows significantly lower bus bandwidth busbw on a particular nodeset that set is likely faulty to sub divide if a nodeset is faulty divide its hostfile in half and re test to narrow down the binary search until you pinpoint the individual misbehaving node analyze nccl logs if the benchmark method doesn t pinpoint a node analyze detailed nccl logs to enable debug logging set the following environment variables in the shell session where you plan to run your workload export nccl_debug info export nccl_debug_subsys init net coll export nccl_debug_file log_directory nccl_log h p replace log_directory with the directory where you want to store your logs setting nccl_debug_file with h and p creates unique non interleaved log files for each process if you are running a multi node workload using mpirun you must propagate these variables to all nodes by using the x flag for example mpirun x nccl_debug x nccl_debug_subsys x nccl_debug_file to find the first error use the following command to find the earliest timeout or failure events across all log files grep nccl warn net fastrak log_directory sed s net fastrak 1 g sort head n 20 replace log_directory with the directory where your logs are stored to count collective operations a straggler node completes fewer collective operations count opcount entries for suspect ranks grep opcount log_directory nccl_log hostname pid wc l replace the following log_directory the directory where your logs are stored hostname the hostname of the node pid the process id of the nccl process to gather more logging data before a job aborts temporarily increase the data transfer timeout export nccl_fastrak_data_transfer_timeout_ms 3600000 monitor gpu thermal throttling a3 series vms can experience performance degradation if they consistently reach temperatures greater than 87 c under load to check for gpu thermal throttling across nodes in a cluster use either nvidia smi or dcgmi using nvidia smi to check the current temperature and throttling status of all gpus on a node run the following command nvidia smi query gpu timestamp name pci bus_id temperature gpu clocks_throttle_reasons hw_slowdown format csv in the output a value of active in the clocks_throttle_reasons hw_slowdown column indicates the gpu is being throttled due to high temperatures using dcgmi the nvidia data center gpu manager dcgm diagnostic suite includes checks for thermal violations to run a level 1 diagnostic run the following command dcgmi diag r 1 a result of warn or fail in the thermal section indicates a thermal violation occurred during the test if a thermal violation is accompanied by clock throttling the gpu is likely overheating and requires further investigation open a support case if you re unable to resolve the issues by using the guidance on this page gather the following information and open a support case project id and a list of all instance names or ids in the cluster list of suspect nodes identified through troubleshooting complete non interleaved nccl logs with debug settings enabled output from hardware health checks dcgmi nvidia smi exact benchmark or workload command that is failing relevant log files such as host engine and diagnostic logs to gather these run gather dcgm logs sh located in usr local dcgm scripts in default installations nvidia bug report run nvidia bug report sh for blackwell gpus follow generate nvidia bug report for blackwell gpus details about any recent changes that were made to your environment preceding the failure review xid messages after you create a vm that has attached gpus you must install nvidia device drivers on your gpu vms so that your applications can access the gpus however sometimes these drivers return error messages an xid message is an error report from the nvidia driver that is printed to the operating system s kernel log or event log for your linux vm these messages are placed in the var log messages file for more information about xid messages including potential causes see nvidia documentation the following section provides guidance on handling some xid messages grouped by the most common types gpu memory errors gpu system processor gsp errors and illegal memory access errors gpu memory errors gpu memory is the memory that is available on a gpu that can be used for temporary stor...
|
