Meta tags:
description= Resolve issues with GPU VMs;
Headings (most frequently used words):
gpu, troubleshoot, vms, using, errors, and, nvidia, issues, support, xid, messages, nccl, memory, stay, organized, with, collections, save, categorize, content, based, on, your, preferences, by, dcgm, performance, for, a3, open, case, review, reset, gpus, what, next, diagnose, identify, faulty, node, run, benchmark, test, analyze, logs, monitor, thermal, throttling, smi, dcgmi, gsp, illegal, access, other, common, error, products, pricing, resources, engage,
Text of the page (most frequently used words):
the (181), and (135), create (119), vms (115), for (101), with (87), #troubleshoot (67), gpu (62), about (61), instance (50), manage (46), using (44), disk (43), server (39), view (38), disks (35), use (34), that (34), your (34), nvidia (33), performance (33), windows (33), mig (33), cloud (32), gpus (32), configure (32), overview (31), xid (30), compute (30), set (29), you (28), google (27), reservation (27), machine (26), engine (26), from (25), access (24), workloads (22), error (21), monitor (21), hyperdisk (21), sql (21), node (20), images (20), linux (19), logs (19), network (19), this (18), errors (18), run (18), following (18), cluster (18), resources (17), delete (17), reset (17), memory (17), workload (17), image (17), policies (16), host (16), reservations (16), instances (16), attached (15), stop (15), load (15), snapshots (15), groups (15), ssh (15), recommendations (15), configuration (15), issues (14), management (14), creation (14), virtual (14), apply (14), custom (14), connect (14), are (13), data (13), storage (13), benchmark (13), bulk (13), microsoft (13), all (12), type (12), sole (12), availability (12), persistent (12), best (12), regional (12), nodes (11), can (11), application (11), high (11), based (11), practices (11), stateful (11), install (10), metadata (10), migs (10), machines (10), migrate (10), add (10), zones (10), page (9), types (9), then (9), file (9), dcgm (9), ecc (9), recovery (9), files (9), output (9), nccl (9), across (9), number (9), licenses (9), managed (9), tenant (9), multiple (9), openshift (9), shutdown (9), time (9), start (9), hpc (9), license (8), review (8), issue (8), message (8), messages (8), dcgmi (8), manager (8), applications (8), enable (8), group (8), scale (8), service (8), import (8), cpu (8), resource (8), login (8), optimize (8), internal (8), choose (8), disaster (8), agent (8), existing (8), update (8), guest (8), boot (8), keys (8), h4d (8), flexibility (8), dns (8), optimized (8), tpu (8), events (7), thumb (7), information (7), case (7), directory (7), smi (7), example (7), driver (7), hardware (7), test (7), report (7), default (7), available (7), drivers (7), command (7), list (7), transfer (7), operations (7), multi (7), networking (7), commitments (7), startup (7), balancing (7), manually (7), idle (7), addresses (7), request (7), deploy (7), back (7), mysql (7), zonal (7), remove (7), maintenance (7), templates (7), snapshot (7), backup (7), center (6), code (6), support (6), see (6), down (6), more (6), details (6), send (6), recreate (6), customer (6), name (6), identify (6), single (6), highly (6), operating (6), log (6), environment (6), diagnostic (6), scripts (6), non (6), thermal (6), series (6), export (6), log_directory (6), nodeset (6), build (6), monitoring (6), usage (6), security (6), migration (6), updates (6), instant (6), nested (6), mode (6), demand (6), pools (6), failover (6), extension (6), change (6), new (6), customized (6), spot (6), samples (5), system (5), understand (5), after (5), care (5), 0x00000000 (5), common (5), illegal (5), resolve (5), gsp (5), bug (5), row (5), process (5), get (5), failure (5), h100 (5), section (5), local (5), health (5), active (5), analyze (5), faulty (5), bandwidth (5), tools (5), databases (5), accounts (5), future (5), virtualization (5), placement (5), modify (5), capacity (5), schedules (5), audit (5), symphony (5), options (5), email (5), between (5), systems (5), enhanced (5), capabilities (5), tenancy (5), restore (5), regions (5), control (5), workstation (5), faq (5), flex (5), português (4), español (4), other (4), need (4), persists (4), move (4), specify (4), have (4), reboot (4), some (4), complete (4), such (4), has (4), diag (4), debug (4), bit (4), retirement (4), together (4), event (4), enabled (4), these (4), gather (4), troubleshooting (4), project (4), during (4), throttling (4), value (4), query (4), check (4), increase (4), hostname (4), where (4), replace (4), net (4), plan (4), running (4), hostfile (4), per (4), perform (4), how (4), pro (4), distributed (4), authentication (4), ubuntu (4), console (4), tcp (4), pmu (4), ssd (4), share (4), projects (4), upgrade (4), live (4), autoscaling (4), cross (4), region (4), requests (4), calendar (4), metrics (4), clusters (4), containers (4), container (4), database (4), uses (4), postgresql (4), extensions (4), work (4), zone (4), suspend (4), location (4), resize (4), graceful (4), locations (4), plans (4), replication (4), bound (4), action (3), terms (3), status (3), content (3), under (3), apache (3), next (3), repair (3), tool (3), only (3), target (3), located (3), kubernetes (3), most (3), one (3), indicates (3), bus (3), resolution (3), returned (3), caused (3), out (3), when (3), timeout (3), remap (3), pending (3), contains (3), documentation (3), must (3), changes (3), were (3), blackwell (3), through (3), names (3), format (3), logging (3), count (3), setting (3), each (3), mpirun (3), nccl_debug_file (3), info (3), method (3), all_reduce_perf (3), full (3), error_severity (3), recommended (3), latest (3), versions (3), guide (3), guides (3), infrastructure (3), observability (3), analytics (3), automatic (3), encryption (3), serial (3), configurations (3), patterns (3), higher (3), discounts (3), cuds (3), manual (3), scalable (3), resilient (3), autoscale (3), balancer (3), prevent (3), policy (3), testing (3), ibm (3), spectrum (3), deploying (3), prepare (3), deployment (3), rhel (3), protect (3), certificates (3), state (3), disable (3), properties (3), account (3), schedule (3), lifecycle (3), design (3), size (3), pool (3), creating (3), workstations (3), tpus (3), product (3), 한국어 (2), 日本語 (2), עברית (2), brasil (2), italiano (2), indonesia (2), français (2), américa (2), latina (2), deutsch (2), english (2), sign (2), site (2), youtube (2), training (2), started (2), github (2), release (2), pricing (2), products (2), last (2), updated (2), 2026 (2), utc (2), licensed (2), its (2), feedback (2), into (2), resetting (2), restart (2), gke (2), automate (2), requires (2), executable (2), home (2), bin (2), var (2), than (2), might (2), require (2), steps (2), pci (2), 0x02a (2), 149 (2), means (2), off (2), cases (2), degradation (2), different (2), cuda (2), bounds (2), level (2), processor (2), corrects (2), affect (2), reserved (2), recording (2), double (2), a100 (2), dynamic (2), which (2), detects (2), reports (2), dbe (2), guidance (2), including (2), kernel (2), device (2), preceding (2), generate (2), checks (2), settings (2), interleaved (2), suspect (2), open (2), violation (2), likely (2), fail (2), warn (2), suite (2), includes (2), temperatures (2), clocks_throttle_reasons (2), hw_slowdown (2), temperature (2), either (2), job (2), collective (2), pid (2), stored (2), grep (2), opcount (2), nccl_log (2), find (2), fastrak (2), variables (2), unique (2), nccl_debug (2), nccl_debug_subsys (2), want (2), pinpoint (2), divide (2), results (2), shows (2), benchmarks (2), num_gpus_per_node (2), total_processes (2), hostfile_name (2), slots (2), slurm (2), operation (2), describes (2), h200 (2), ultra (2), severity (2), retirements (2), drain (2), help (2), error_id (2), diagnose (2), ensure (2), version (2), sdk (2), languages (2), frameworks (2), costs (2), industry (2), solutions (2), hybrid (2), multicloud (2), pipelines (2), hosting (2), development (2), quota (2), commitment (2), consumption (2), standard (2), registration (2), sles (2), port (2), collecting (2), arm (2), core (2), api (2), rdp (2), general (2), app (2), latency (2), visible (2), cores (2), automatically (2), committed (2), generation (2), underutilized (2), insights (2), utilization (2), globally (2), web (2), autohealing (2), iis (2), external (2), shared (2), reserve (2), autoscaler (2), autoscalers (2), activity (2), provider (2), place (2), services (2), redis (2), cloning (2), writer (2), always (2), alwayson (2), volumes (2), authenticate (2), synchronization (2), global (2), switch (2), byol (2), payg (2), byos (2), families (2), securing (2), simulate (2), topology (2), repairs (2), failures (2), template (2), suspended (2), stopped (2), distribution (2), physical (2), cancel (2), once (2), limit (2), interfaces (2), ipv6 (2), static (2), decrease (2), basic (2), consistent (2), scoped (2), copy (2), appliances (2), asynchronous (2), throughput (2), encrypt (2), mount (2), extreme (2), balanced (2), credentials (2), organization (2), user (2), advanced (2), rdma (2), created (2), specific (2), preemptible (2), provisioning (2), models (2), rtx (2), vws (2), constraints (2), accelerator (2), apis (2), reference (2), technology (2), areas (2), close (2), subscribe, newsletter, our, third, decade, climate, join, cookies, privacy, tech, twitter, blog, engage, certification, architecture, getting, notes, community, forums, contact, sales, marketplace, easy, easytounderstand, solved, problem, solvedmyproblem, otherup, hard, hardtounderstand, incorrect, sample, incorrectinformationorsamplecode, missing, missingtheinformationsamplesineed, otherdown, tell, except, otherwise, noted, java, registered, trademark, oracle, affiliates, developers, creative, commons, attribution, what, stage, persist, alternatively, also, whenever, lib, sudo, less, mentions, known, affecting, firmware, b200, 0000, netir_link_evt, fatal, xc0, link, 485c6, not, able, communicate, fallen, nvlink, very, rare, cause, levels, coverage, duration, gdb, memcheck, typically, trying, already, freed, dereferencing, invalid, pointer, array, fault, graphics, exception, xids, 120, collect, 119, rpc, microcontroller, runs, handles, low, functions, uncontained, contained, correctable, shouldn, informational, needed, rate, allows, heal, least, two, rows, bank, remapped, remapper, remapping, their, suggested, resolutions, prior, was, supported, later, releases, introduced, recommends, keeping, used, temporary, protected, correction, sbe, provides, handling, grouped, potential, causes, printed, placed, however, sometimes, return, any, recent, made, follow, installations, usr, relevant, failing, exact, identified, ids, unable, result, occurred, accompanied, clock, overheating, further, investigation, violations, column, being, throttled, due, timestamp, bus_id, csv, current, experience, they, consistently, reach, greater, before, aborts, temporarily, nccl_fastrak_data_transfer_timeout_ms, 3600000, straggler, completes, fewer, entries, ranks, first, earliest, sed, sort, head, shell, session, creates, propagate, flag, store, init, coll, doesn, detailed, sub, half, narrow, binary, search, until, individual, misbehaving, hangs, significantly, lower, particular, busbw, execute, against, individually, total, mpi, processes, launch, hosts, ld_library_path, hostfiles, separate, listing, hostnames, depends, highgpu, nodesets, logical, sets, partitions, causing, systematically, subsets, like, large, jobs, responding, poorly, often, occurs, because, underperform, slow, entire, analyzing, edge, mega, values, dcgm_errors, dcgmerrorseverity_enum, take, note, corresponds, resolves, dcgm_error_reset, tip, snippet, recoverable, provided, advice, debugging, instructions, category, tests, gpu_id, allocated, 23376170169, bytes, warnings, error_category, warning, detected, reported, include, taking, actionable, installed, installation, model, managing, environments, bottlenecks, save, categorize, preferences, stay, organized, collections, concurrent, renewal, nvme, resizing, pay, viewing, sudoers, screenshots, soft, lockups, locks, rescue, inaccessible, dumps, panic, fstab, unresponsive, suspension, reboots, shutdowns, connectivity, tips, nics, dpdk, improve, resiliency, communication, reduce, compact, idpf, interface, irdma, nic, customize, threads, sustained, merge, split, extend, renew, without, purchase, cost, savings, overcommit, cpus, floating, reliable, udp, backends, routing, https, scaling, consuming, consume, combine, cud, allow, sharing, decisions, signals, serving, predictions, organize, labels, replica, states, observe, tensorflow, inference, tensorrt5, learning, monte, carlo, methods, spark, forwarding, others, integrate, clustering, exchange, 2016, sharepoint, strategies, passive, inactive, setups, built, integrations, validation, deployed, transition, hammerdb, bucket, aws, ec2, pacemaker, s2d, block, netapp, hot, standby, drbd, architectures, client, private, address, mailjet, mailgun, sendgrid, sending, blue, green, deployments, lamp, joomla, asp, interactive, mongodb, flask, terraform, servers, over, mtls, random, generator, virtio, rng, accurate, protocol, ntp, append, els, restrictions, packages, deprecate, family, trusted, red, hat, knowledgebase, functionality, risks, vpc, controls, confidential, kek, secure, expiration, shielded, attributes, predefined, handle, notices, unmanaged, preserved, turn, alternate, fails, repairing, maintain, click, upgrades, override, selectively, applying, accelerate, outage, rebalance, reenable, proactive, redistribution, shape, distribute, define, accidental, deletion, spread, edit, rename, resume, referrers, source, uuid, recover, corrupted, duplicate, clones, alerts, backups, vss, regionally, protection, considerations, synchronous, consistency, failback, replicate, make, provisioned, iops, limits, evaluate, kms, supplied, symbolic, links, practice, ram, attach, additional, exapools, verify, identity, ptr, record, rates, tags, restrict, password, powershell, sac, securely, apps, root, vpn, bastion, iap, connection, browser, connections, networks, within, iso, imported, prerequisites, importing, exporting, bring, own, path, detach, reattach, constraint, accounting, provision, examples, team, slice, slices, adds, scenarios, deterministic, base, requirements, ready, aci, configured, preempted, historical, alternative, similar, displays, ops, subnet, public, gpudirect, minimum, platform, quickstarts, iam, roles, permissions, office, licensing, premium, strategy, gemini, bare, metal, accelerators, platforms, scores, purpose, discover, free, skip, main,
Text of the page (random words):
figure sql server on google cloud using google cloud netapp volumes use file storage to configure sql server failover cluster instance use block storage to configure sql server always on availability groups configure windows set up alwayson availability groups using an internal load balancer set up alwayson availability groups using a distributed network name set up a failover cluster vm that uses s2d set up a failover cluster vm with multi writer disks configure linux set up a sql server cluster on linux with always on availability groups and pacemaker set up a sql server failover cluster on linux with multi writer disks migrate migrate a sql server database from aws ec2 to compute engine migrate a sql server database from windows to linux disaster recovery disaster recovery for microsoft sql server disaster recovery for microsoft sql server on persistent disk disaster recovery for microsoft sql server on hyperdisk deploying microsoft sql server for multi regional disaster recovery back up sql server databases to a google cloud storage bucket back up sql server databases using instant snapshots cloning a microsoft sql server database on compute engine load test sql server using hammerdb redis deployment options for redis on google cloud containers containers on compute engine deploy containers on vms and managed instance groups configure options to run your container transition from the container startup agent prepare for the shutdown of the container startup agent prevent the creation of vms that use the container metadata migrate containers that were deployed on vms during vm creation openshift workloads openshift on google cloud overview plan for openshift on google cloud overview of cluster services for openshift automatic configuration validation for openshift clusters built in integrations for openshift best practices for high availability with openshift disaster recovery for openshift on google cloud disaster recovery strategies for active passive and active inactive setups with openshift microsoft windows windows workloads best practices for windows server vms setting up active directory best practices for running active directory on google cloud deploy microsoft sharepoint server on compute engine deploying microsoft exchange server 2016 on compute engine windows server perform an in place upgrade of windows server run windows server failover clustering ibm spectrum symphony integrate ibm spectrum symphony with google cloud install the compute engine symphony provider install the google kubernetes symphony provider troubleshoot ibm spectrum symphony others load testing distributed load testing using kubernetes ssh port forwarding and load testing analytics monte carlo methods using apache spark machine learning run tensorflow inference workloads with tensorrt5 and nvidia t4 gpu monitor monitor logs view audit logs view usage reports view compute engine operations migrate from activity logs to audit logs view activity logs monitor resources monitor vm and sole tenant node usage observe and monitor vms monitor gpu performance monitor gpu performance on linux vms monitor gpu performance on windows vms monitor disks monitor disk health monitor the replica states of regional disks monitor disks list of metrics for pools monitor pools monitor reservations organize resources using labels scale autoscale groups of vms about autoscaling groups of vms create and manage autoscalers scale based on cpu utilization scale based on predictions scale based on load balancing serving capacity scale based on monitoring metrics scale based on schedules use an autoscaling policy with multiple signals manage autoscalers understand autoscaler decisions view autoscaler logs autoscale node groups reserve vm capacity choose a reservation type sharing reservations best practices for shared reservations allow a project to share reservations on demand reservations about on demand reservations create an on demand reservation for a single project for multiple projects combine an on demand reservation with a cud modify an on demand reservation delete an on demand reservation future reservations about future reservations create a reservation request for a single project for multiple projects modify a reservation request delete a reservation request future reservations in calendar mode about future reservations in calendar mode create a reservation request in calendar mode delete a reservation request in calendar mode view reservations or reservation requests consume a reservation prevent vms from consuming reservations load balancing about load balancing and scaling add an instance group to a load balancer request routing to a multi region external https load balancer cross region load balancing for microsoft iis backends set up internal tcp udp load balancing build reliable and scalable applications use autohealing for highly available applications use load balancing for highly available applications use autoscaling for highly scalable applications globally autoscale a web service on compute engine patterns for scalable and resilient applications patterns for using floating ip addresses on compute engine optimize resource utilization use recommendations to manage resources apply machine type recommendations to vms configure machine type recommendations apply machine type recommendations to migs view and apply idle resources recommendations view and understand vm insights view and understand mig insights manage idle vm recommendations idle vm recommendations overview view and apply idle vm recommendations configure idle vm recommendations manage reservation recommendations reservation recommendations overview view and apply idle reservation recommendations view and apply underutilized reservation recommendations configure idle reservation recommendations configure underutilized reservation recommendations overcommit cpus on sole tenant vms manual live migration about manual live migration manually live migrate vms share sole tenant node groups next generation dynamic resource management cost savings get discounts for committed usage about commitments and committed use discounts cuds resource based cuds purchase resource based commitments without attached reservations with attached reservations for os licenses manage resource based commitments renew commitments automatically extend commitment terms merge and split commitments upgrade commitments share resource based cuds across projects get discounts for sustained usage disk performance optimize hyperdisk performance optimize persistent disk performance optimize local ssd performance workload performance set the number of threads per core customize the number of visible cpu cores analyze the cpu performance using the pmu pmu overview enable the pmu in vms manage the pmu in vms network performance network bandwidth use google virtual nic use irdma network driver use idpf network interface configure a vm with higher bandwidth reduce latency by using compact placement policies optimize tcp network communication optimize tcp network performance optimize tcp network resiliency benchmark higher bandwidth vms optimize app latency with load balancing use dpdk to improve network performance network performance and gpu vms networking and gpu machines use higher network bandwidth patterns for using multiple host nics troubleshoot general tips troubleshoot connectivity troubleshoot rdp troubleshoot ssh troubleshoot os login troubleshoot vms troubleshoot vm operations troubleshoot vm creation troubleshoot resource availability errors troubleshoot bulk api vm creation troubleshoot vm reboots and shutdowns troubleshoot vm suspension troubleshoot vm updates troubleshoot unresponsive vms troubleshoot vm startup troubleshoot fstab errors troubleshoot kernel panic collecting core dumps rescue an inaccessible vm troubleshoot cpu bus locks troubleshoot cpu soft lockups troubleshoot vm configurations troubleshoot arm vms troubleshoot gpu vms troubleshoot nvidia gpu errors generate a nvidia bug report for blackwell gpus troubleshoot nested virtualization troubleshoot using vm screenshots troubleshoot sole tenant nodes troubleshoot vm performance issues troubleshoot sudoers files troubleshoot windows vms troubleshoot windows vms collecting diagnostic information troubleshoot using the serial console troubleshoot using the serial console viewing serial port output troubleshoot instance groups troubleshoot managed instance groups migs troubleshoot os management troubleshoot licenses troubleshoot image import and export troubleshooting sles pay as you go registration troubleshooting ubuntu pro registration troubleshoot metadata server troubleshoot metadata server troubleshoot networking issues troubleshoot common networking issues troubleshoot network drivers troubleshoot vm performance issues troubleshoot storage troubleshoot disk creation troubleshoot full disks and disk resizing troubleshoot disk encryption troubleshoot nvme disks troubleshoot instant snapshots troubleshoot standard snapshots troubleshoot reservations and commitments troubleshoot reservation creation troubleshoot reservation consumption troubleshooting reservation monitoring troubleshoot reservation updates troubleshoot future reservation creation and updates troubleshoot automatic commitment renewal troubleshoot quota errors troubleshoot concurrent operation quota errors troubleshoot workload authentication troubleshoot default service accounts troubleshoot workload to workload authentication ai and ml application development application hosting compute data analytics and pipelines databases distributed hybrid and multicloud industry solutions migration networking observability and monitoring security storage access and resources management costs and usage management infrastructure as code sdk languages frameworks and tools home documentation compute compute engine guides send feedback troubleshoot gpu vms stay organized with collections save and categorize content based on your preferences this guide describes how to diagnose and resolve common issues with compute engine vms that have attached gpus including hardware errors and performance bottlenecks troubleshoot gpu vms by using nvidia dcgm nvidia data center gpu manager dcgm is a suite of tools for managing and monitoring nvidia data center gpus in cluster environments if you want to use dcgm to troubleshoot issues in your gpu environment then complete the following ensure that you are using the latest recommended nvidia driver for the gpu model that is attached to your vm to review driver versions see recommended nvidia driver versions ensure that you installed the latest version of dcgm to install the latest version see dcgm installation diagnose issues when you run a dcgmi diagnostic command the issues reported by the diagnostic tool include next steps for taking action on the issue the following example shows the actionable output from the dcgmi diag r memory j command category hardware tests name gpu memory results gpu_id 0 info gpu 0 allocated 23376170169 bytes 98 3 status fail warnings warning pending page retirements together with a dbe were detected on gpu 0 drain the gpu and reset it or reboot the node to resolve this issue error_id 83 error_category 10 error_severity 6 from the preceding output snippet you can see that gpu 0 has pending page retirements that are caused by a non recoverable error the output provided the unique error_id and advice on debugging the issue for this example output it is recommended that you drain the gpu and reboot the vm in most cases following the instructions in this section of the output can help to resolve the issue pro tip take note of the error severity in the example output a value of error_severity 6 corresponds to a dcgm_error_reset which means that a reset resolves issues with this severity value for a full list of error_severity values review the dcgmerrorseverity_enum section on the dcgm_errors github file troubleshoot gpu performance issues for a3 vms the a3 machine series is available with nvidia h200 or h100 gpus attached this series includes a3 ultra h200 a3 mega h100 a3 high h100 and a3 edge h100 machine types identify a faulty node large scale training or benchmark jobs on a multi node gpu cluster can stop responding or perform poorly this often occurs because one or more nodes underperform and slow down the entire operation this section describes how to identify a faulty node or host machine by either running an nccl benchmark test or analyzing nccl logs run nccl benchmark test to identify the group of nodes causing the failure systematically test subsets of your cluster by using nccl benchmarks like all_reduce_perf to identify your nodesets group your nodes into logical sets for example partitions in slurm to create hostfiles create a separate hostfile for each nodeset listing hostnames and the number of gpus per node the number of slots you specify depends on the gpu count of your a3 vm type for example a3 highgpu 8g vms have 8 gpus so you must specify slots 8 to run benchmarks execute the all_reduce_perf benchmark against each nodeset individually mpirun x ld_library_path hostfile hostfile_name n total_processes build all_reduce_perf b 1g e 8g f 2 g num_gpus_per_node replace the following hostfile_name the name of the hostfile that contains the list of nodes and the number of gpus per node for the nodeset total_processes the total number of mpi processes to launch across all hosts in the nodeset num_gpus_per_node the number of gpus per node for all a3 machine types this value is 8 to analyze results if a job hangs or shows significantly lower bus bandwidth busbw on a particular nodeset that set is likely faulty to sub divide if a nodeset is faulty divide its hostfile in half and re test to narrow down the binary search until you pinpoint the individual misbehaving node analyze nccl logs if the benchmark method doesn t pinpoint a node analyze detailed nccl logs to enable debug logging set the following environment variables in the shell session where you plan to run your workload export nccl_debug info export nccl_debug_subsys init net coll export nccl_debug_file log_directory nccl_log h p replace log_directory with the directory where you want to store your logs setting nccl_debug_file with h and p creates unique non interleaved log files for each process if you are running a multi node workload using mpirun you must propagate these variables to all nodes by using the x flag for example mpirun x nccl_debug x nccl_debug_subsys x nccl_debug_file to find the first error use the following command to find the earliest timeout or failure events across all log files grep nccl warn net fastrak log_directory sed s net fastrak 1 g sort head n 20 replace log_directory with the directory where your logs are stored to count collective operations a straggler node completes fewer collective operations count opcount entries for suspect ranks ...
|
