3dgs-training-debugger
Troubleshoot 3D Gaussian Splatting training problems with targeted diagnostics for numerical instability, memory exhaustion, convergence failures, and rendering defects. This skill helps identify root causes and apply fixes to get your 3DGS models training smoothly again.
3dgs-training-debugger helps diagnose NaN failures by checking learning rate schedules, gradient clipping thresholds, and initialization ranges. Start by reducing your initial learning rate by 50%, enable gradient clipping (typical range 0.0–1.0), and verify that camera poses and image data are normalized. If NaNs persist, inspect your densification triggers—over-aggressive Gaussian creation can destabilize gradients. Review loss weighting across RGB, SH, and regularization terms to ensure no component dominates.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-26
3dgs-training-debugger helps diagnose NaN failures by checking learning rate schedules, gradient clipping thresholds, and initialization ranges. Start by reducing your initial learning rate by 50%, enable gradient clipping (typical range 0.0–1.0), and verify that camera poses and image data are normalized. If NaNs persist, inspect your densification triggers—over-aggressive Gaussian creation can destabilize gradients. Review loss weighting across RGB, SH, and regularization terms to ensure no component dominates.
Use it when
- 3dgs-training-debugger addresses OOM crashes by analyzing batch size, image resolution, and Gaussian count growth.
- 3dgs-training-debugger tackles divergence by examining learning rate decay, optimizer state, and checkpoint integrity.
Verify before relying
Read SKILL.md below before installing (4 files). Open directory: indexed for reading, not audited.
Install
jaccen/Awesome-Gaussian-Skills/3dgs-training-debugger · repository language: TypeScript
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I fix 3DGS training NaN loss?
3dgs-training-debugger helps diagnose NaN failures by checking learning rate schedules, gradient clipping thresholds, and initialization ranges. Start by reducing your initial learning rate by 50%, enable gradient clipping (typical range 0.0–1.0), and verify that camera poses and image data are normalized. If NaNs persist, inspect your densification triggers—over-aggressive Gaussian creation can destabilize gradients. Review loss weighting across RGB, SH, and regularization terms to ensure no component dominates.
What causes CUDA out of memory errors in 3DGS training?
3dgs-training-debugger addresses OOM crashes by analyzing batch size, image resolution, and Gaussian count growth. Reduce batch size or image resolution first; even halving resolution frees significant VRAM. Monitor Gaussian population during training—if it grows unchecked, tighten densification thresholds or cap the maximum Gaussian count. Profile memory per training step to isolate whether the bottleneck is model parameters, gradients, or intermediate activations.
How can I resolve training divergence in 3D gaussian splatting?
3dgs-training-debugger tackles divergence by examining learning rate decay, optimizer state, and checkpoint integrity. Verify your learning rate schedule doesn't drop too aggressively; a cosine or exponential decay is often more stable than step-based. If resuming from checkpoint, confirm optimizer momentum and adaptive learning rates (Adam's m/v buffers) are loaded correctly. Reduce densification frequency or disable it temporarily to isolate whether Gaussian creation is triggering instability.
What fixes 3DGS floaters and artifacts during rendering?
3dgs-training-debugger identifies artifact sources like over-densified regions, poor SH coefficient initialization, or opacity collapse. Check whether floaters cluster near scene boundaries—this suggests densification is creating Gaussians in low-information regions. Increase regularization weight on opacity to prevent dead Gaussians. Validate that your SH degree matches scene complexity; too low a degree causes banding, too high can overfit. Review camera pose accuracy, as misaligned views cause ghosting.
How do I optimize VRAM usage and resolve out-of-memory crashes?
3dgs-training-debugger provides VRAM optimization by profiling memory allocation across stages. Batch size is the primary lever—halving it often cuts memory by 40–50%. Reduce image resolution or use progressive resolution training. Implement gradient checkpointing to trade compute for memory. Cap Gaussian count explicitly and monitor densification triggers. For multi-GPU setups, ensure data parallelism doesn't replicate full model state; use gradient accumulation to simulate larger batches without proportional memory cost.
What should I tune to improve 3DGS convergence and final quality?
3dgs-training-debugger guides hyperparameter tuning by prioritizing learning rate, densification schedule, and regularization. Start with learning rate 0.0025 for position and 0.0005 for SH; adjust based on loss curve smoothness. Densification frequency (e.g., every 100 steps) and thresholds control Gaussian growth—too frequent causes instability, too sparse leaves artifacts. Balance RGB loss against opacity and scale regularization; typical weights are 1.0 RGB, 0.2 opacity, 0.01 scale. Monitor validation metrics every 1000 steps to catch overfitting early.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
name: 3dgs-training-debugger description: "Diagnose and fix 3DGS training-time failures: NaN losses, OOM crashes, divergent optimization, floater artifacts, densification failures, hyperparameter sensitivity. Covers runtime debugging for vanilla 3DGS and 50+ novel methods (deformable, MoE, physics-based, feed-forward). Detects 50+ runtime failure patterns. Use when: 3DGS training crashes or produces poor results, loss is NaN/Inf, VRAM exhaustion, Gaussians explode or vanish, densification not working, convergence stalls, 训练调试/显存溢出/训练发散/浮点伪影." license: Apache-2.0 user-invocable: true metadata: version: "1.0.0" author: jaccen tags: ["3dgs", "gaussian-splatting", "training", "debugging", "oom", "convergence", "hyperparameters", "distributed"] when_to_use: - "3DGS training crashes with OOM or CUDA error" - "Training loss becomes NaN or Inf" - "Gaussian count explodes or vanishes during training" - "Densification is not triggering or over-triggering" - "Training converges slowly or plateaus prematurely" - "Visual artifacts appear (floaters, blur, ghosting) after training" - "Checkpoint resume produces different results" - "Distributed
(truncated - see the full file via the links below)
File tree — 4 files
skills/3dgs-training-debugger/SKILL.md
skills/3dgs-training-debugger/references/convergence-trajectories.md
skills/3dgs-training-debugger/references/runtime-bug-patterns.md
skills/3dgs-training-debugger/references/vram-gpu-table.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Diagnose and fix 3DGS training failures (NaN, OOM, divergence, artifacts)”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill analyzes CUDA implementations of 3D Gaussian Splatting techniques, identifying correctness issues and performance bottlenecks in GPU kernels. It draws on comprehensive knowledge of 3DGS methods to provide targeted feedback on memory access patterns, synchronization, and computational efficiency. Perfect for researchers and engineers optimizing rendering pipelines.
This skill guides you through adapting NeRF-based approaches into 3D Gaussian Splatting implementations. It provides a methodical framework for translating model architectures, training pipelines, and optimization strategies across these two distinct 3D representation paradigms. Use it to modernize existing NeRF work or explore the technical bridge between classical neural radiance fields and contemporary splatting methods.
This skill streamlines research workflows by parsing 3D Gaussian Splatting and NeRF papers to automatically extract key technical metadata. It transforms unstructured academic PDFs into organized, queryable data—eliminating manual note-taking across fragmented sources. Ideal for researchers cataloging methods, tracking innovations, or building comprehensive literature databases.
Transform CAD meshes into 3D Gaussian Splatting representations and vice versa. This skill bridges traditional mesh-based geometry with modern neural 3D graphics, enabling seamless interoperability between formats for rendering, editing, and visualization tasks.
This skill transforms 3D Gaussian Splatting representations into parametric CAD models, enabling designers and engineers to work with extracted geometry in standard modeling tools. By bridging the gap between neural scene representations and traditional CAD workflows, it streamlines the process of converting spatial data into editable, structured models for further design iteration.
This skill streamlines experimental design for 3D Gaussian Splatting research by organizing a comprehensive catalog of 789+ techniques across 25 distinct categories. Researchers can explore proven methodologies, benchmark approaches, and validate experimental frameworks before submission. The interactive explorer helps identify relevant prior work and establish solid foundations for novel contributions.
More skills 3dgs-articulated-reasoner (Apache-2.0) · 3dgs-compression-deploy (Apache-2.0)