3dgs-training-debugger
Troubleshoot 3D Gaussian Splatting training problems with targeted diagnostics for numerical instability, memory exhaustion, convergence failures, and rendering defects. This skill helps identify root causes and apply fixes to get your 3DGS models training smoothly again.
3dgs-training-debugger helps diagnose NaN failures by checking learning rate schedules, gradient clipping thresholds, and initialization ranges. Start by reducing your initial learning rate by 50%, enable gradient clipping (typical range 0.0–1.0), and verify that camera poses and image data are normalized. If NaNs persist, inspect your densification triggers—over-aggressive Gaussian creation can destabilize gradients. Review loss weighting across RGB, SH, and regularization terms to ensure no component dominates.
AI-generated summary based on this skill's SKILL.md
Install
jaccen/Awesome-Gaussian-Skills/3dgs-training-debugger · repository language: TypeScript
git clone https://github.com/jaccen/Awesome-Gaussian-Skills
cp -r Awesome-Gaussian-Skills/skills/3dgs-training-debugger ~/.claude/skills/3dgs-training-debuggerFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I fix 3DGS training NaN loss?
3dgs-training-debugger helps diagnose NaN failures by checking learning rate schedules, gradient clipping thresholds, and initialization ranges. Start by reducing your initial learning rate by 50%, enable gradient clipping (typical range 0.0–1.0), and verify that camera poses and image data are normalized. If NaNs persist, inspect your densification triggers—over-aggressive Gaussian creation can destabilize gradients. Review loss weighting across RGB, SH, and regularization terms to ensure no component dominates.
What causes CUDA out of memory errors in 3DGS training?
3dgs-training-debugger addresses OOM crashes by analyzing batch size, image resolution, and Gaussian count growth. Reduce batch size or image resolution first; even halving resolution frees significant VRAM. Monitor Gaussian population during training—if it grows unchecked, tighten densification thresholds or cap the maximum Gaussian count. Profile memory per training step to isolate whether the bottleneck is model parameters, gradients, or intermediate activations.
How can I resolve training divergence in 3D gaussian splatting?
3dgs-training-debugger tackles divergence by examining learning rate decay, optimizer state, and checkpoint integrity. Verify your learning rate schedule doesn't drop too aggressively; a cosine or exponential decay is often more stable than step-based. If resuming from checkpoint, confirm optimizer momentum and adaptive learning rates (Adam's m/v buffers) are loaded correctly. Reduce densification frequency or disable it temporarily to isolate whether Gaussian creation is triggering instability.
What fixes 3DGS floaters and artifacts during rendering?
3dgs-training-debugger identifies artifact sources like over-densified regions, poor SH coefficient initialization, or opacity collapse. Check whether floaters cluster near scene boundaries—this suggests densification is creating Gaussians in low-information regions. Increase regularization weight on opacity to prevent dead Gaussians. Validate that your SH degree matches scene complexity; too low a degree causes banding, too high can overfit. Review camera pose accuracy, as misaligned views cause ghosting.
How do I optimize VRAM usage and resolve out-of-memory crashes?
3dgs-training-debugger provides VRAM optimization by profiling memory allocation across stages. Batch size is the primary lever—halving it often cuts memory by 40–50%. Reduce image resolution or use progressive resolution training. Implement gradient checkpointing to trade compute for memory. Cap Gaussian count explicitly and monitor densification triggers. For multi-GPU setups, ensure data parallelism doesn't replicate full model state; use gradient accumulation to simulate larger batches without proportional memory cost.
What should I tune to improve 3DGS convergence and final quality?
3dgs-training-debugger guides hyperparameter tuning by prioritizing learning rate, densification schedule, and regularization. Start with learning rate 0.0025 for position and 0.0005 for SH; adjust based on loss curve smoothness. Densification frequency (e.g., every 100 steps) and thresholds control Gaussian growth—too frequent causes instability, too sparse leaves artifacts. Balance RGB loss against opacity and scale regularization; typical weights are 1.0 RGB, 0.2 opacity, 0.01 scale. Monitor validation metrics every 1000 steps to catch overfitting early.
SKILL.md
rendered from the published skill — quoted content, verbatim
name: 3dgs-training-debugger description: "Diagnose and fix 3DGS training-time failures: NaN losses, OOM crashes, divergent optimization, floater artifacts, densification failures, hyperparameter sensitivity. Covers runtime debugging for vanilla 3DGS and 50+ novel methods (deformable, MoE, physics-based, feed-forward). Detects 50+ runtime failure patterns. Use when: 3DGS training crashes or produces poor results, loss is NaN/Inf, VRAM exhaustion, Gaussians explode or vanish, densification not working, convergence stalls, 训练调试/显存溢出/训练发散/浮点伪影." license: Apache-2.0 user-invocable: true metadata: version: "1.0.0" author: jaccen tags: ["3dgs", "gaussian-splatting", "training", "debugging", "oom", "convergence", "hyperparameters", "distributed"] when_to_use: - "3DGS training crashes with OOM or CUDA error" - "Training loss becomes NaN or Inf" - "Gaussian count explodes or vanishes during training" - "Densification is not triggering or over-triggering" - "Training converges slowly or plateaus prematurely" - "Visual artifacts appear (floaters, blur, ghosting) after training" - "Checkpoint resume produces different results" - "Distributed
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 4 files
skills/3dgs-training-debugger/SKILL.md
skills/3dgs-training-debugger/references/convergence-trajectories.md
skills/3dgs-training-debugger/references/runtime-bug-patterns.md
skills/3dgs-training-debugger/references/vram-gpu-table.md