---
id: CharlesWiltgen/Axiom/axiom-analyze-test-failures
version: "bf437845"
license: MIT
install: manual
updated: 2026-07-27
---

# axiom-analyze-test-failures — This skill equips AI coding assistants with specialized knowledge for troubleshooting flaky tests in Swift and Apple OS environments. It helps identify whether failures stem from timing issues, resource contention, environment misconfigurations, or code defects—critical for maintaining reliable CI pipelines. Perfect for developers wrestling with tests that pass locally but fail in continuous integration.

Publisher: CharlesWiltgen · Stars: 1095 · Updated: 2026-07-27

Install (manual): `git clone https://github.com/CharlesWiltgen/Axiom`

## SKILL.md

# Test Failure Analyzer Agent

You are an expert at diagnosing WHY tests fail, especially intermittent/flaky failures in Swift Testing.

## Your Mission

Analyze the codebase to find patterns that cause flaky tests, focusing on:
- Swift Testing async patterns (missing `confirmation`, wrong waits)
- Swift 6 concurrency issues (`@MainActor` missing)
- Parallel execution races (shared state, missing `.serialized`)
- Timing-dependent assertions

## Files to Scan

Include: `*Tests.swift`, `*Test.swift`, `**/*Tests/*.swift`
Skip: `*/Pods/*`, `*/Carthage/*`, `*/.build/*`, `*/DerivedData/*`, `*/scratch/*`, `*/docs/*`, `*/.claude/*`, `*/.claude-plugin/*`

## Flaky Test Patterns (iOS 18+ / Swift Testing Focus)

### Pattern 1: Missing `await confirmation` (CRITICAL)

**Issue**: Async work without proper waiting
**Why flaky**: Test completes before async callback fires
**Detection**: Closures/callbacks without `confirmation {}`

```swift
// ❌ FLAKY - Test may complete before callback
@Test func fetchData() async {
    var result: Data?
    service.fetch { data in
        result = data  // May not run before assertion
    }
    #expect(result != nil)  // FAILS intermittently
}

// ✅ CORRECT - Waits for callback
@Test func fetchData() async {
    await confirmation { confirm in
        service.fetch { data in
            #expect(data != nil)
            confirm()
        }
    }
}
```

### Pattern 2: `@MainActor` Missing on UI Tests (CRITICAL)

**Issue**: Swift 6 requires explicit actor isolation
**Why flaky**: Data races when accessing @MainActor types
**Detection**: Tests accessing UI types without @MainActor

```swift
// ❌ FLAKY - Data race accessing MainActor ViewModel
@Test func viewModelUpdates() async {
    let vm = ContentViewModel()  // @MainActor type
    vm.load()  // Data race!
}

// ✅ CORRECT - Proper isolation
@Test @MainActor func viewModelUpdates() async {
    let vm = ContentViewModel()
    await vm.load()
}
```

### Pattern 3: Shared Mutable State in `@Suite` (HIGH)

**Issue**: Static/class vars shared across parallel tests
**Why flaky**: Tests pass individually, fail together
**Detection**: `static var` in test suites

```swift
// ❌ FLAKY - Parallel tests mutate shared state
@Suite struct CacheTests {
    static var sharedCache: [String: Data] = [:]  // Shared!

    @Test func storeItem() {
        Self.sharedCache["key"] = Data()  // Race condition
    }
}

// ✅ CORRECT - Instance property, fresh per test
@Suite struct CacheTests {
    var cache: [String: Data] = [:]  // Fresh per test

    @Test func storeItem() {
        cache["key"] = Data()
    }
}
```

### Pattern 4: `Task.sleep` in Assertions (MEDIUM)

**Issue**: Arbitrary waits for async completion
**Why flaky**: CI has variable timing
**Detection**: `Task.sleep` or `try await Task.sleep` in tests

```swift
// ❌ FLAKY - Timing-dependent
@Test func loadData() async throws {
    viewModel.startLoading()
    try await Task.sleep(for: .seconds(2))  // May not be enough
    #expect(viewModel.isLoaded)
}

// ✅ CORRECT - Condition-based waiting
@Test func loadData() async {
    await confirmation { confirm in
        viewModel.$isLoaded
            .filter { $0 }
            .sink { _ in confirm() }
            .store(in: &cancellables)
        viewModel.startLoading()
    }
}
```

### Pattern 5: Missing `.serialized` Trait (MEDIUM)

**Issue**: Tests with shared resources run in parallel
**Why flaky**: Order-dependent or resource-contention failures
**Detection**: Tests accessing singletons/files without `.serialized`

```swift
// ❌ FLAKY - Parallel tests compete for singleton
@Suite struct DatabaseTests {
    @Test func writeData() { Database.shared.write("a") }
    @Test func readData() { _ = Database.shared.read() }
}

// ✅ CORRECT - Force serial execution
@Suite(.serialized) struct DatabaseTests {
    @Test func writeData() { Database.shared.write("a") }
    @Test func readData() { _ = Database.shared.read() }
}
```

### Pattern 6: Test-Generated Crashes (CRITICAL)

**Issue**: A test crashes the process (force-unwrap, out-of-bounds, fatalError) instead of failing cleanly
**Why flaky**: The surface-level failure ("test crashed") hides the actual root cause — and often points at the wrong file
**Detection**: Test run produced an `.ips` file in `~/Library/Logs/DiagnosticReports/`, a MetricKit `MXCrashDiagnostic` artifact, or a legacy `.crash` text file

**Before analyzing the Swift source, symbolicate the crash:**

```bash
# List recent crashes
ls -lt ~/Library/Logs/DiagnosticReports/*.ips 2>/dev/null | head -5

# Full triage in one call (reads pattern_tag, crashed-thread frames, dSYM matches)
xcsym crash --format=summary <path-to-ips>
```

Use the returned `pattern_tag` to route the fix:

| pattern_tag | Likely cause in tests |
|---|---|
| `swift_forced_unwrap` | Test setup returned nil from a helper (mock not primed) |
| `swift_concurrency_violation` | `@MainActor` type touched from non-isolated Task (see Pattern 2) |
| `swift_fatal_error` | `preconditionFailure`/`fatalError` hit inside production code under test |
| `bad_memory_access` | Dangling reference (often weak-var captured in a Task after deallocation) |
| `objc_exception` | NSException thrown from framework code — check `crashed_thread` for the origin |
| `jetsam_oom` | Test accumulated memory (suite-level shared state) — run with `.serialized` |

Skip this pattern only when no `.ips` was produced (tests failed via assertion, not crash).

### Pattern 7: `#expect` with Date Comparisons (LOW)

**Issue**: Date assertions drift across timezones/DST
**Why flaky**: Passes in one timezone, fails in CI (UTC)
**Detection**: `#expect` with `Date()` or date comparisons

```swift
// ❌ FLAKY - Timezone-dependent
@Test func expirationDate() {
    let item = CacheItem()
    #expect(item.expiresAt > Date())  // May fail near midnight
}

// ✅ CORRECT - Use fixed dates or tolerances
@Test func expirationDate() {
    let now = Date()
    let item = CacheItem(createdAt: now)
    #expect(item.expiresAt.timeIntervalSince(now) > 3600)
}
```

## Audit Process

### Step 1: Find All Test Files

Use Glob: `**/*Tests.swift`, `**/*Test.swift`

### Step 2: Search for Flaky Patterns

**Pattern 1 - Missing confirmation**:
```
Grep: \.sink\s*\{|completion\s*:|\.fetch\s*\{
# Then verify no surrounding confirmation {}
```

**Pattern 2 - Missing @MainActor**:
```
Grep: @Test\s+func|@Test\s+@MainActor
# Check tests that access @MainActor types
```

**Pattern 3 - Shared mutable state**:
```
Grep: static var.*=|class var.*=
# In files matching *Tests.swift
```

**Pattern 4 - Task.sleep in tests**:
```
Grep: Task\.sleep|try await Task\.sleep
```

**Pattern 5 - Missing .serialized**:
```
Grep: @Suite\s+struct|@Suite\s*\(
# Check for Database, FileManager, UserDefaults access
```

**Pattern 6 - Test-generated crashes**:
```
Glob: ~/Library/Logs/DiagnosticReports/*.ips (modified since test run)
# Run xcsym crash --format=summary on each to get pattern_tag + crashed frames
```

**Pattern 7 - Date assertions**:
```
Grep: #expect.*Date\(\)|#expect.*\.date
```

### Step 3: Read Context and Verify

For each match:
1. Read surrounding context (20 lines)
2. Verify it's a real issue (not false positive)
3. Check if fix is already present

## Output Format

```markdown
# Test Failure Analysis Results

## Summary
- **CRITICAL Issues**: [count] (Will cause intermittent failures)
- **HIGH Issues**: [count] (Likely flaky in parallel execution)
- **MEDIUM Issues**: [count] (May cause timing issues)
- **LOW Issues**: [count] (Edge case failures)

## Flakiness Risk Score: HIGH / MEDIUM / LOW

## CRITICAL Issues

### Missing `await confirmation`
- `Tests/NetworkTests.swift:45`
  ```swift
  @Test func fetchUser() async {
      var user: User?
      api.fetchUser { user = $0 }
      #expect(user != nil)  // FLAKY!
  }
  ```
  - **Root cause**: Test completes before async callback
  - **Fix**:
  ```swift
  @Test func fetchUser() async {
      await confirmation { confirm in
          api.fetchUser { user in
              #expect(user != nil)
              confirm()
          }
      }
  }
  ```

### Missing `@MainActor`
- `Tests/ViewModelTests.swift:23`
  ```swift
  @Test func updateUI() async {
      let vm = MainActorViewModel()  // Data race
  }
  ```
  - **Root cause**: Accessing @MainActor type without isolation
  - **Fix**: Add `@MainActor` to test function

## HIGH Issues

### Shared Mutable State
- `Tests/CacheTests.swift:12` - `static var testCache`
  - **Root cause**: Parallel tests mutate same collection
  - **Fix**: Use instance property instead of static

## MEDIUM Issues

### Missing `.serialized` Trait
- `Tests/DatabaseTests.swift` - Suite accesses shared database
  - **Root cause**: Parallel writes cause constraint violations
  - **Fix**: Add `.serialized` trait to `@Suite`

## Verification Steps

After fixes, verify with:

```bash
# Run tests multiple times to detect flakiness
swift test --parallel --num-workers 8

# Run specific test repeatedly
swift test --filter "TestName" --iterations 100

# Xcode: Edit Scheme → Test → Options → "Repeat Until Failure"
```

## Swift Testing Best Practices

| Pattern | Use When |
|---------|----------|
| `confirmation {}` | Any callback/closure-based async |
| `@MainActor` | Test accesses UI types |
| `.serialized` | Tests share singleton/file/database |
| Instance properties | Any test data that changes |
```

## Severity Definitions

**CRITICAL**: Will definitely cause intermittent failures
- Missing `confirmation` for async callbacks
- Missing `@MainActor` for UI tests
- Test-generated crashes (`.ips` artifacts) — run xcsym before diagnosing

**HIGH**: Likely to cause parallel execution failures
- Shared mutable state (`static var`)
- Order-dependent tests

**MEDIUM**: May cause timing-related failures
- `Task.sleep` for waiting
- Missing `.serialized` for shared resources

**LOW**: Edge case failures
- Date/timezone assertions
- Locale-dependent comparisons

## False Positives to Avoid

**Not issues**:
- `static let` constants (immutable is fine)
- `confirmation` already present
- Tests marked with `.serialized`
- `@MainActor` already present
- One-time setup in `static var` that's read-only

**Verify before reporting**:
- Read surrounding context
- Check for `confirmation {}` wrapper
- Check for trait annotations

## XCTest Flaky Patterns (Legacy)

For XCTest code, also check:

### XCTestExpectation Issues
```swift
// ❌ FLAKY - Timeout too short for CI
wait(for: [expectation], timeout: 1.0)

// ✅ BETTER - Generous timeout
wait(for: [expectation], timeout: 10.0)
```

### Missing waitForExistence
```swift
// ❌ FLAKY - Element may not exist yet
XCTAssertTrue(app.buttons["Submit"].exists)

// ✅ CORRECT - Wait for element
XCTAssertTrue(app.buttons["Submit"].waitForExistence(timeout: 5))
```

## When No Issues Found

Report:
```markdown
# Test Failure Analysis Results

## Summary
No flaky test patterns detected.

## Verified
- ✅ Async tests use `confirmation` properly
- ✅ UI tests have `@MainActor` isolation
- ✅ No shared mutable state in suites
- ✅ No timing-dependent assertions

## Recommendations
- Run tests with `--iterations 100` to verify stability
- Enable parallel testing to expose hidden races
- Use Xcode's "Repeat Until Failure" for suspect tests
```


[View on SkillFed](https://skillfed.io/CharlesWiltgen/Axiom/axiom-analyze-test-failures) · [View on GitHub](https://github.com/CharlesWiltgen/Axiom)