Diagnose Hung XCTest Runs on a Cloud Mac with Watchdogs and Process Samples

An XCTest job that normally finishes in a little over ten minutes is still running after forty minutes on a cloud Mac. The final log line only shows that a test case started, CPU usage is low, and the result bundle was never finalized. Stopping the job at this point produces nothing more than a “timeout” result. It does not reveal whether the main thread is deadlocked, an asynchronous callback never returned, the simulator service became unreachable, or the test process is still waiting for a condition that can never be satisfied.

The key to handling this kind of failure is not adding more retries, but preserving the state before termination. Process relationships, repeated thread samples, simulator state, raw logs, and the incomplete xcresult should all belong to the same run record.

Separate slow tests from genuinely hung tests

A single overall timeout can easily produce false positives. Initial compilation, simulator startup, and test execution have different sources of latency, so they should be separated first:

1. Complete compilation with build-for-testing.
2. Run the tests with test-without-building.
3. Record build duration, test duration, and the time of the most recent log change separately.
4. When a test exceeds its historical limit, capture evidence instead of terminating it immediately.

The global limit prevents a job from running indefinitely, while an inactivity limit detects cases where the process is still alive but no longer making progress. Inactivity should not simply mean that standard output is empty: some long-running tests naturally produce very little output. A more reliable approach is to monitor the log file modification time, whether the test process is still alive, and whether the xcresult directory continues to change.

A timeout is a safeguard, not a root cause. It becomes diagnostically useful only when enough evidence is preserved at the moment it occurs.

Establish a stable, reproducible test entry point

Fix the scheme, test plan, destination device, and DerivedData path first. A diagnostic run should not also change Xcode, the simulator runtime, or dependency versions, because that makes the captured states from separate runs impossible to compare reliably.

set -euo pipefail

WORKSPACE="Demo.xcworkspace"
SCHEME="Demo"
PLAN="Regression"
DERIVED="$PWD/.build/DerivedData"
RESULT="$PWD/.artifacts/TestRun.xcresult"

rm -rf "$RESULT"
mkdir -p "$PWD/.artifacts"

xcodebuild build-for-testing \
  -workspace "$WORKSPACE" \
  -scheme "$SCHEME" \
  -testPlan "$PLAN" \
  -destination "platform=iOS Simulator,name=iPhone 16" \
  -derivedDataPath "$DERIVED"

xcodebuild test-without-building \
  -workspace "$WORKSPACE" \
  -scheme "$SCHEME" \
  -testPlan "$PLAN" \
  -destination "platform=iOS Simulator,name=iPhone 16" \
  -derivedDataPath "$DERIVED" \
  -resultBundlePath "$RESULT"

If the problem only occurs in one test class, first narrow the scope with -only-testing:TargetName/TestClass. Do not begin by rerunning the entire test plan in parallel, because that adds process and simulator noise.

Add an evidence-capturing watchdog to the test process

macOS does not provide a universally available GNU timeout command by default, so CI scripts can implement the same behavior with a background watchdog. The correct sequence is: start the test, wait for the limit, capture evidence, send TERM, allow time for cleanup, and only then send KILL.

set -euo pipefail

OUT="$PWD/.artifacts/hang-$(date +%Y%m%d-%H%M%S)"
LIMIT="${TEST_TIMEOUT_SECONDS:-1200}"
mkdir -p "$OUT"

xcodebuild test-without-building \
  -workspace Demo.xcworkspace \
  -scheme Demo \
  -testPlan Regression \
  -destination "platform=iOS Simulator,name=iPhone 16" \
  -derivedDataPath "$PWD/.build/DerivedData" \
  -resultBundlePath "$OUT/TestRun.xcresult" \
  >"$OUT/xcodebuild.log" 2>&1 &

test_pid=$!

(
  sleep "$LIMIT"
  if kill -0 "$test_pid" 2>/dev/null; then
    date -u >"$OUT/timeout-at.txt"
    ps -axo pid,ppid,state,%cpu,%mem,etime,command >"$OUT/host-processes.txt"
    xcrun simctl list devices >"$OUT/simulators.txt"
    xcrun simctl spawn booted ps -axo pid,ppid,state,etime,command \
      >"$OUT/simulator-processes.txt" 2>&1 || true

    for round in 1 2 3; do
      xcrun simctl spawn booted sample DemoTests-Runner 5 1 \
        >"$OUT/test-runner-sample-$round.txt" 2>&1 || true
      sleep 3
    done

    kill -TERM "$test_pid" 2>/dev/null || true
    sleep 10
    kill -KILL "$test_pid" 2>/dev/null || true
  fi
) &

watchdog_pid=$!

set +e
wait "$test_pid"
status=$?
set -e

kill "$watchdog_pid" 2>/dev/null || true
wait "$watchdog_pid" 2>/dev/null || true
exit "$status"

Update the test Runner name to match the project artifact. If the name is not stable, locate the PID in the simulator process list by filtering for the test bundle path, then pass that PID to sample. Three short samples make it easier than one long sample to determine whether the call stack has truly stopped advancing.

Identify the wait type from thread stacks

Start by checking whether the main thread and test thread remain at the same locations in all three samples. Common patterns include:

The main thread is waiting for the test thread

If the main thread is blocked on a semaphore, condition variable, or synchronous dispatch while the test thread is waiting for a callback that must run on the main thread, the likely cause is a deadlock created by synchronously waiting for an asynchronous API. The fix is to use an XCTest expectation and ensure the waiting operation does not block the queue responsible for delivering the callback.

The test thread is waiting for an external condition

If the call stack remains blocked on a socket, file coordination, database lock, or interprocess communication, check dependent services, temporary-directory permissions, and locks left behind by earlier tests. Do not attribute every wait to the network. Use process and file state to establish what the thread is actually waiting for.

The call stack is still changing

If the hotspots change across the three samples, the job may simply be slow. In that case, record per-test-case durations, reduce the data set, or define a reasonable expectation timeout for the test itself instead of increasing the global limit indefinitely.

Archive xcresult together with system state

Even if the result bundle was not finalized cleanly, preserve it as-is rather than deleting it in the failure path. A complete result can be exported as searchable JSON with the following command:

xcrun xcresulttool get \
  --legacy \
  --path ".artifacts/TestRun.xcresult" \
  --format json \
  >".artifacts/xcresult.json"

Each timeout directory should contain at least:

• the complete xcodebuild output and exit code;
• the scheme, test plan, destination, and Xcode version;
• host and simulator process lists;
• three thread samples captured at intervals;
• simulator device state;
• the original xcresult and the exported structured result;
• the current commit identifier and test entry parameters.

When running on a PingMini cloud Mac, each job should also use one explicitly assigned simulator device set so that another concurrent job cannot change its state. Confirm the currently available configurations and nodes in the console, and do not mix machine differences into the same reproduction cycle.

Turn the fix into a regression check

After finding the root cause, do not simply remove the watchdog. Keep timeout evidence collection, but change the threshold to a budget based on historical percentiles. Also add a minimal regression case for the fixed scenario. For example, a deadlock fix should verify that the callback completes within a defined interval, while a resource-lock fix should verify that lock files and child processes are cleaned up when the test ends.

Finally, verify three things: normal jobs do not trigger sampling, abnormal jobs exit within a bounded amount of time, and the archive contains enough evidence to answer “which process is waiting for what?” Once those conditions are met, an intermittent XCTest hang is no longer just a red job that requires a manual rerun. It becomes an engineering problem that can be investigated systematically and driven toward resolution.

Frequently asked questions