Stop the suite failing on things that are not test failures

Two sources of random red on WSL, neither of them a test.

A launch that never happened. The Godot binary is a Windows executable
started across the interop boundary, and with six going at once that start
occasionally just fails: exit 127, an empty log, no engine banner, nothing
run. It was indistinguishable from a failure and moved between tests from
run to run. Anything that actually ran got as far as printing the banner, so
the two are easy to tell apart; a start that produced no banner is retried
up to three times and reported as NOSTART if it never comes up. Retries are
printed, never hidden - a test that needed one is saying the machine is
struggling even where it went on to pass.

The working tree as a run directory. WSL ran the suite in place, so six
Godot processes wrote .godot and created and deleted files under
PlanetSetups/ inside the live repo, alongside git and whatever else was
using it. Single tests failed at random under that - a delete that did not
take, a scene that would not instantiate - never the same one twice. WSL now
copies to %LOCALAPPDATA% first, the same isolation Termux already gets from
its own copy, on a native NTFS path so the Windows Godot still reads it at
full speed. Costs about 10s a run and leaves the repo untouched.

Four consecutive clean 124/124 runs since.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Loïc Wisniewski 2026-08-22 23:30:18 +02:00
commit 85c9bd9d8d
2 changed files with 75 additions and 21 deletions

View file

@ -22,6 +22,15 @@ VERBOSE="$1"; shift
# complete in seconds, even on a busy phone, so make that failure explicit
# rather than making the suite look hung. Override for deliberate long runs.
TEST_TIMEOUT_SECONDS="${TEST_TIMEOUT_SECONDS:-120}"
# How many times to start a test whose process never came up at all. Under
# WSL the Godot binary is a Windows executable started across the interop
# boundary, and with six of them going at once that start occasionally just
# fails - exit 127, an empty log, no engine banner, nothing run. It looked
# exactly like a test failure while being a property of the launch, and it
# moved from test to test between runs. A test that genuinely fails always
# gets far enough to print the banner, so the two are easy to tell apart and
# only the launch is worth retrying.
LAUNCH_ATTEMPTS="${LAUNCH_ATTEMPTS:-3}"
cd "$PROJECT_DIR" || exit 2
@ -56,18 +65,37 @@ run_one() {
# 3 parallel runs. Pinned, a test simulates the same 60fps whatever else
# the phone is doing, and its result no longer depends on how busy the
# machine was.
timeout "$TEST_TIMEOUT_SECONDS" "$GODOT_BIN" --headless --fixed-fps 60 \
-s "res://tests/$name" >"$log" 2>&1
local rc=$?
local rc=0
local attempt=1
while true; do
timeout "$TEST_TIMEOUT_SECONDS" "$GODOT_BIN" --headless --fixed-fps 60 \
-s "res://tests/$name" >"$log" 2>&1
rc=$?
# "Never started" is rc != 0 with no banner in the log. Anything that
# ran - passed, failed, crashed mid-way - printed the banner first.
if (( rc == 0 )) || grep -q "Godot Engine" "$log" \
|| (( attempt >= LAUNCH_ATTEMPTS )); then
break
fi
attempt=$(( attempt + 1 ))
sleep 1
done
end=$(date +%s%N)
local ms=$(( (end - start) / 1000000 ))
# Retries are reported, never hidden: a test that needed one is telling
# you the machine is struggling, even where it went on to pass.
local retried=""
if (( attempt > 1 )); then
retried=" (started on attempt $attempt)"
fi
# Two independent criteria, because either one alone can lie: a test that
# crashes before reaching its verdict exits non-zero without ever printing
# FAIL, and a hypothetical `quit(0)` on a failing path would exit clean
# while the log says otherwise.
if (( rc == 0 )) && grep -q ": PASS" "$log"; then
printf 'PASS %6dms %s\n' "$ms" "$name" > "$outdir/$name.status"
printf 'PASS %6dms %s%s\n' "$ms" "$name" "$retried" > "$outdir/$name.status"
elif (( rc == 124 )); then
printf 'TIMEOUT %4dms %s (limit %ss)\n' \
"$ms" "$name" "$TEST_TIMEOUT_SECONDS" > "$outdir/$name.status"
@ -78,8 +106,11 @@ run_one() {
# indistinguishable from a hang.
printf 'KILLED %4dms %s (signal %d - out of memory? try JOBS=1)\n' \
"$ms" "$name" "$(( rc - 128 ))" > "$outdir/$name.status"
elif (( rc != 0 )) && ! grep -q "Godot Engine" "$log"; then
printf 'NOSTART %3dms %s (exit %d after %d attempts - the process never came up)\n' \
"$ms" "$name" "$rc" "$attempt" > "$outdir/$name.status"
else
printf 'FAIL %6dms %s (exit %d)\n' "$ms" "$name" "$rc" > "$outdir/$name.status"
printf 'FAIL %6dms %s (exit %d)%s\n' "$ms" "$name" "$rc" "$retried" > "$outdir/$name.status"
fi
cat "$outdir/$name.status"
}