Stop the suite failing on things that are not test failures
Two sources of random red on WSL, neither of them a test. A launch that never happened. The Godot binary is a Windows executable started across the interop boundary, and with six going at once that start occasionally just fails: exit 127, an empty log, no engine banner, nothing run. It was indistinguishable from a failure and moved between tests from run to run. Anything that actually ran got as far as printing the banner, so the two are easy to tell apart; a start that produced no banner is retried up to three times and reported as NOSTART if it never comes up. Retries are printed, never hidden - a test that needed one is saying the machine is struggling even where it went on to pass. The working tree as a run directory. WSL ran the suite in place, so six Godot processes wrote .godot and created and deleted files under PlanetSetups/ inside the live repo, alongside git and whatever else was using it. Single tests failed at random under that - a delete that did not take, a scene that would not instantiate - never the same one twice. WSL now copies to %LOCALAPPDATA% first, the same isolation Termux already gets from its own copy, on a native NTFS path so the Windows Godot still reads it at full speed. Costs about 10s a run and leaves the repo untouched. Four consecutive clean 124/124 runs since. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
12bd5eb7bf
commit
85c9bd9d8d
2 changed files with 75 additions and 21 deletions
|
|
@ -22,6 +22,15 @@ VERBOSE="$1"; shift
|
|||
# complete in seconds, even on a busy phone, so make that failure explicit
|
||||
# rather than making the suite look hung. Override for deliberate long runs.
|
||||
TEST_TIMEOUT_SECONDS="${TEST_TIMEOUT_SECONDS:-120}"
|
||||
# How many times to start a test whose process never came up at all. Under
|
||||
# WSL the Godot binary is a Windows executable started across the interop
|
||||
# boundary, and with six of them going at once that start occasionally just
|
||||
# fails - exit 127, an empty log, no engine banner, nothing run. It looked
|
||||
# exactly like a test failure while being a property of the launch, and it
|
||||
# moved from test to test between runs. A test that genuinely fails always
|
||||
# gets far enough to print the banner, so the two are easy to tell apart and
|
||||
# only the launch is worth retrying.
|
||||
LAUNCH_ATTEMPTS="${LAUNCH_ATTEMPTS:-3}"
|
||||
|
||||
cd "$PROJECT_DIR" || exit 2
|
||||
|
||||
|
|
@ -56,18 +65,37 @@ run_one() {
|
|||
# 3 parallel runs. Pinned, a test simulates the same 60fps whatever else
|
||||
# the phone is doing, and its result no longer depends on how busy the
|
||||
# machine was.
|
||||
timeout "$TEST_TIMEOUT_SECONDS" "$GODOT_BIN" --headless --fixed-fps 60 \
|
||||
-s "res://tests/$name" >"$log" 2>&1
|
||||
local rc=$?
|
||||
local rc=0
|
||||
local attempt=1
|
||||
while true; do
|
||||
timeout "$TEST_TIMEOUT_SECONDS" "$GODOT_BIN" --headless --fixed-fps 60 \
|
||||
-s "res://tests/$name" >"$log" 2>&1
|
||||
rc=$?
|
||||
# "Never started" is rc != 0 with no banner in the log. Anything that
|
||||
# ran - passed, failed, crashed mid-way - printed the banner first.
|
||||
if (( rc == 0 )) || grep -q "Godot Engine" "$log" \
|
||||
|| (( attempt >= LAUNCH_ATTEMPTS )); then
|
||||
break
|
||||
fi
|
||||
attempt=$(( attempt + 1 ))
|
||||
sleep 1
|
||||
done
|
||||
end=$(date +%s%N)
|
||||
local ms=$(( (end - start) / 1000000 ))
|
||||
|
||||
# Retries are reported, never hidden: a test that needed one is telling
|
||||
# you the machine is struggling, even where it went on to pass.
|
||||
local retried=""
|
||||
if (( attempt > 1 )); then
|
||||
retried=" (started on attempt $attempt)"
|
||||
fi
|
||||
|
||||
# Two independent criteria, because either one alone can lie: a test that
|
||||
# crashes before reaching its verdict exits non-zero without ever printing
|
||||
# FAIL, and a hypothetical `quit(0)` on a failing path would exit clean
|
||||
# while the log says otherwise.
|
||||
if (( rc == 0 )) && grep -q ": PASS" "$log"; then
|
||||
printf 'PASS %6dms %s\n' "$ms" "$name" > "$outdir/$name.status"
|
||||
printf 'PASS %6dms %s%s\n' "$ms" "$name" "$retried" > "$outdir/$name.status"
|
||||
elif (( rc == 124 )); then
|
||||
printf 'TIMEOUT %4dms %s (limit %ss)\n' \
|
||||
"$ms" "$name" "$TEST_TIMEOUT_SECONDS" > "$outdir/$name.status"
|
||||
|
|
@ -78,8 +106,11 @@ run_one() {
|
|||
# indistinguishable from a hang.
|
||||
printf 'KILLED %4dms %s (signal %d - out of memory? try JOBS=1)\n' \
|
||||
"$ms" "$name" "$(( rc - 128 ))" > "$outdir/$name.status"
|
||||
elif (( rc != 0 )) && ! grep -q "Godot Engine" "$log"; then
|
||||
printf 'NOSTART %3dms %s (exit %d after %d attempts - the process never came up)\n' \
|
||||
"$ms" "$name" "$rc" "$attempt" > "$outdir/$name.status"
|
||||
else
|
||||
printf 'FAIL %6dms %s (exit %d)\n' "$ms" "$name" "$rc" > "$outdir/$name.status"
|
||||
printf 'FAIL %6dms %s (exit %d)%s\n' "$ms" "$name" "$rc" "$retried" > "$outdir/$name.status"
|
||||
fi
|
||||
cat "$outdir/$name.status"
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue