metaheavens/spaceship
Watch
1
0
Fork
You've already forked spaceship
 0
Code Issues 4 Pull requests Projects 1 Releases Packages Wiki Activity

test suite is too slow to run, improve it #8

New issue
Closed
opened 2026-08-10 21:25:25 +00:00 by metaheavens · 1 comment
metaheavens commented 2026-08-10 21:25:25 +00:00
Owner
Copy link
No description provided.
metaheavens commented 2026-08-11 12:57:16 +00:00
Author
Owner
Copy link

Done. 1075s -> ~55s for the full 38-test suite, and it no longer depends on how busy the phone is.

I timed every test individually before changing anything, because the obvious theory was wrong. The suite was not slow because tests wait for simulated time - most wait under 2 seconds. Splitting the fixed cost with a stopwatch script (tests/_profile_startup.gd) found it:

engine boot to _initialize:  967 ms
load(Main.tscn):           8564 ms   <-- here
instantiate():                7 ms
add_child() (_ready):         0 ms

load("res://Main.tscn") was 8.5 seconds, and nearly every test loads it. The cause is the filesystem: the project lives under ~/storage/shared, which is Android's FUSE bridge, read through proot on top of that. The same load from a native path is ~2s.

What changed

1. Tests run from a copy on Termux's own filesystem. run_tests.sh syncs the project (~1.4s for the whole suite, .godot import cache kept warm between runs) into ~/.cache/spaceship-tests and runs there. This is the single biggest win and it also sidesteps the FUSE mtime trap, since the copy gets fresh content every run.

2. Tests run 6 at a time. Each test is its own Godot process - they call quit() and stamp on the GameState autoload, so they cannot share one - which makes the suite a pile of independent, single-core-bound jobs on an 8-core phone. Output is buffered per test and printed only for failures.

3. One proot-distro login for the whole run instead of one per test (~0.5s x 38). The container-side half lives in tests/_run_inside.sh.

4. --fixed-fps 60. Parallelism alone made the suite lie: test_camera_settles and test_planet_prompt_map_interaction failed on 2 of 3 runs. A test's elapsed counted REAL seconds, so six processes at once stretched every frame and the gameplay under test saw deltas it never sees in the game. Pinned to 1/60, a test simulates the same 60fps whatever else the machine is doing, and its result no longer depends on load.

5. test_extractor_income fixture shortened. It sat through a full hauler round trip, and the cycle length is distance/speed - so the fixture distance was the runtime. 4000 units = ~31s of simulated travel; 600 units = ~4.6s. Nothing in it asserts on the distance, only that payout and drawn position track the same progress.

A test now counts as passed only if it both exits 0 and prints : PASS, so a crash before reaching its verdict can no longer be mistaken for success.

A real bug this surfaced

Pinning the frame rate broke test_joystick_control - and it turned out to be the game, not the test.

Ship._physics_process sets velocity = Vector2.ZERO the moment a ship is within arrive_radius of its slot, even though the anchor is still dragging that slot forward at cruise_speed underneath it. Fleet.stop() braked from that instantaneous velocity, so releasing the joystick on any frame where the ships happened to be seated stopped the fleet dead instead of coasting.

How often that happens depends on the frame rate, because the anchor advances cruise_speed * delta per frame. At the ~25fps the headless tests used to run at, it steps ~10 units, ships always lag and always report motion - which is the only reason releasing the stick ever looked right. At a steady 60fps it steps ~4 units and they are often seated. On a fast device this bug is visible in the real game.

Fixed in Fleet.stop() by braking from the ship's true motion, velocity + fleet_velocity - the fleet's own cruise velocity, which each ship already carries as context (and which is zero for a point-to-point move_to, where velocity alone is the whole story). fleet_velocity is now cleared on stop so a halted ship does not keep facing a heading it is not travelling on.

The test deserved part of the blame too: it asserted centroid.x > moving_centroid.x, which any float wobble satisfies. It now asserts the actual coast distance against the physics - cruise_speed^2 / (2 * acceleration) = ~75 units. Measured 75.1 with the fix, 0.0 without it.

Verification

Three consecutive full runs after the fix: 38/38, 38/38, 38/38 in 83s / 52s / 56s. No flakes.

Usage

bash tests/run_tests.sh              # everything
bash tests/run_tests.sh formation    # only tests matching a substring
JOBS=1 bash tests/run_tests.sh       # serial, for debugging a hang
VERBOSE=1 bash tests/run_tests.sh    # print passing tests' output too
IN_PLACE=1 bash tests/run_tests.sh   # run from the project dir itself
Done. **1075s -> ~55s** for the full 38-test suite, and it no longer depends on how busy the phone is. I timed every test individually before changing anything, because the obvious theory was wrong. The suite was not slow because tests wait for simulated time - most wait under 2 seconds. Splitting the fixed cost with a stopwatch script (`tests/_profile_startup.gd`) found it: ``` engine boot to _initialize: 967 ms load(Main.tscn): 8564 ms <-- here instantiate(): 7 ms add_child() (_ready): 0 ms ``` `load("res://Main.tscn")` was 8.5 seconds, and nearly every test loads it. The cause is the filesystem: the project lives under `~/storage/shared`, which is Android's FUSE bridge, read through proot on top of that. The same load from a native path is ~2s. ## What changed **1. Tests run from a copy on Termux's own filesystem.** `run_tests.sh` syncs the project (~1.4s for the whole suite, `.godot` import cache kept warm between runs) into `~/.cache/spaceship-tests` and runs there. This is the single biggest win and it also sidesteps the FUSE mtime trap, since the copy gets fresh content every run. **2. Tests run 6 at a time.** Each test is its own Godot process - they call `quit()` and stamp on the `GameState` autoload, so they cannot share one - which makes the suite a pile of independent, single-core-bound jobs on an 8-core phone. Output is buffered per test and printed only for failures. **3. One `proot-distro login` for the whole run** instead of one per test (~0.5s x 38). The container-side half lives in `tests/_run_inside.sh`. **4. `--fixed-fps 60`.** Parallelism alone made the suite lie: `test_camera_settles` and `test_planet_prompt_map_interaction` failed on 2 of 3 runs. A test's `elapsed` counted REAL seconds, so six processes at once stretched every frame and the gameplay under test saw deltas it never sees in the game. Pinned to 1/60, a test simulates the same 60fps whatever else the machine is doing, and its result no longer depends on load. **5. `test_extractor_income` fixture shortened.** It sat through a full hauler round trip, and the cycle length is distance/speed - so the fixture distance *was* the runtime. 4000 units = ~31s of simulated travel; 600 units = ~4.6s. Nothing in it asserts on the distance, only that payout and drawn position track the same `progress`. A test now counts as passed only if it *both* exits 0 and prints `: PASS`, so a crash before reaching its verdict can no longer be mistaken for success. ## A real bug this surfaced Pinning the frame rate broke `test_joystick_control` - and it turned out to be the game, not the test. `Ship._physics_process` sets `velocity = Vector2.ZERO` the moment a ship is within `arrive_radius` of its slot, even though the anchor is still dragging that slot forward at `cruise_speed` underneath it. `Fleet.stop()` braked from that instantaneous `velocity`, so releasing the joystick on any frame where the ships happened to be seated stopped the fleet **dead** instead of coasting. How often that happens depends on the frame rate, because the anchor advances `cruise_speed * delta` per frame. At the ~25fps the headless tests used to run at, it steps ~10 units, ships always lag and always report motion - which is the only reason releasing the stick ever looked right. At a steady 60fps it steps ~4 units and they are often seated. **On a fast device this bug is visible in the real game.** Fixed in `Fleet.stop()` by braking from the ship's true motion, `velocity + fleet_velocity` - the fleet's own cruise velocity, which each ship already carries as context (and which is zero for a point-to-point `move_to`, where `velocity` alone is the whole story). `fleet_velocity` is now cleared on stop so a halted ship does not keep facing a heading it is not travelling on. The test deserved part of the blame too: it asserted `centroid.x > moving_centroid.x`, which any float wobble satisfies. It now asserts the actual coast distance against the physics - `cruise_speed^2 / (2 * acceleration)` = ~75 units. Measured 75.1 with the fix, 0.0 without it. ## Verification Three consecutive full runs after the fix: **38/38, 38/38, 38/38** in 83s / 52s / 56s. No flakes. ## Usage ``` bash tests/run_tests.sh # everything bash tests/run_tests.sh formation # only tests matching a substring JOBS=1 bash tests/run_tests.sh # serial, for debugging a hang VERBOSE=1 bash tests/run_tests.sh # print passing tests' output too IN_PLACE=1 bash tests/run_tests.sh # run from the project dir itself ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
metaheavens/spaceship#8
No description provided.