Repository navigation
fix(github_runner_cluster): stop a wedged Colima profile before starting it - #253
Merged
Merged
Conversation
…ing it A Colima VM that Lima's VZ driver has put into an error state is still listed as running, with its host agent alive and its Docker socket dead. The role's start found the profile running and did nothing, so the runtime tasks failed and nothing in the role could recover the host. When the runtime does not answer and `colima list` shows the profile as running, the role now ends it with `colima stop --force` and starts it again from the same disk. Nothing in the VM's disk, containers or volumes is removed. A profile that is listed as stopped, one whose runtime answers and a run under --check are left as before.
…opping it A single failed docker info is no longer enough to stop a VM. A profile that Colima lists as running while its runtime is silent is stopped only when docker info keeps failing on repeated bounded tries, a command in the guest fails, the VM's host agent is older than a boot grace period and the Docker context points at that profile's own socket. The stop is colima stop first and colima stop --force only if that fails. The stop has its own switch, github_runner_cluster_runtime_force_stop_wedged, separate from autostart. Switched off, or when a condition does not hold, the run fails with the signals it saw and the commands to recover by hand. Under --check every read is made and nothing is stopped.
…'s age live A timer set with alarm before exec does not end a Go program, which ignores SIGALRM, so a runtime that accepts a connection and never replies held the run, and a clean stop could block for ever. The probes, the guest command and the clean stop now run under files/run_bounded.pl, which gives the command its own process group, sends TERM at the deadline and KILL after a grace period, and reports a timeout distinctly. The first probe of the runtime is bounded the same way. The age of the VM's host agent now comes from the live process the pid file names, and only when that process is Lima's host agent. A pid file that outlived its process, as one from before a reboot does, or one that names another program, leaves the age unknown, which never counts as old.
… correctly A context whose endpoint is not a Colima socket is refused earlier, in the filter that reads the home from it, so the stop's own comparison is reached only when the home variable names another home than the context's socket. The test now sets exactly that, and the test of the variable winning over the context no longer asserts that the context is not read, since the comparison reads it.
… a started VM answer The report of a VM whose host agent is young now says "for less than" the grace period, matching the other wording of the age. The test stand-in for a restarted VM clears the hang modes of the stopped one, as a real start does.
…g hosts The Docker context is read for the Colima home only when no home is named, as before, so a host that names its home because it has no context that says (DOCKER_HOST, an empty or unknown context) runs again. The stop reads the context itself and may fail to: an unreadable context means the socket is not proven to be the profile's own, so nothing is stopped and the report says so. The supervisor ends the command's process group when it is itself sent TERM, INT or HUP and exits 143. Only exit 124 or the tool's own non-zero status counts as a failed probe; a supervisor that cannot start the command (127), is misused (255) or has no perl fails the run and names it. The deadline uses the monotonic clock, a command that ends in the last poll interval reports its own status, and the host agent is matched on the program's basename and its hostagent argument.
… its message When the command module cannot start perl it reports "Error executing command" with the operating system's error number as rc (2 for a missing file, 13 for a permission error) or 257, so keying on one rc missed all but the missing file. A filter, cluster_supervisor_failed, now says whether a supervised command's result shows that the supervisor could not run: the supervisor's own 127 and 255, or that message with any rc. The probe, the guest command and the stop all use it, in place of five copies of an expression. The test of a command that ends before its deadline is labelled as the regression guard it is.
|
🎉 This PR is included in version 3.2.2 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Colima VM in Lima's VZ error state is still listed as running with its host agent alive, so a start does nothing and the Docker socket stays dead. The role now stops such a profile (colima stop, then colima stop --force if that fails) and starts it again from the same disk, but only when docker info keeps failing on repeated bounded tries, a command in the guest fails, the VM's host agent process has been running for longer than a boot grace period and the Docker context points at that profile's own socket. Every probe, the guest command and the clean stop run under files/run_bounded.pl, which owns the deadline (TERM to the process group, then KILL after a grace), because Go programs ignore SIGALRM. The agent's age is read from the live process the pid file names, never from the file's date. The stop has its own switch, github_runner_cluster_runtime_force_stop_wedged, separate from autostart. When the switch is off or a condition does not hold the run fails with the signals it saw and the commands to recover by hand. Under --check every read is made and nothing is stopped. No disk, container or volume is removed; a forced stop ends the guest without a graceful shutdown. Tests cover the confirmed case, a clean stop that fails, probes and stops that never reply and ignore SIGALRM and TERM, a daemon that answers on a later try, a guest that answers, a booting VM, a stale or foreign pid file, a context pointing at another socket, the switch, check mode and a second run, and the supervisor script itself.