Repository navigation
Conversation
Script.runSimpleBashScriptWithFullResult now includes error output in exceptions, enabling retry logic for transient virsh errors. Snapshot markers store VM name, disk label, overlay path for recovery. reconcileSnapshotMarkers() fully implemented with blockcommit for running VMs and qemu-img commit for offline cases. Agent reconnects instead of System.exit(1) on ReadyCommand failure. Removed duplicate SNAPSHOT_MARKER_DIR constant.
|
Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14697 +/- ##
==========================================
Coverage 20.77% 20.78%
- Complexity 20847 20859 +12
==========================================
Files 6053 6053
Lines 548222 548428 +206
Branches 73236 73268 +32
==========================================
+ Hits 113896 113984 +88
- Misses 420984 421088 +104
- Partials 13342 13356 +14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JoaoJandre
left a comment
There was a problem hiding this comment.
Although the PR is not in draft, it looks unfinished; have you tested it?
Also, I'm not sure this is the best approach, but maybe with a working PR my thoughts will change.
|
|
||
| String actualDiskLabel = StringUtils.isNotEmpty(diskLabel) ? diskLabel : disk.getDiskLabel(); | ||
| try { | ||
| resource.mergeDeltaIntoBaseFile(vm, actualDiskLabel, volumePath, null, true, "reconcile", null, conn); |
There was a problem hiding this comment.
Informing 'reconcile' here will fail on mergeSnapshotIntoBaseFileWithoutEvents during the overlay delete section. Since the snapshot name is used to determine its path.
There was a problem hiding this comment.
Good catch, you are right. 'mergeSnapshotIntoBaseFileWithoutEvents' calls getSnapshotTemporaryPath(baseFilePath, snapshotName) to find the overlay, and passing "reconcile" there would construct a completely wrong path. Fixed it, the actual snapshotName from the marker is passed through now.
| logger.info("VM [{}] disk [{}] is already on base [{}]. Overlay [{}] is orphaned. Removing marker.", | ||
| vmName, disk.getDiskLabel(), volumePath, overlayPath); | ||
| marker.delete(); |
There was a problem hiding this comment.
Shouldn't we check if the overlay file exists and delete it if needed?
There was a problem hiding this comment.
Yes, makes sense. If the VM has already pivoted back to the base, the overlay is just lying around wasting disk space. Added the cleanup! we check if the overlay exists and delete it before removing the marker.
| if (markerTimestamp > 0 && baseFile.lastModified() > markerTimestamp) { | ||
| logger.error("Base file [{}] has been modified since snapshot marker was written (base modified={}, marker={}). " | ||
| + "The overlay [{}] cannot be safely committed. Manual intervention required — keeping marker.", | ||
| volumePath, baseFile.lastModified(), markerTimestamp, overlayPath); | ||
| return; | ||
| } |
There was a problem hiding this comment.
To me it seems like this will always be the case. We write the marker before starting the snapshot process, no?
There was a problem hiding this comment.
You are right, this was a bug. Marker gets written before the snapshot starts, so after a normal snapshot the base mtime will always be newer than the marker timestamp. Changed the approach! now we compare baseFile.lastModified() against overlayFile.lastModified(). If base was modified after the overlay was created, something else wrote to it and we cannot safely commit.
| logger.info("Reconnecting instead of exiting to avoid losing in-flight work (e.g. snapshot overlays)."); | ||
| reconnect(link); |
There was a problem hiding this comment.
When the management server sees that the agent has disconnected, it will immediately stop waiting for command responses from the agent. This does not seem like an effective change without any changes to the management server as well.
There was a problem hiding this comment.
Fair point. MS drops the connection context the moment it sees agent disconnect, so waiting for responses nobody is listening for does not help. Simplified this stripped the polling loop, just reconnect directly. Still better than the old System.exit(1) since it does not kill in-flight libvirt operations. But the real safety net is the marker-based reconciliation on startup.
|
|
||
| validateConvertResult(convertResult, snapshotPath); | ||
| } catch (LibvirtException e) { | ||
| removeSnapshotMarker(primaryPool, diskPath, snapshotName); |
There was a problem hiding this comment.
shouldn't we leave the marker in case of exceptions?
There was a problem hiding this comment.
Correct, removed that call. If a LibvirtException happens the overlay may still be on disk, and removing the marker means reconciliation will never find it on restart.
| * - Domain gone, overlay exists: offline qemu-img commit | ||
| * - Base modified since marker: cannot safely commit, alert and keep marker | ||
| */ | ||
| protected void reconcileSnapshotMarkers() { |
There was a problem hiding this comment.
Is this called anywhere?
There was a problem hiding this comment.
You are right, it was not wired up. Added the call in configure() at the end, so it runs when the storage processor initializes on agent startup.
| } | ||
| } | ||
|
|
||
| protected void reconcileSingleMarker(File marker) { |
There was a problem hiding this comment.
this seems to be called only by the reconcileSnapshotMarkers method, which does not seem to be called anywhere
There was a problem hiding this comment.
Same as above! reconcileSnapshotMarkers() is now called from configure(), so the whole chain is live.
0c3cc8b to
5ce4411
Compare
Description
This PR fixes a data loss problem that shows up when a volume snapshot of a
running VM gets interrupted midway, either because the agent dies or the merge
fails silently. In that situation the VM keeps running off an overlay file,
but volumes.path is still pointing at the base. So the next time the VM
starts, it comes up on the base and all the writes that went to the overlay
are gone, with nothing telling you it happened.
There are a few parts to the fix. Script.runSimpleBashScriptWithFullResult()
now carries the actual virsh error output in the exception it throws. Before
this the retry logic had nothing to match against, so now it can recognise the
transient ones like "disk already in active block job" and retry properly. The
snapshot markers also store vmName, diskLabel and overlayPath now, which gives
reconcileSnapshotMarkers() enough to work with, and that method is implemented
fully in this PR. For a running VM that is sitting on an overlay it does a
blockcommit, and for the offline case it falls back to qemu-img commit, with
safety checks so a modified base file is never overwritten.
Along with that, waitForCommandsAndExit is now waitForCommandsAndReconnect. It
calls reconnect() instead of System.exit(1), because exiting was killing
snapshot merges that were still in flight. And there was a duplicate
SNAPSHOT_MARKER_DIR constant that was breaking the build, so that is removed.
Fixes: #14691
Types of changes