Skip to content

Fix interrupted KVM snapshot leaving VM on orphan overlay - #14697

Open
sidshas03 wants to merge 3 commits into
apache:mainfrom
sidshas03:fix/14691-snapshot-orphan-overlay
Open

sidshas03 wants to merge 3 commits into
apache:mainfrom
sidshas03:fix/14691-snapshot-orphan-overlay

Conversation

@sidshas03

Copy link
Copy Markdown

Description

This PR fixes a data loss problem that shows up when a volume snapshot of a
running VM gets interrupted midway, either because the agent dies or the merge
fails silently. In that situation the VM keeps running off an overlay file,
but volumes.path is still pointing at the base. So the next time the VM
starts, it comes up on the base and all the writes that went to the overlay
are gone, with nothing telling you it happened.

There are a few parts to the fix. Script.runSimpleBashScriptWithFullResult()
now carries the actual virsh error output in the exception it throws. Before
this the retry logic had nothing to match against, so now it can recognise the
transient ones like "disk already in active block job" and retry properly. The
snapshot markers also store vmName, diskLabel and overlayPath now, which gives
reconcileSnapshotMarkers() enough to work with, and that method is implemented
fully in this PR. For a running VM that is sitting on an overlay it does a
blockcommit, and for the offline case it falls back to qemu-img commit, with
safety checks so a modified base file is never overwritten.

Along with that, waitForCommandsAndExit is now waitForCommandsAndReconnect. It
calls reconnect() instead of System.exit(1), because exiting was killing
snapshot merges that were still in flight. And there was a duplicate
SNAPSHOT_MARKER_DIR constant that was breaking the build, so that is removed.

Fixes: #14691

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Script.runSimpleBashScriptWithFullResult now includes error output in exceptions, enabling retry logic for transient virsh errors. Snapshot markers store VM name, disk label, overlay path for recovery. reconcileSnapshotMarkers() fully implemented with blockcommit for running VMs and qemu-img commit for offline cases. Agent reconnects instead of System.exit(1) on ReadyCommand failure. Removed duplicate SNAPSHOT_MARKER_DIR constant.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 12:47
@boring-cyborg

boring-cyborg Bot commented Nov 13, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@JoaoJandre
JoaoJandre self-requested a review November 13, 2026 12:58
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 42.72300% with 122 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.78%. Comparing base (2f34b0c) to head (5ce4411).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java 39.45% 102 Missing and 10 partials ⚠️
agent/src/main/java/com/cloud/agent/Agent.java 60.86% 7 Missing and 2 partials ⚠️
...s/src/main/java/com/cloud/utils/script/Script.java 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##               main   #14697    +/-   ##
==========================================
  Coverage     20.77%   20.78%            
- Complexity    20847    20859    +12     
==========================================
  Files          6053     6053            
  Lines        548222   548428   +206     
  Branches      73236    73268    +32     
==========================================
+ Hits         113896   113984    +88     
- Misses       420984   421088   +104     
- Partials      13342    13356    +14     
Flag Coverage Δ
uitests 3.86% <ø> (ø)
unittests 22.21% <42.72%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI balanced review requested due to automatic review settings November 13, 2026 16:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@JoaoJandre JoaoJandre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although the PR is not in draft, it looks unfinished; have you tested it?
Also, I'm not sure this is the best approach, but maybe with a working PR my thoughts will change.


String actualDiskLabel = StringUtils.isNotEmpty(diskLabel) ? diskLabel : disk.getDiskLabel();
try {
resource.mergeDeltaIntoBaseFile(vm, actualDiskLabel, volumePath, null, true, "reconcile", null, conn);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informing 'reconcile' here will fail on mergeSnapshotIntoBaseFileWithoutEvents during the overlay delete section. Since the snapshot name is used to determine its path.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, you are right. 'mergeSnapshotIntoBaseFileWithoutEvents' calls getSnapshotTemporaryPath(baseFilePath, snapshotName) to find the overlay, and passing "reconcile" there would construct a completely wrong path. Fixed it, the actual snapshotName from the marker is passed through now.

Comment on lines +3828 to +3830
logger.info("VM [{}] disk [{}] is already on base [{}]. Overlay [{}] is orphaned. Removing marker.",
vmName, disk.getDiskLabel(), volumePath, overlayPath);
marker.delete();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we check if the overlay file exists and delete it if needed?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, makes sense. If the VM has already pivoted back to the base, the overlay is just lying around wasting disk space. Added the cleanup! we check if the overlay exists and delete it before removing the marker.

Comment on lines +3868 to +3873
if (markerTimestamp > 0 && baseFile.lastModified() > markerTimestamp) {
logger.error("Base file [{}] has been modified since snapshot marker was written (base modified={}, marker={}). "
+ "The overlay [{}] cannot be safely committed. Manual intervention required — keeping marker.",
volumePath, baseFile.lastModified(), markerTimestamp, overlayPath);
return;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it seems like this will always be the case. We write the marker before starting the snapshot process, no?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, this was a bug. Marker gets written before the snapshot starts, so after a normal snapshot the base mtime will always be newer than the marker timestamp. Changed the approach! now we compare baseFile.lastModified() against overlayFile.lastModified(). If base was modified after the overlay was created, something else wrote to it and we cannot safely commit.

Comment on lines +691 to +692
logger.info("Reconnecting instead of exiting to avoid losing in-flight work (e.g. snapshot overlays).");
reconnect(link);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the management server sees that the agent has disconnected, it will immediately stop waiting for command responses from the agent. This does not seem like an effective change without any changes to the management server as well.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. MS drops the connection context the moment it sees agent disconnect, so waiting for responses nobody is listening for does not help. Simplified this stripped the polling loop, just reconnect directly. Still better than the old System.exit(1) since it does not kill in-flight libvirt operations. But the real safety net is the marker-based reconciliation on startup.


validateConvertResult(convertResult, snapshotPath);
} catch (LibvirtException e) {
removeSnapshotMarker(primaryPool, diskPath, snapshotName);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we leave the marker in case of exceptions?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, removed that call. If a LibvirtException happens the overlay may still be on disk, and removing the marker means reconciliation will never find it on restart.

* - Domain gone, overlay exists: offline qemu-img commit
* - Base modified since marker: cannot safely commit, alert and keep marker
*/
protected void reconcileSnapshotMarkers() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this called anywhere?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, it was not wired up. Added the call in configure() at the end, so it runs when the storage processor initializes on agent startup.

}
}

protected void reconcileSingleMarker(File marker) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to be called only by the reconcileSnapshotMarkers method, which does not seem to be called anywhere

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above! reconcileSnapshotMarkers() is now called from configure(), so the whole chain is live.

Copilot AI balanced review requested due to automatic review settings November 17, 2026 15:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 17, 2026 15:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@sidshas03
sidshas03 force-pushed the fix/14691-snapshot-orphan-overlay branch from 0c3cc8b to 5ce4411 Compare November 17, 2026 16:02
Copilot AI balanced review requested due to automatic review settings November 18, 2026 01:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Interrupted running-VM volume snapshot leaves the VM on an orphan overlay, disk reverts on next start

3 participants