Skip to content

CKS: running clusters cannot be scaled live on a default install, because the KVM SystemVM template is not dynamically scalable #14704

Open

Description

@andrijapanicsb

In short

#13226 added live offering changes for running CKS clusters on KVM. On a default install that feature never works: CKS nodes use the SystemVM template, which is registered as not dynamically scalable, so every node is created as not scalable and the scale fails. Proposal: register KVM SystemVM templates as dynamically scalable by default. We tested the side effects and found nothing that should stop this.

How to reproduce

On a default KVM zone with enable.dynamic.scale.vm=true, create a CKS cluster without choosing a node template, then change its service offering while it is running. The scale fails with Unable to scale VM instance ... as it does not have tools to support dynamic scaling.

Why it matters

The CKS documentation says "A running or stopped Kubernetes cluster can be scaled ... to a desired cluster size and service offering". On a default install, only stopped clusters can. To use the live feature, an operator has to find out on their own that the SystemVM template's flag must be turned on. They also have to turn it on for each existing node while the cluster is stopped, because a VM's flag can only be enabled when its template allows it: dynamic scaling cannot be enabled for instances the template does not have scaling enabled.

Cause

  • vm_template.dynamically_scalable defaults to 0, and SystemVmTemplateRegistration never sets it, so the KVM SystemVM template is always false. The column dates from 2013, when it meant "the template has XenServer/VMware tools". KVM needs no guest tools for hot-plug.
  • A user VM, including a CKS node, is scalable only when its template, its offering and enable.dynamic.scale.vm all allow it. So the template flag blocks every CKS node.

What we tested (standard x86_64 KVM hosts)

Situation Result
Default install: live offering change of a running CKS cluster Fails: "does not have tools to support dynamic scaling"
Template flag on, node flags on, same change Works: all nodes resized live, no reboot, guests see the added CPUs
Template flag on, global setting on: newly created router, SSVM, console proxy They become dynamically scalable too, and lose about 67 MB of usable memory each (a 64 MB kernel I/O buffer for the hot-plug memory range): 457 → 390 MiB on 512 MiB routers and SSVM, 960 → 894 MiB on the 1 GiB console proxy
Template flag on, global setting off No change for system VMs or CKS nodes; nothing becomes scalable (checked in the code)
Existing VMs after the template flag is changed Unchanged; each VM keeps the flag it was created with, so nothing breaks
VM whose vCPU maximum exceeds the machine type limit Refused by libvirt on every host; CloudStack tries the other hosts and returns an error, no crash

The memory cost applies only where an operator has turned on enable.dynamic.scale.vm, which is already required to use live scaling. Anyone who has made the SystemVM template scalable by hand, to get live CKS scaling, already has it today.

vCPU maximum limits seen on the hosts we checked

A dynamically scalable VM gets the host's CPU count as its vCPU maximum. These are the machine type limits libvirt reports:

Host QEMU / libvirt pc (i440fx) Q35
AlmaLinux 9.1 7.0.0 / 8.5.0 240 710
Ubuntu 24.04.4 8.2.2 / 10.0.0 255 1024
Ubuntu 26.04.1 10.2.1 / 12.0.0 255 4096
Debian 13 11.0.2 / 11.3.0 255 4096

Only hosts with more threads than these limits would be affected, and kvm.cpu.dynamic.scaling.capacity caps the maximum per cluster. On Q35, more than 255 vCPUs also needs an IOMMU in extended interrupt mode (<ioapic driver='qemu'/> plus <iommu model='intel'><driver intremap='on' eim='on'/></iommu>).

Proposal

Make KVM SystemVM templates dynamically scalable by default, from the next release on:

  1. In SystemVmTemplateRegistration, set the flag when a KVM SystemVM template is registered, for fresh installs and new template versions.
  2. In the upgrade SQL, set it on all existing KVM SystemVM templates, since installs upgraded across several releases keep several of them:
    UPDATE cloud.vm_template SET dynamically_scalable = 1 WHERE type = 'SYSTEM' AND hypervisor_type = 'KVM' AND removed IS NULL;
    The SQL runs before the new template row is created, so both 1 and 2 are needed.
  3. Mention in the release notes and the CKS docs that existing CKS nodes keep their old flag until they are updated or recreated.

Side note: system VMs ignore their offering's dynamic-scaling flag

Routers, SSVM and console proxy copy the template's flag and never look at their service offering's dynamic-scaling setting (NetworkHelperImpl, SecondaryStorageManagerImpl, ConsoleProxyManagerImpl). We set it to off on those system offerings, recreated the VMs, and they still came up scalable. User VMs check template, offering and the global setting together since #4643; system VMs were not updated then. @harikrishna-patnala, as the author of #4643 (and of the original system VM scaling change, CLOUDSTACK-3042), could you confirm whether leaving system VMs out was an omission, or deliberate for a reason we're missing? If it's an omission, it's a separate small bug, not part of this proposal.

Suggested testing for the fix

  • Fresh install: the KVM SystemVM template is registered with dynamically_scalable = 1.
  • Upgrade from 4.20 / 4.22 / 4.23: every existing KVM SystemVM template row is set to 1, and the new one too.
  • A new CKS cluster on the default template scales live, with enable.dynamic.scale.vm on.
  • Recreated routers, SSVM and console proxy start normally, with the global setting on and off.
  • Existing VMs and CKS nodes are unchanged after the upgrade.

Activity

  1. github-actions commented on Nov 16, 2026

    @github-actions

    Warning

    threat detection engine error
    The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

    What happened

    The threat detection engine failed to produce results.

    Review the workflow run logs for details.

    🎯 Triage report

    On a default KVM install, a running CKS cluster cannot be scaled live. The KVM SystemVM template is registered with dynamically_scalable = 0, so CKS nodes built from it are not dynamically scalable, and a live offering change fails with "does not have tools to support dynamic scaling". The reporter proposes registering KVM SystemVM templates as dynamically scalable, both for new registrations and through the upgrade SQL for existing ones. PR #14705 is already open for this.

    📊 Assessment

    Dimension Value Reasoning
    Type type:bug The documented live offering change does not work on a default install
    Component component:cks, component:kvm CKS node scaling, and the KVM SystemVM template flag
    Severity Severity:Minor Blocked by default; a manual template change works around it, and no data loss is reported
    Labels type:bug, component:cks, component:kvm, Severity:Minor Based on the issue text
    Coding agent Needs more info The fix is specified and PR #14705 exists, but the system VM question below needs a maintainer decision

    🔗 Similar issues

    💡 Notes and suggestions
    • Reproduction: on a KVM zone with enable.dynamic.scale.vm=true, create a CKS cluster with the default node template, then change its service offering while it is running.
    • The upgrade SQL has to cover every existing KVM SystemVM template row, because the template-registration change runs too late for the new row.
    • Existing CKS nodes keep their old flag until they are recreated. The report asks for this to be stated in the release notes and CKS docs.
    • Open question for maintainers: routers, SSVM and console proxy copy the template flag and ignore their offering's dynamic-scaling setting. The reporter asks whether this was deliberate. Making system VMs scalable also reduces usable memory by about 67 MB on small system VMs, which the report measures.

    Generated by Daily Issue Triage · copilot · haiku55 · ~6 AIC · ⊞ 3.8K · ◷

    Add this agentic workflow to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/daily-issue-triage.md@139fa09ef17cde4135e696ff8e7af09bc5ddfa6e
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions