Repository navigation
Fix sync ironic KeyError for fake-hardware nodes - #2903
Conversation
Pass the node updates through mask_secrets before printing them in the "Updating baremetal node" task output of _sync_ironic_device, as the dry-run path already does. sync_ironic now hands each device's template variables to _sync_ironic_device, so secret values rendered into other strings are masked as well. The attributes sent to Ironic are unchanged; mask_secrets works on a deep copy. Until now only the dry-run output was masked. The live sync printed the node updates verbatim, with only the driver password removed (it is dropped from the diff because Ironic never returns it). Any value under a key containing "password" or "secret" or starting with "ironic_osism_" in instance_info, extra or properties went to the task output in clear text, as did a secret rendered into a non-secret value such as kernel parameters. The collection of secret values from template variables moves out of _sync_ironic_device_dry_run into _collect_secret_values, shared by both paths. template_vars is an optional argument of _sync_ironic_device; without it, masking falls back to the key-based rules. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
Look up the driver's password key in driver_params with .get() at both places where the ironic sync compares driver_info, so a driver without an entry is treated as having no password key. This affects the real sync (_sync_ironic_device) and the dry-run update branch (_sync_ironic_device_dry_run). driver_params only lists ipmi and redfish. After computing the driver_info diff, the sync drops the password key from it because Ironic never returns the password. That key was looked up with driver_params[driver]["password"], which raises KeyError for any other driver, e.g. fake-hardware. The other uses of driver_params in this module already check membership first. The sync now creates a new node as a stub with only driver and automated_clean, then always runs the update procedure. The stub has no driver_info, so every new fake-hardware node produces a driver_info diff and fails on its first sync: Could not fully synchronize device node1 with ironic: 'fake-hardware' The node is left in enroll without instance_info or extra and is never moved to manageable/available, so nothing can be deployed on it. A config-only workaround does not exist: Ironic's redfish hardware type has no fake power interface, so a node without a real BMC needs the fake-hardware driver. This is the mode metalbox's fake Ironic setup uses. For a driver without a driver_params entry that does carry a password in driver_info (e.g. ilo_password), the password now stays in the driver_info diff. It is sent to Ironic on every sync, which is harmless, and it appears in the "Updating baremetal node" output, where the masking of that output hides it. Add regression tests for the create path, using the real deep_compare so the stub node's missing driver_info produces the diff, and for the dry-run update branch. A further test checks that an unmapped driver's password is masked in the update output. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
There was a problem hiding this comment.
Hey - I've found 4 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="osism/tasks/conductor/ironic.py" line_range="408" />
<code_context>
request_id,
- f"Updating baremetal node for {device.name} with {node_updates}\n",
+ f"Updating baremetal node for {device.name} with "
+ f"{mask_secrets(node_updates, secret_values=secret_values)}\n",
)
# NOTE: Pop target_raid_config from node_attributes since they cannot be set using node updates
</code_context>
<issue_to_address>
**Secrets in extra fields are logged**
When a serialized `extra` field contains a secret under a secret-named key and its value is not a collected template secret, `mask_secrets` treats the serialized value as an ordinary string and does not inspect its keys, so the update message prints the secret in clear text.
Inspect serialized `extra` values and redact values under secret-named keys before logging the update.
Also at `osism/tasks/conductor/ironic.py:409`.
</issue_to_address>
### Comment 2
<location path="osism/tasks/conductor/ironic.py" line_range="408" />
<code_context>
request_id,
- f"Updating baremetal node for {device.name} with {node_updates}\n",
+ f"Updating baremetal node for {device.name} with "
+ f"{mask_secrets(node_updates, secret_values=secret_values)}\n",
)
# NOTE: Pop target_raid_config from node_attributes since they cannot be set using node updates
</code_context>
<issue_to_address>
**Unmapped credentials appear in task output**
When an unmapped driver has a credential in `driver_info` under a key not recognized by the masking rules, such as `ilo_token`, `mask_secrets` does not recognize the credential key and `_collect_secret_values` does not contain its value, so the update message prints the credential in clear text.
Mask credentials using driver-specific secret metadata or extend the masking policy to cover credential fields beyond the current key patterns.
</issue_to_address>
### Comment 3
<location path="osism/tasks/conductor/ironic.py" line_range="409" />
<code_context>
- f"Updating baremetal node for {device.name} with {node_updates}\n",
+ f"Updating baremetal node for {device.name} with "
+ f"{mask_secrets(node_updates, secret_values=secret_values)}\n",
)
# NOTE: Pop target_raid_config from node_attributes since they cannot be set using node updates
target_raid_config = node_attributes.pop("target_raid_config", None)
</code_context>
<issue_to_address>
**Overlapping secrets remain visible**
When two collected secret values overlap and the shorter value is replaced first, `mask_secrets` replaces values one at a time, so it masks only the prefix of the longer secret and leaves its suffix visible in the update output.
Replace longer secrets before shorter ones, or otherwise ensure overlapping values are fully masked.
Also at `osism/tasks/conductor/ironic.py:408`.
</issue_to_address>
### Comment 4
<location path="osism/tasks/conductor/ironic.py" line_range="364" />
<code_context>
+ or "secret" in key.lower()
+ or key.lower().startswith("ironic_osism_")
+ ):
+ secret_values.add(value)
+ return secret_values
+
</code_context>
<issue_to_address>
**Empty secrets garble update output**
When a password, secret, or `ironic_osism_*` template variable has an empty value, `_collect_secret_values` adds the empty string, so `mask_secrets` applies `str.replace("", "xxx")` and inserts `xxx` between every character in the logged update.
Exclude empty values when collecting secrets for masking.
Also at `osism/tasks/conductor/ironic.py:365`.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 4 findings to address first, and the sync can now proceed for drivers without a password entry in driver_params, so incorrect node attributes may be written to Ironic instead of the sync stopping with a KeyError. Those stored settings can be corrected by a later update, but reverting this code alone would not undo them.
Blocking findings: osism/tasks/conductor/ironic.py:408, osism/tasks/conductor/ironic.py:408, osism/tasks/conductor/ironic.py:409, osism/tasks/conductor/ironic.py:364
fd18b5a to
18500ba
Compare
|
Secrets in extra fields / Overlapping secrets / Empty secrets: Valid, but these are existing limitations of the shared mask_secrets and the dry-run output, which this PR doesn't change. The live output now gets the same masking as the dry-run, no stricter. Listed under "Known limitations" in the description. Unmapped credentials: Same: masking works by key name, and widening the patterns is a separate change. |
osism sync ironicfails on every new node whoseironic_parameters.driverisfake-hardware, the mode metalbox'sscripts/devel/fake-ironic-mode.shswitches to:The node is created in
enrolland left there, withoutinstance_info/extra, so nothing can be deployed on it.Cause
After computing the
driver_infodiff, the sync drops the driver's password key from it, because Ironic never returns the password. That key is looked up withdriver_params[driver]["password"], anddriver_paramsonly listsipmiandredfish, so any other driver raisesKeyError. This became reachable on the first sync once new nodes started being created as stubs (onlydriverandautomated_clean) followed by the regular update procedure: the stub has nodriver_info, so there is always a diff. Before that, a fake-hardware node only hit the lookup if itsdriver_infochanged between syncs.Changes
mask_secretswith the device's secret template values, exactly as the dry-run output already does. The payload sent to Ironic is unchanged. This comes first because the fix below lets the sync print updates for drivers it used to crash on: a driver without adriver_paramsentry that carries a password (e.g.ilo_password) keeps it in the diff..get(), so a driver without an entry has no password key to drop.Known limitations
The live output now has the same masking as the dry-run output, no stricter. These limitations of
mask_secretsand the display code apply to both and are not changed here:*password*,*secret*,ironic_osism_*) and by the values of secret template variables. Credentials under other names (e.g.ipmi_hex_kg_key,snmp_community) are printed.extraare JSON strings, so secret keys inside them (e.g. in the vault-decryptedfrr_parameters) are not seen by key-based masking.Testing
sync_ironicpassing template variables through; a fake-hardware node on the create path (with the realdeep_compare) reachingbaremetal_node_update; the dry-run update branch for fake-hardware; an unmapped driver's password masked in the output.🤖 Generated with Claude Code