Skip to content

[rush] Fix declared environment snapshot hashing - #6203

Open
Yashwanth Reddy Verupaka (yashwanth195) wants to merge 3 commits into
microsoft:mainfrom
yashwanth195:user/yverupaka/rush-environment-hashing-pr
Open

Yashwanth Reddy Verupaka (yashwanth195) wants to merge 3 commits into
microsoft:mainfrom
yashwanth195:user/yverupaka/rush-environment-hashing-pr

Conversation

@yashwanth195

Copy link
Copy Markdown
Contributor

Summary

Fix Rush cache identity for declared environment inputs: capture supplied values at snapshot creation, distinguish unset from empty, and resolve Windows environment names case-insensitively.

Details

  • Reuse EnvironmentMap for snapshot capture, explicit per-operation environments, and previous-snapshot comparison. Preserve memoization, undeclared-input exclusion, and non-Windows case sensitivity.
  • Hash JSON tuples to prevent delimiter collisions. Operations hashing declared environment variables intentionally receive new cache keys; no public API changes.
  • Standalone four-file fix. Intentionally excludes the source snapshot's additional-file/output-folder validation and all sharding APIs, plugins, and fixtures.

How it was tested

Windows, Node 24.22.1, Rush 5.182.0:

Command Result
node common\scripts\install-run-rush.js build --to @microsoft/rush-lib --parallelism 4 Passed: 36 successful operations, 4 no-op; includes TypeScript, lint, API Extractor, and webpack.
From libraries\rush-lib: node ..\..\common\scripts\install-run-rushx.js _phase:test --test-path-pattern InputsSnapshot --max-workers 1 --disable-code-coverage Passed: 46 tests, 0 failures.
node common\scripts\install-run-rush.js prettier Passed.
node common\scripts\install-run-rush.js change --verify --target-branch upstream/main --no-fetch Passed.
git diff --cached --check Passed.

Coverage includes absent/empty/sentinel values, captured caller mutations, separator collisions, Windows and Linux name semantics, explicit operation environments before/after memoization, and previous-snapshot reuse. Jest reported fixture package-name collision warnings; the suite passed.

Build/tests preceded the final rebase onto c2acd010079c6c08e63ba25d0b136ed2c1b16ccd. Upstream changed only unrelated Heft plugin selection; all four validated files were verified byte-identical after rebasing. No broad sharding or coverage matrix was run.

Impacted documentation

None. The patch changefile documents the intentional cache-key change.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Case sensitivity vs. insensitivity needs to be globally consistent in Rush so that builds produce the same results on either platform.

return getStrictHash(input, project, useOperationEnvironment ? environment : undefined);
}

it('distinguishes absence, empty values and literal sentinel strings', () => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does absent vs. empty string matter?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unset value takes the default in process.env.PUBLIC_PATH ?? '/default/', while an empty string overrides it. Those can produce different outputs, so their cache hashes must differ. Added a regression covering that distinction.

* Selects the compatibility contract for operation inputs.
* @beta
*/
export type InputSnapshotPolicy = 'legacy' | 'strict-v1';

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a strict correctness fix, the only backwards compatibility required is to ensure that we don't produce the same hash for different content.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the legacy/strict policy split. Declared environment inputs now use an unambiguous encoding separated from the old encoding. Operations without declared environment inputs retain their existing hash bytes.

/**
* The environment to use for `dependsOnEnvVars`. By default performs a snapshot of process.env upon construction.
* The environment for `dependsOnEnvVars`. Strict queries capture its values upon construction.
* Legacy queries retain an explicitly supplied object, or copy process.env by default.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The architecture assumes the input object is immutable, and enforcing that is left as an exercise for the caller. In general components were designed to assume callers respect immutability constraints in the interests of performance.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the defensive copies and duplicate state. The producer captures one EnvironmentMap, and snapshots retain it under the existing caller-owned immutability contract.

Comment on lines +348 to +357
this.#projectMetadataMap =
(previousSnapshot &&
previousSnapshot.#tryDeriveProjectMetadataMap(
params,
environment,
nodeVersion,
globalAdditionalHashes,
operationAdditionalFileCount
)) ??
createProjectMetadataMap(hashes, lookupByPath, projectMap, rootDir);
(previousSnapshot && areEnvironmentsEqual(environment, previousSnapshot.#environment)
? previousSnapshot.#tryDeriveProjectMetadataMap(
params,
nodeVersion,
globalAdditionalHashes,
operationAdditionalFileCount,
previousSnapshot.#projectMetadataMap
)
: undefined) ?? createProjectMetadataMap(hashes, lookupByPath, projectMap, rootDir);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

    this.#projectMetadataMap =
      previousSnapshot?.#tryDeriveProjectMetadataMap(
        params,
        environment,
        nodeVersion,
        globalAdditionalHashes,
        operationAdditionalFileCount
      ) ??
      createProjectMetadataMap(hashes, lookupByPath, projectMap, rootDir);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this to #tryDeriveProjectMetadataMap(...) ?? createProjectMetadataMap(...). The helper handles an absent previous snapshot because our parser rejects optional chaining on private members.

Comment on lines +951 to +954
function getOutputLookupPath(rootDir: string, filePath: string): string {
const relativePath: string = Path.convertToSlashes(path.relative(rootDir, path.resolve(rootDir, filePath)));
return process.platform === 'win32' ? relativePath.toLowerCase() : relativePath;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filePath inputs are very explicitly constrained in their shape. Any .. or . path components are invalid, so there's no reason to perform path resolution. The provided path is purely a dictionary key into the additionalHashes table.

All delimiters are already normalized to /, so there's no reason to perform delimeter normalization.

Case sensitivity is file system specific, not OS specific.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to make the inputs snapshot case insensitive, we need to do so everywhere. We can't have the tooling producing different enforcement results on Windows vs. Linux.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed both. The supplied canonical paths are used directly as dictionary keys, matching the existing behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the OS-specific file-key folding. File-key handling now matches upstream and remains case-sensitive on every platform.

return true;
}

function areEnvironmentMapsEqual(a: EnvironmentMap, b: EnvironmentMap): boolean {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this functionality to EnvironmentMap itself, since it seems useful; that will also facilitate optimization like not creating superfluous temporary arrays.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added EnvironmentMap.equals() with focused tests and the API report. Both snapshot-reuse checks now use it without allocating temporary arrays.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:heft-sharding Sharding feature for heft especially for jest

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

2 participants