Skip to content

Push referenced libraries with cloud commands' --push - #749

Draft
quantconnect-kronos[bot] wants to merge 2 commits into
masterfrom
kronos/bug-719-push-referenced-libraries
Draft

quantconnect-kronos[bot] wants to merge 2 commits into
masterfrom
kronos/bug-719-push-referenced-libraries

Conversation

@quantconnect-kronos

@quantconnect-kronos quantconnect-kronos Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #719

What changed

lean cloud backtest, lean cloud optimize and lean cloud live deploy with --push resolve the project through CloudProjectManager.get_cloud_project, which pushed only the project itself. A library the project references was not uploaded, so the cloud compiled and ran against the library's older cloud contents. lean cloud push already uploads them, because PushManager.push_project passes ProjectManager.get_project_libraries(project) to push_projects.

Both push branches of get_cloud_project (local project directory, and cloud name or id mapped to a local directory) now pass the project's libraries, including nested ones, to push_projects, the same way. Since push_projects returns False when any project or library fails to push, a failed library upload now raises PushFailedError and the command stops before compiling.

The fix commit is cherry-picked from #725 by @pcbeingused333, keeping its authorship, with its test adapted to the current resolver, which follows a project directory renamed by the push (#734).

Backwards compatible: projects without libraries push exactly as before.

Tests

  • test_get_cloud_project_pushes_libraries_referenced_by_the_project (by project name and by cloud id) runs the real push path. It checks that the changed library is uploaded and that the project is updated with the library's cloud id.
  • test_get_cloud_project_raises_when_push_of_referenced_library_fails (both inputs): a failed library upload raises PushFailedError.
  • Existing push tests now assert the new push_projects call.

These new and updated tests fail on master (7 failed) and pass with the change.

python -m pytest -s -rs: 2866 passed, 81 skipped, 1 failed. The failure, tests/ui/test_app.py::test_command_process_runs_the_cli_as_a_child_process, fails the same way on master in my environment, where it compares against an installed lean package version ('lean 1.0.233' == 'lean 0.0.0-dev'). It is not related to this change.

Tested end to end with a Python project that references a Python library. Both were pushed once, then the library function was changed from returning "v1" to "v2". The algorithm logs the library's value and raises when it isn't "v2":

  • master: lean cloud backtest <project> --push pushed only the project ([1/1]), and the backtest failed with Stale library version v1.
  • This branch: it pushed the library and then the project ([1/2], [2/2]), and the backtest completed.
  • This branch, with the library's push failing (its cloud id pointing at a missing project): the command stopped with Cannot push '<project>' to the cloud, aborting so the cloud does not run outdated files before compiling.

Related, not changed here: #741 (a library renamed by the push leaves the project's reference broken) also affects lean cloud push.

Open decision

Review found that a library the user can only read now makes --push fail, where master runs the command. Waiting for a maintainer to choose how to handle it (options in the comments): keep and clarify the error, make library failures warnings, or skip libraries that need no push.

pcbeingused333 and others added 2 commits November 17, 2026 20:09
Run the real push path for a project that references a library, by
project name and by cloud id: the changed library is uploaded, the
project is updated with the library's cloud id, and a failed library
upload stops the command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing a project's referenced libraries with --push is the right approach: get_cloud_project now calls push_projects the same way PushManager.push_project (lean cloud push) does, on both branches. CI is green on 2bc17dd. I ran the cloud tests locally (173 passed). The 7 new and updated tests in test_cloud_project_manager.py fail when cloud_project_manager.py is reverted to master and pass with the change.

A person needs to decide: libraries the user can only read

Every library push now has to succeed for --push to go ahead, even when the library has no local changes. PushManager._push_metadata always sends projects.update with the files and doesn't check for changes first. The cloud turns down that update from any user without write permission on the library, whatever it contains. That happens when a library is shared with the user read-only, or another collaborator holds its write lock. For a user whose project references such a library, lean cloud backtest/optimize/live deploy <project> --push now always fails, and nothing they can do locally fixes it. --force doesn't help because write permission is checked first.

I reproduced it end to end. A project references a library that is shared with the user read-only and has no local changes:

  • master: lean cloud backtest <project> --push pushed the project ([1/1]), compiled and ran the backtest to completion.
  • This branch: [1/2] Pushing 'Library/...' → Cannot push 'Library/...': You do not have write permission on this project., then Error: Cannot push '<project>' to the cloud, aborting so the cloud does not run outdated files, exit code 1, nothing compiled. The error names the project rather than the library. It also suggests --force, which doesn't apply here.
  • lean cloud push --project <project> on this branch still only warns and exits 0, as on master.

#719 asks for a failed library upload to stop the command, so the cloud doesn't run stale library code. That holds when the user changed the library. When the user can't write to the library, the cloud copy is the only version they can use, and stopping the command blocks a workflow that works on master. Neither push_projects nor get_cloud_project can tell these cases apart today. The only workaround is to drop --push and push separately. The ways forward I see, for a QuantConnect maintainer to choose:

  1. Accept the limitation: users of read-only libraries lose --push. Say so in the description and, ideally, in the error message.
  2. Make only the project's own push failure stop the command, and let library failures stay warnings, as in lean cloud push. This drops part of what #719 asks for (a changed library that fails to upload would run stale).
  3. Skip pushing a library when the push would change nothing, or when the user can't write to it. This needs a change in PushManager beyond what this PR touches, such as comparing with the cloud files first.

Other notes

  • Tests cover a changed library and a failed library upload, for both the local name and the cloud id inputs, through the real push path. They don't cover a library the user can't write to. Whichever option is chosen should come with a test for it.
  • Not caused here and already tracked: #741 (a library renamed by the push leaves the project's reference broken).
  • No other repositories need an update.

Risk: risk:high. The change stops --push from working for projects with read-only libraries (shown above), and which behavior is intended is a product decision. Once that's decided and covered by a test, the change is small and easy to revert.

if push:
if not self._push_manager.push_projects([local_path]):
libraries = self._project_manager.get_project_libraries(local_path)
if not self._push_manager.push_projects([local_path], libraries):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read-only libraries are pushed here too, and when one fails the command aborts (shown in the review body). _push_metadata always sends an update, even when the library has no local changes, and the cloud rejects it from a user without write permission on the library. A person needs to decide how to handle this; the options are in the review body.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:high High risk and removed kronos:review:working review: in progress labels Nov 17, 2026
@quantconnect-kronos

Copy link
Copy Markdown
Contributor Author

Confirmed: with this change, a project that references a library the user can only read (shared read-only, or locked by another collaborator) can no longer use --push on lean cloud backtest/optimize/live deploy. Every library push sends projects.update, which the cloud rejects without write permission. On master those commands skip the library and run.

I don't see a small change that keeps both halves of #719: changed libraries are uploaded, and a failed upload stops the command. The CLI can't tell "failed to upload my changes" from "can't write a library I didn't change" without new checks in PushManager. A maintainer needs to pick one of these options:

  1. Keep as is: read-only libraries block --push. I'd update the error to name the library and suggest pushing without --push, plus a test.
  2. Library failures warn, project failures stop: same as lean cloud push. Changed libraries are still uploaded when possible, but a changed library that fails to upload would run stale in the cloud.
  3. Skip libraries that need no push: compare the library's files with the cloud copy, or check the user's permission from the project's collaborators, before updating. This is the most complete, but it changes PushManager's push path for lean cloud push too.

My suggestion is 3 if a wider change is acceptable, otherwise 2. I'll build whichever you choose.

@quantconnect-kronos quantconnect-kronos Bot added the kronos:review Request review label Nov 17, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Nov 17, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new commits since my last review: the head is still 2bc17dd, and CI is green on it. Since then the description has an "Open decision" section and the author's comment lists the options. Both match what I found.

The open finding still holds. With this change, a project that references a library the user can only read can no longer run lean cloud backtest/optimize/live deploy --push. On master it works. A QuantConnect maintainer still needs to choose one of these:

  1. Keep this behavior, with an error that names the library.
  2. Make library push failures warnings, as lean cloud push does.
  3. Skip libraries that need no push. This changes PushManager.

Nobody has decided on this PR or on #719 yet. Whichever option is chosen needs a test for a library the user can't write to.

No other repositories need an update.

Risk: risk:high stays. The change breaks --push for users of read-only libraries until the decision above is made and covered by a test.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 17, 2026
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:idle develop: paused and removed kronos:develop:working develop: in progress labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kronos:develop:idle develop: paused risk:high High risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud commands with --push do not upload referenced libraries

1 participant