Repository navigation
Push referenced libraries with cloud commands' --push - #749
quantconnect-kronos[bot] wants to merge 2 commits into
Conversation
Run the real push path for a project that references a library, by project name and by cloud id: the changed library is uploaded, the project is updated with the library's cloud id, and a failed library upload stops the command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pushing a project's referenced libraries with --push is the right approach: get_cloud_project now calls push_projects the same way PushManager.push_project (lean cloud push) does, on both branches. CI is green on 2bc17dd. I ran the cloud tests locally (173 passed). The 7 new and updated tests in test_cloud_project_manager.py fail when cloud_project_manager.py is reverted to master and pass with the change.
A person needs to decide: libraries the user can only read
Every library push now has to succeed for --push to go ahead, even when the library has no local changes. PushManager._push_metadata always sends projects.update with the files and doesn't check for changes first. The cloud turns down that update from any user without write permission on the library, whatever it contains. That happens when a library is shared with the user read-only, or another collaborator holds its write lock. For a user whose project references such a library, lean cloud backtest/optimize/live deploy <project> --push now always fails, and nothing they can do locally fixes it. --force doesn't help because write permission is checked first.
I reproduced it end to end. A project references a library that is shared with the user read-only and has no local changes:
master:lean cloud backtest <project> --pushpushed the project ([1/1]), compiled and ran the backtest to completion.- This branch:
[1/2] Pushing 'Library/...'→Cannot push 'Library/...': You do not have write permission on this project., thenError: Cannot push '<project>' to the cloud, aborting so the cloud does not run outdated files, exit code 1, nothing compiled. The error names the project rather than the library. It also suggests--force, which doesn't apply here. lean cloud push --project <project>on this branch still only warns and exits 0, as onmaster.
#719 asks for a failed library upload to stop the command, so the cloud doesn't run stale library code. That holds when the user changed the library. When the user can't write to the library, the cloud copy is the only version they can use, and stopping the command blocks a workflow that works on master. Neither push_projects nor get_cloud_project can tell these cases apart today. The only workaround is to drop --push and push separately. The ways forward I see, for a QuantConnect maintainer to choose:
- Accept the limitation: users of read-only libraries lose
--push. Say so in the description and, ideally, in the error message. - Make only the project's own push failure stop the command, and let library failures stay warnings, as in
lean cloud push. This drops part of what #719 asks for (a changed library that fails to upload would run stale). - Skip pushing a library when the push would change nothing, or when the user can't write to it. This needs a change in
PushManagerbeyond what this PR touches, such as comparing with the cloud files first.
Other notes
- Tests cover a changed library and a failed library upload, for both the local name and the cloud id inputs, through the real push path. They don't cover a library the user can't write to. Whichever option is chosen should come with a test for it.
- Not caused here and already tracked: #741 (a library renamed by the push leaves the project's reference broken).
- No other repositories need an update.
Risk: risk:high. The change stops --push from working for projects with read-only libraries (shown above), and which behavior is intended is a product decision. Once that's decided and covered by a test, the change is small and easy to revert.
| if push: | ||
| if not self._push_manager.push_projects([local_path]): | ||
| libraries = self._project_manager.get_project_libraries(local_path) | ||
| if not self._push_manager.push_projects([local_path], libraries): |
There was a problem hiding this comment.
Read-only libraries are pushed here too, and when one fails the command aborts (shown in the review body). _push_metadata always sends an update, even when the library has no local changes, and the cloud rejects it from a user without write permission on the library. A person needs to decide how to handle this; the options are in the review body.
|
Confirmed: with this change, a project that references a library the user can only read (shared read-only, or locked by another collaborator) can no longer use I don't see a small change that keeps both halves of #719: changed libraries are uploaded, and a failed upload stops the command. The CLI can't tell "failed to upload my changes" from "can't write a library I didn't change" without new checks in
My suggestion is 3 if a wider change is acceptable, otherwise 2. I'll build whichever you choose. |
There was a problem hiding this comment.
No new commits since my last review: the head is still 2bc17dd, and CI is green on it. Since then the description has an "Open decision" section and the author's comment lists the options. Both match what I found.
The open finding still holds. With this change, a project that references a library the user can only read can no longer run lean cloud backtest/optimize/live deploy --push. On master it works. A QuantConnect maintainer still needs to choose one of these:
- Keep this behavior, with an error that names the library.
- Make library push failures warnings, as
lean cloud pushdoes. - Skip libraries that need no push. This changes
PushManager.
Nobody has decided on this PR or on #719 yet. Whichever option is chosen needs a test for a library the user can't write to.
No other repositories need an update.
Risk: risk:high stays. The change breaks --push for users of read-only libraries until the decision above is made and covered by a test.
Closes #719
What changed
lean cloud backtest,lean cloud optimizeandlean cloud live deploywith--pushresolve the project throughCloudProjectManager.get_cloud_project, which pushed only the project itself. A library the project references was not uploaded, so the cloud compiled and ran against the library's older cloud contents.lean cloud pushalready uploads them, becausePushManager.push_projectpassesProjectManager.get_project_libraries(project)topush_projects.Both push branches of
get_cloud_project(local project directory, and cloud name or id mapped to a local directory) now pass the project's libraries, including nested ones, topush_projects, the same way. Sincepush_projectsreturnsFalsewhen any project or library fails to push, a failed library upload now raisesPushFailedErrorand the command stops before compiling.The fix commit is cherry-picked from #725 by @pcbeingused333, keeping its authorship, with its test adapted to the current resolver, which follows a project directory renamed by the push (#734).
Backwards compatible: projects without libraries push exactly as before.
Tests
test_get_cloud_project_pushes_libraries_referenced_by_the_project(by project name and by cloud id) runs the real push path. It checks that the changed library is uploaded and that the project is updated with the library's cloud id.test_get_cloud_project_raises_when_push_of_referenced_library_fails(both inputs): a failed library upload raisesPushFailedError.push_projectscall.These new and updated tests fail on
master(7 failed) and pass with the change.python -m pytest -s -rs: 2866 passed, 81 skipped, 1 failed. The failure,tests/ui/test_app.py::test_command_process_runs_the_cli_as_a_child_process, fails the same way onmasterin my environment, where it compares against an installedleanpackage version ('lean 1.0.233' == 'lean 0.0.0-dev'). It is not related to this change.Tested end to end with a Python project that references a Python library. Both were pushed once, then the library function was changed from returning
"v1"to"v2". The algorithm logs the library's value and raises when it isn't"v2":master:lean cloud backtest <project> --pushpushed only the project ([1/1]), and the backtest failed withStale library version v1.[1/2],[2/2]), and the backtest completed.Cannot push '<project>' to the cloud, aborting so the cloud does not run outdated filesbefore compiling.Related, not changed here: #741 (a library renamed by the push leaves the project's reference broken) also affects
lean cloud push.Open decision
Review found that a library the user can only read now makes
--pushfail, wheremasterruns the command. Waiting for a maintainer to choose how to handle it (options in the comments): keep and clarify the error, make library failures warnings, or skip libraries that need no push.