Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ release (0.x.y) only fixes bugs. Breaking changes are called out below.

## Unreleased

- `datasets.import_big_file_data` docstring: on the current backend a data-lake import
answers a job map like any other import (checked live on the sandbox); the empty-body
`{}` case is for older backends.
- `read_tables` (and `read_table`, `read_records`, `read_all_records`): when the export
has no rows at all, the dataflow's `bigData` flag is looked up. A data-lake dataflow
gets a definite warning pointing to `export_to_file()`; a dataflow that answers
Expand Down
13 changes: 8 additions & 5 deletions docs/reference/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -1114,11 +1114,14 @@ Import file to dataset data (Large files)
| Design dataset | CUSTODIAN, STEWARD, EDITOR WRITE |
| EU dataset | CUSTODIAN, STEWARD |

Overrides the generated method. An empty response body returns `{}` (data lake
datasets answer `void`: the import is done inside the request, MasterDataLakes `DatasetControllerImpl:391-422`;
other datasets answer a job map), and `timeout` overrides the client's timeout for
this call only (a lake import can run for minutes). `preparation_code` is sent as
`code`.
Overrides the generated method. The answer is a job map (`jobId`, `pollingUrl`;
poll it with `jobs.wait_for_job`), also for data lake datasets on the current
backend (`MasterOneVersion`; checked live on the sandbox, where a two-row lake
import finished in about 20 s). Older backends answer a lake import with an
empty body (`void`, the import is done inside the request; `MasterDataLakes`
`DatasetControllerImpl:391-422`), returned as `{}`. `timeout` overrides the
client's timeout for this call only (a lake import can run for minutes).
`preparation_code` is sent as `code`.

**Live-tested notes:**

Expand Down
26 changes: 16 additions & 10 deletions reportnet_api/resources/datasets_ext.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,11 +265,14 @@ def import_big_file_data(

EU dataset: CUSTODIAN, STEWARD

Overrides the generated method. An empty response body returns `{}` (data lake
datasets answer `void`: the import is done inside the request, MasterDataLakes `DatasetControllerImpl:391-422`;
other datasets answer a job map), and `timeout` overrides the client's timeout for
this call only (a lake import can run for minutes). `preparation_code` is sent as
`code`."""
Overrides the generated method. The answer is a job map (`jobId`, `pollingUrl`;
poll it with `jobs.wait_for_job`), also for data lake datasets on the current
backend (`MasterOneVersion`; checked live on the sandbox, where a two-row lake
import finished in about 20 s). Older backends answer a lake import with an
empty body (`void`, the import is done inside the request; `MasterDataLakes`
`DatasetControllerImpl:391-422`), returned as `{}`. `timeout` overrides the
client's timeout for this call only (a lake import can run for minutes).
`preparation_code` is sent as `code`."""
params = {
"dataflowId": dataflow_id,
"delimiter": delimiter,
Expand Down Expand Up @@ -640,11 +643,14 @@ async def import_big_file_data(

EU dataset: CUSTODIAN, STEWARD

Overrides the generated method. An empty response body returns `{}` (data lake
datasets answer `void`: the import is done inside the request, MasterDataLakes `DatasetControllerImpl:391-422`;
other datasets answer a job map), and `timeout` overrides the client's timeout for
this call only (a lake import can run for minutes). `preparation_code` is sent as
`code`."""
Overrides the generated method. The answer is a job map (`jobId`, `pollingUrl`;
poll it with `jobs.wait_for_job`), also for data lake datasets on the current
backend (`MasterOneVersion`; checked live on the sandbox, where a two-row lake
import finished in about 20 s). Older backends answer a lake import with an
empty body (`void`, the import is done inside the request; `MasterDataLakes`
`DatasetControllerImpl:391-422`), returned as `{}`. `timeout` overrides the
client's timeout for this call only (a lake import can run for minutes).
`preparation_code` is sent as `code`."""
params = {
"dataflowId": dataflow_id,
"delimiter": delimiter,
Expand Down