Add bigquery example #521
Add bigquery example #521
Conversation
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
|
cc @pradithya / @vglocus can you guys take a look and help review? |
|
|
||
| @workflow | ||
| def full_bigquery_wf(version: int): | ||
| return bigquery_task_templatized_query(version=version) |
There was a problem hiding this comment.
What is the output type? I think it'll be interesting to demonstrate how to use the output in the sample.
There was a problem hiding this comment.
The type will be StructuredDataset. it's a new type introduced after flytekit v0.30.0b3 and is a superset of flyte schema.
It can transparently read different data sources from (BigQuery, S3,...) and convert result tables or parquet file to pd.Dataframe or pyarrow.Table.
There was a problem hiding this comment.
for more detail, you can check flyteorg/flytekit#785
There was a problem hiding this comment.
@pradithya so we were going to use Flyte schema, but we have been working on a better representation for structureddatasets in general- dataframes, tensors
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
|
@pingsutw should we merge this? |
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
|
It's ready to be merged. |
|
cc @kumare3 @wild-endeavor @pradithya mind taking a look |
Signed-off-by: Kevin Su [email protected]
Add Bigquery example, Related to flyteorg/flyte#1776