Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Added docs on how to handle a large response when using FlyteRemote.sync #2932

Merged
merged 17 commits into from
Oct 11, 2022
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions rsts/community/troubleshoot.rst
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,11 @@ Troubles With Inconsistent Names for Pods and Downstream Resources
- Flyte uses the format ``executionid-node-id-attempt`` from the node to assign a name to a Kubernetes pod or downstream resource.
- But if this is an invalid name for a Kubernetes pod, Flyte assigns a valid name of random characters instead.

Troubles with handling large responses in ``FlyteRemote.sync``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- ``Received message larger than max (xxx vs. 4194304)`` usually crops up when the message size is too large.
- To fix this, edit the ``flyte-admin-base-config`` config map using the command ``kubectl edit cm flyte-admin-base-config -n flyte`` to increase the ``maxMessageSizeBytes`` value.


I Still Need Help!
Expand Down
4 changes: 2 additions & 2 deletions rsts/concepts/data_management.rst
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,8 @@ Between Tasks
Bringing in Your Own Datastores for Raw Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Flytekit has a pluggable data persistence layer as explained in :std:ref:`data.extend:extend data persistence layer`. This is driven by PROTOCOL.
Flytekit has a pluggable data persistence layer as explained in :ref:`extend data persistence layer <https://docs.flyte.org/projects/flytekit/en/latest/data.extend.html#data-persistence-layer>`. This is driven by PROTOCOL.
samhita-alla marked this conversation as resolved.
Show resolved Hide resolved
For example, it is theoretically possible to use S3 ``s3://`` for metadata and GCS ``gcs://`` for raw data. It is also possible to create your own protocol ``my_fs://``, to change how data is stored and accessed.
But for Metadata, the data should be accessible to Flyte control plane.

Data persistence is also pluggable. By default, it supports all major blob stores and uses an interface defined in Flytestdlib.
Data persistence is also pluggable. By default, it supports all major blob stores and uses an interface defined in Flytestdlib.