Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XComObjectStorageBackend can't deserialize_value on S3 path #39797

Closed
1 of 2 tasks
Rmjp opened this issue May 24, 2024 · 2 comments
Closed
1 of 2 tasks

XComObjectStorageBackend can't deserialize_value on S3 path #39797

Rmjp opened this issue May 24, 2024 · 2 comments
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet

Comments

@Rmjp
Copy link

Rmjp commented May 24, 2024

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I use s3 storage in xcom backend and it Error on xcom (Invalid Key), I find it because In _is_relative_to(ObjectStoragePath(data), p) return False because p from base_path that contain conn_id@... but data from the xcom (str(base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uuid4()}{suffix}")) ) that did't contain conn_id anymore. The _is_relative_to check storage_options(conn_id) that make it return False.

What you think should happen instead?

It should return True because data come from p (serialize_value and deserialize_value on XCOM).

How to reproduce

Use XComObjectStorageBackend in xcombackend. And xcom_objectstorage_path = s3://conn_id@airflow-test/ with conn_id = amazon web services, endpoint = "https://example.com".

Operating System

docker airflow slim-2.9.1-python3.9 run on Ubuntu 22.04.4 LTS host.

Versions of Apache Airflow Providers

apache-airflow-providers-common-io 1.3.1
apache-airflow-providers-amazon 8.19.0

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@Rmjp Rmjp added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels May 24, 2024
Copy link

boring-cyborg bot commented May 24, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@Rmjp
Copy link
Author

Rmjp commented May 24, 2024

Sorry, I didn't see #39602

@Rmjp Rmjp closed this as completed May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Development

No branches or pull requests

1 participant