Remove pyarrow as a direct dependency

[thomasjpfan]

Tracking issue

Continues flyteorg/flyte#4418

Why are the changes needed?

From flyteorg/flyte#4418 (comment), pyarrow is the largest dependency. This PR removes the dependency and lazy loads it.

What changes were proposed in this pull request?

With this PR, pyarrow is now lazy loaded. The lazy loading mechanism is the same as the one used for pandas.

How was this patch tested?

In two of the test environments, pyarrow is removed to make sure flytekit works without pyarrow installed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.32%. Comparing base (4be4e33) to head (99f7922).

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2228       +/-   ##
===========================================
+ Coverage   76.16%   91.32%   +15.15%     
===========================================
  Files         243      144       -99     
  Lines       21282     6651    -14631     
  Branches     3915        0     -3915     
===========================================
- Hits        16210     6074    -10136     
+ Misses       4427      577     -3850     
+ Partials      645        0      -645     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomasjpfan]

I am putting this as a draft. Removing pyarrow removes the indirect dependency on numpy. (pyarrow was the only library that depends on numpy).

I need to make sure flytekit works without numpy installed as well.

[thomasjpfan]

@pingsutw This PR removes one of the biggest dependencies. On Linux amd64, pyarrow uncompressed is 140MB.

The downside is that pandas_df.to_parquet will not work out of the box, (which is used by StructuredDataset)

[pingsutw]

The downside is that pandas_df.to_parquet will not work out of the box, (which is used by StructuredDataset)

I just tested it and the error message is clear, so I think it's fine.

import pandas as pd

from flytekit import task, workflow, StructuredDataset, ImageSpec

df = pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [1, 22]})

new_flytekit = "git+https://github.com/thomasjpfan/flytekit.git@603d3996bc4ed6907ba6c0296ae395e80f8e1dfc"
image_spec = ImageSpec(
    base_image="python:3.10-slim-bookworm",
    registry="pingsutw",
    packages=[new_flytekit, "pandas"],
    apt_packages=["git"]
)


@task(container_image=image_spec)
def t1(sd: StructuredDataset) -> StructuredDataset:
    print(sd.open(pd.DataFrame).all())
    return sd


@task(container_image=image_spec.with_packages("pyarrow"))
def t2(sd: StructuredDataset) -> StructuredDataset:
    print(sd.open(pd.DataFrame).all())
    return sd


@workflow
def wf():
    t1(sd=StructuredDataset(df))
    t2(sd=StructuredDataset(df))

[pingsutw]

@eapolinario @wild-endeavor wdyt