Python SDK: start_offline_to_online_ingestion Fails with default ingestion jar configuration

[jpugliesi]

Expected Behavior

In the minimal_ride_hailing.ipynb example notebook, I expect the following cell to run:

job = client.start_offline_to_online_ingestion(
    driver_statistics,
    datetime(2020, 10, 10),
    datetime(2020, 10, 20)
)
# expect offline to online ingestion job to run

Current Behavior

In the minimal_ride_hailing.ipynb example notebook, the following cell:

job = client.start_offline_to_online_ingestion(
    driver_statistics,
    datetime(2020, 10, 10),
    datetime(2020, 10, 20)
)

Produces the following error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-72-e6363419621a> in <module>
      2     driver_statistics,
      3     datetime(2020, 10, 10),
----> 4     datetime(2020, 10, 20)
      5 )

~/.local/lib/python3.7/site-packages/feast/client.py in start_offline_to_online_ingestion(self, feature_table, start, end)
   1065                 feature_table=feature_table,
   1066                 start=start,
-> 1067                 end=end,
   1068             )
   1069         else:

~/.local/lib/python3.7/site-packages/feast/pyspark/launcher.py in start_offline_to_online_ingestion(client, project, feature_table, start, end)
    250             ),
    251             deadletter_path=client._config.get(opt.DEADLETTER_PATH),
--> 252             stencil_url=client._config.get(opt.STENCIL_URL),
    253         )
    254     )

~/.local/lib/python3.7/site-packages/feast/pyspark/launchers/aws/emr.py in offline_to_online_ingestion(self, ingestion_job_params)
    256 
    257         jar_s3_path = _upload_jar(
--> 258             self._staging_location, ingestion_job_params.get_main_file_path()
    259         )
    260         step = _sync_offline_to_online_step(

~/.local/lib/python3.7/site-packages/feast/pyspark/launchers/aws/emr_utils.py in _upload_jar(jar_s3_prefix, local_path)
    127 
    128 def _upload_jar(jar_s3_prefix: str, local_path: str) -> str:
--> 129     with open(local_path, "rb") as f:
    130         return _s3_upload(
    131             f,

FileNotFoundError: [Errno 2] No such file or directory: 'https://storage.googleapis.com/feast-jobs/spark/ingestion/feast-ingestion-spark-develop.jar'

Steps to reproduce

1. install feast

pip install feast==0.8.2

2. Run the cells in the example notebook. Note that I have configured the following Client fields, but not the spark_ingestion_jar config (this config works fine for defining features in feast):

client = Client(
    core_url='feast-feast-core.feast-dev:6565',
    spark_launcher="emr",
    emr_cluster_id="<redacted>",
    emr_region="<redacted>",
    spark_staging_location="<redacted>",
    emr_log_location="<redacted>",
    historical_feature_output_location="<redacted>"
)

Specifications

• Version: 0.8.2

[jpugliesi]

I now see this appears related to #1266

[jpugliesi]

I get a similar error even when defining spark_ingestion_jar configuration on the client, i.e. spark_ingestion_jar=s3://my-bucket/feast-ingestion.jar (with a valid jar of course):

1. The jar exists:

$ aws s3 ls s3://my-bucket/feast-ingestion.jar
2021-01-19 23:44:12   45031646 feast-ingestion.jar

2. Failed attempt to kick off ingestion:

job = client.start_offline_to_online_ingestion(
    driver_statistics,
    datetime(2020, 10, 10),
    datetime(2020, 10, 20)
)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-127-e6363419621a> in <module>
      2     driver_statistics,
      3     datetime(2020, 10, 10),
----> 4     datetime(2020, 10, 20)
      5 )

~/.local/lib/python3.7/site-packages/feast/client.py in start_offline_to_online_ingestion(self, feature_table, start, end)
   1167                 feature_table=feature_table,
   1168                 start=start,
-> 1169                 end=end,
   1170             )
   1171         else:

~/.local/lib/python3.7/site-packages/feast/pyspark/launcher.py in start_offline_to_online_ingestion(client, project, feature_table, start, end)
    280             ),
    281             deadletter_path=client._config.get(opt.DEADLETTER_PATH),
--> 282             stencil_url=client._config.get(opt.STENCIL_URL),
    283         )
    284     )

~/.local/lib/python3.7/site-packages/feast/pyspark/launchers/aws/emr.py in offline_to_online_ingestion(self, ingestion_job_params)
    269 
    270         jar_s3_path = _upload_jar(
--> 271             self._staging_location, ingestion_job_params.get_main_file_path()
    272         )
    273         step = _sync_offline_to_online_step(

~/.local/lib/python3.7/site-packages/feast/pyspark/launchers/aws/emr_utils.py in _upload_jar(jar_s3_prefix, local_path)
     82 
     83 def _upload_jar(jar_s3_prefix: str, local_path: str) -> str:
---> 84     with open(local_path, "rb") as f:
     85         uri = urlparse(os.path.join(jar_s3_prefix, os.path.basename(local_path)))
     86         return urlunparse(

FileNotFoundError: [Errno 2] No such file or directory: 's3://my-bucket/feast-ingestion.jar'