File and load_file should take extra params to give to pandas

[fritz-astronomer]

Please describe the feature you'd like to see
I may need to pass additional parameters for Pandas, describing my data
e.g. if I need to approximate

pd.read_csv("s3://my_bucket/my_file.csv", sep='|')

I am currently unable to utilize `astro-sdk

Describe the solution you'd like
File() should take an kwargs or some other param which can be fed down to pandas or whatever is doing the underlying reading

Are there any alternatives to this feature?
🤷

Acceptance Criteria

• All checks and tests in the CI should pass
• Unit tests (90% code coverage or more, once available)
• Integration tests (if the feature relates to a new database or external service)
• Example DAG
• Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
• Exception handling in case of errors
• Logging (are we exposing useful information to the user? e.g. source and destination)
• Improve the documentation (README, Sphinx, and any other relevant)
• How to use Guide for the feature (example)

[tatiana]

@fritz-astronomer, we are currently improving the load performance, and we won't be using Pandas for many of the load operations. Therefore, I believe it is unlikely that the Astro SDK would support any Pandas kwargs.

That said, I believe supporting different CSV delimiters is a good feature. We could add support for this as part of the File object, passed to the load operation. Would this solve the issue, or do you have additional needs to load this CSV?

[fritz-astronomer]

@tatiana - I think that would suffice.

A note though: I imagine that handling so much edge cases of file specifications could be a lot of code that may otherwise exist though. Maybe File can generally be some thin wrapper on top of some other file parsing python library? 🤷 Thinking of CSV, Avro, Parquet and how each of them can vary in really small annoying ways to parse.

[fhoda]

I had a similar issue, but with wanting to pass Column names. My file doesn't have a header row so I want to be able to use names=[col1, col2, ...], header=None when loading a file to a dataframe with the SDK, but cannot do this. I have a workaround as my next step is a transform step, but if that wasn't the case it would be a lot more difficult to supply column names.

If we allow the ability to load to a pandas dataframe I think we should consider allowing the passing of kwargs down to it.

[pankajastro]

As pointed out by @tatiana we are trying to avoid pandas because of the performance. but still, I have added the pd_kw #1225 to discuss more with the team and conclude on this.

[pankajastro]

possible duplicate #1264