Repo to start learning pyspark. Includes docker-compose of spark master and workers with gcs connectors with optional standalone airflow and inconsistent jupyter notebook. Intention is to mirror work done in https://github.com/HybridNeos/comp653_final through spark dataframes and possibly spark ML while using Airflow instead of dbt as the orchestrator.
-
Notifications
You must be signed in to change notification settings - Fork 0
HybridNeos/pyspark_learning_stack
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Repo to start learning pyspark. Includes docker-compose of spark master and workers with gcs connectors with optional standalone airflow and inconsistent jupyter notebook.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published