Simplify component imports

[felix-datatonic]

Description

• create python packages pipelines, aiplatform-components, bigquery-components where pipelines depend on the component packages
• each package includes:
  • pyproject.toml
  • Pipfile
  • src folder with all package logic
  • tests folder for unit tests
• pipelines package was modified with pipenv install -e ../aiplatform-components ../bigquery-components
  • this allows for code changes to reflect in this package without re-installing the dependent packages
• update Makefile
  • sync assets optionally when running E2E tests (this aligns better with cloud build triggers)
  • setup-all-components installs aiplatform-components and bigquery-components
• update cloud build triggers
  • use make commands where possible
• update KFP version from 1.8.9 to 1.8.21
  • due to the issue outlined in fix(sdk): fix appengine import error

This removes the need to compile each component individually before compiling & running a pipeline.
As a result, the dev workflow has one less step and the overall compilation time locally and in cloud build reduces.
Further, this preserves isolation between each component group and pipelines.

How has this been tested?

• make setup
• make setup-components
• make setup-all-components
• make test-trigger
• make compile-pipeline
• make test-components
• make test-all-components
• make run
• make e2e-tests

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have successfully run the E2E tests, and have included the links to the pipeline runs below
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have updated any relevant documentation to reflect my changes
• I have assigned a reviewer and messaged them

Pipeline run links:

[ghost]

/gcbrun