Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Update Development Environment Setup Guide for MonoRepo #4257

Merged
merged 4 commits into from
Oct 25, 2023
Merged

[Docs] Update Development Environment Setup Guide for MonoRepo #4257

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
305cbc6
contribution guide improvement
Oct 18, 2023
bfe376a
nit
Oct 18, 2023
2ee3633
apply pingsu's advice
Oct 18, 2023
a4c4c38
url update and codespell fixed
Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 30 additions & 38 deletions rsts/community/contribute.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,48 +415,31 @@ that integrates all Flyte components into a single binary.
**4. Build single binary with your own code.**


The following instructions provide guidance on how to build single binary with your customized code, using ``flyteadmin`` as an example.
The following instructions provide guidance on how to build single binary with your customized code under the ``flyteadmin`` as an example.


- **Note** Although we'll use ``flyteadmin`` as an example, these steps can be applied to other Flyte components or libraries as well.
``{flyteadmin}`` below can be substituted with other Flyte components/libraries: ``flyteidl``, ``flyteplugins``, ``flytepropeller``, ``datacatalog``, or ``flytestdlib``.

- **Note** If modifications are needed in multiple components/libraries, the steps will need to be repeated for each component/library.
- **Note** If you want to learn how flyte compiles those components and replace the repositories, you can study how ``go mod edit`` works.

.. code:: shell

# Step1: Fork and clone the {flyteadmin} repository, modify the source code accordingly.
git clone https://github.com/flyteorg/flyteadmin.git
cd flyteadmin

# Step2.1: {Flyteadmin} uses Go 1.19, so make sure to switch to Go 1.19.
# Step1: Install Go. Flyte uses Go 1.19, so make sure to switch to Go 1.19.
export PATH=$PATH:$(go env GOPATH)/bin
go install golang.org/dl/go1.19@latest
go1.19 download
export GOROOT=$(go1.19 env GOROOT)
export PATH="$GOROOT/bin:$PATH"
# Step2.2: You may need to install goimports to fix lint errors.

# You may need to install goimports to fix lint errors.
# Refer to https://pkg.go.dev/golang.org/x/tools/cmd/goimports
go install golang.org/x/tools/cmd/goimports@latest
export PATH=$(go env GOPATH)/bin:$PATH

# Step 3.1: Review the go.mod file in the {flyteadmin} directory to identify the Flyte components/libraries that {flyteadmin} relies on.
# If you have modified any of these components/libraries, use `go mod edit -replace` in the {flyteadmin} repo to replace original components/libraries with your customized ones.
# For instance, if you have also modified `flytepropeller`, run the following commands:
go mod edit -replace github.com/flyteorg/flytepropeller=/home/ubuntu/flytepropeller #replace with your own local path to flytepropeller
# Step2: Go to the {flyteadmin} repository, modify the source code accordingly.
cd flyte/flyteadmin

# Step 3.2: Generate code, fix lint errors and run unit tests for {flyteadmin}.
# Note, flyteidl does not have unit tests, so you can skip the `make test_unit` command.
# Note, flytestdlib only have `make generate` command.
make generate
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Future-Outlier I think we still need to execute the following commands:

make generate
make lint
make test_unit

For instance, to generate the protobuf code, I guess we still need to run make generate under flyteidl folder? Correct me if I am wrong.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right, XD.

make lint
make test_unit

# Step4: Now, you can build the single binary. Go back to Flyte directory, run `go mod edit -replace` to replace the {flyteadmin} code with your own.
go mod edit -replace github.com/flyteorg/flyteadmin=/home/ubuntu/flyteadmin #replace with your own local path to {flyteadmin}

# Step5: Rebuild and rerun the single binary based on your own code.
# Step3: Now, you can build the single binary. Go back to Flyte directory.
go mod tidy
make compile
POD_NAMESPACE=flyte ./flyte start --config flyte-single-binary-local.yaml
Expand All @@ -469,12 +452,10 @@ The following instructions provide guidance on how to build single binary with y
# Step1: Install flytekit
pip install flytekit && export PATH=$PATH:/home/ubuntu/.local/bin

# Step2: The flytesnacks repository provides a lot of useful examples.
git clone https://github.com/flyteorg/flytesnacks && cd flytesnacks

# Step3: Run a hello world example
pyflyte run --remote examples/basics/basics/hello_world.py my_wf
# Step2: Run a hello world example
pyflyte run --remote https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py hello_world_wf
# Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/fd63f88a55fed4bba846 to see execution in the console.
# You can go to the [flytesnacks repository](https://github.com/flyteorg/flytesnacks) to see more useful examples.

**6. Tear down the k3s cluster after finishing developing.**

Expand Down Expand Up @@ -530,20 +511,32 @@ If not, we can start backends with a single command.
source ~/.virtualenvs/flytekit/bin/activate
make setup
pip install -e .
pip install gsutil awscli
# If you are also developing the plugins, execute the following:

# If you are also developing the plugins, consider the following:

# Installing Specific Plugins:
# If you wish to only use few plugins, you can install them individually.
# Take [Flytekit BigQuery Plugin](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-bigquery#flytekit-bigquery-plugin) for example:
# You have to go to the bigquery plugin folder and install it.
cd plugins/flytekit-bigquery/
pip install -e .
# Now you can use the bigquery plugin, and the performance is fast.

# (Optional) Installing All Plugins:
# If you wish to install all available plugins, you can execute the command below.
# However, it's not typically recommended because the current version of plugins does not support
# lazy loading. This can lead to a slowdown in the performance of your Python engine.
cd plugins
pip install -e .
# Now you can use all plugins, but the performance is slow.

# Step2: Modify the source code for flytekit, then run unit tests and lint.
make lint
make test

# Step3: Run a hello world sample to test locally
git clone https://github.com/flyteorg/flytesnacks
cd flytesnacks
python3 examples/basics/basics/hello_world.py my_wf
# Running my_wf() hello world
pyflyte run https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py hello_world_wf
# Running hello_world_wf() hello world

**3. Run workflow in sandbox.**

Expand Down Expand Up @@ -584,9 +577,8 @@ the Flyte cluster, and finally submit the workflow.
docker push ${FLYTE_INTERNAL_IMAGE}

# Step4: Submit a hello world workflow to the Flyte cluster
git clone https://github.com/flyteorg/flytesnacks
cd flytesnacks
pyflyte run --image ${FLYTE_INTERNAL_IMAGE} --remote examples/basics/basics/hello_world.py my_wf
pyflyte run --image ${FLYTE_INTERNAL_IMAGE} --remote https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/basics/basics/hello_world.py hello_world_wf
# Go to http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f5c17e1b5640c4336bf8 to see execution in the console.

How to setup dev environment for flyteconsole?
Expand Down
Loading