From 1ae898c0bd9c1bb2abf47af9251461b2963be79d Mon Sep 17 00:00:00 2001 From: Evan Tahler Date: Wed, 18 May 2022 08:46:04 -0700 Subject: [PATCH] More ways to reset your local environment (#12826) --- .../developing-locally.md | 49 ++++++++++++++----- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/docs/contributing-to-airbyte/developing-locally.md b/docs/contributing-to-airbyte/developing-locally.md index 7bfdd219e077..ebddea233cae 100644 --- a/docs/contributing-to-airbyte/developing-locally.md +++ b/docs/contributing-to-airbyte/developing-locally.md @@ -76,6 +76,7 @@ export CPPFLAGS="-I/usr/local/opt/openssl/include" ## Run in `dev` mode with `docker-compose` These instructions explain how to run a version of Airbyte that you are developing on (e.g. has not been released yet). + ```bash SUB_BUILD=PLATFORM ./gradlew build VERSION=dev docker-compose up @@ -97,7 +98,7 @@ SUB_BUILD=PLATFORM ./gradlew :airbyte-tests:acceptanceTests Test containers start Airbyte locally, run the tests, and shutdown Airbyte after running the tests. If you want to run acceptance tests against local Airbyte that is not managed by the test containers, you need to set `USE_EXTERNAL_DEPLOYMENT` environment variable to true: ```bash -USE_EXTERNAL_DEPLOYMENT=true SUB_BUILD=PLATFORM ./gradlew :airbyte-tests:acceptanceTests +USE_EXTERNAL_DEPLOYMENT=true SUB_BUILD=PLATFORM ./gradlew :airbyte-tests:acceptanceTests ``` ## Run formatting automation/tests @@ -105,13 +106,17 @@ USE_EXTERNAL_DEPLOYMENT=true SUB_BUILD=PLATFORM ./gradlew :airbyte-tests:accepta Airbyte runs a code formatter as part of the build to enforce code styles. You should run the formatter yourself before submitting a PR (otherwise the build will fail). The command to run formatting varies slightly depending on which part of the codebase you are working in. + ### Platform + If you are working in the platform run `SUB_BUILD=PLATFORM ./gradlew format` from the root of the repo. ### Connector + If you are working on an individual connectors run: `./gradlew :airbyte-integrations::format`. ### Connector Infrastructure + Finally, if you are working in any module in `:airbyte-integrations:bases` or `:airbyte-cdk:python`, run `SUB_BUILD=CONNECTORS_BASE ./gradlew format`. Note: If you are contributing a Python file without imports or function definitions, place the following comment at the top of your file: @@ -124,14 +129,14 @@ Note: If you are contributing a Python file without imports or function definiti ### Develop on `airbyte-webapp` -* Spin up Airbyte locally so the UI can make requests against the local API. -* Stop the `webapp`. +- Spin up Airbyte locally so the UI can make requests against the local API. +- Stop the `webapp`. ```bash docker-compose stop webapp ``` -* Start up the react app. +- Start up the react app. ```bash cd airbyte-webapp @@ -139,13 +144,13 @@ npm install npm start ``` -* Happy Hacking! +- Happy Hacking! ### Connector Specification Caching -The Configuration API caches connector specifications. This is done to avoid needing to run Docker everytime one is needed in the UI. Without this caching, the UI crawls. If you update the specification of a connector and need to clear this cache so the API / UI picks up the change, you have two options: +The Configuration API caches connector specifications. This is done to avoid needing to run Docker everytime one is needed in the UI. Without this caching, the UI crawls. If you update the specification of a connector and need to clear this cache so the API / UI picks up the change, you have two options: -1. Go to the Admin page in the UI and update the version of the connector. Updating to any version, including the one you're already on, will trigger clearing the cache. +1. Go to the Admin page in the UI and update the version of the connector. Updating to any version, including the one you're already on, will trigger clearing the cache. 2. Restart the server by running the following commands: ```bash @@ -157,26 +162,49 @@ VERSION=dev docker-compose up Sometimes you'll want to reset the data in your local environment. One common case for this is if you are updating an connector's entry in the database \(`airbyte-config/init/src/main/resources/config`\), often the easiest thing to do, is wipe the local database and start it from scratch. To reset your data back to a clean install of Airbyte, follow these steps: -* Delete the datastore volumes in docker +- Delete the datastore volumes in docker ```bash VERSION=dev docker-compose down -v ``` -* Remove the data on disk +- Remove the data on disk ```bash rm -rf /tmp/dev_root rm -rf /tmp/airbyte_local ``` -* Rebuild the project +- Rebuild the project ```bash SUB_BUILD=PLATFORM ./gradlew clean build VERSION=dev docker-compose up -V ``` +While not as common as the above steps, you may also get into a position where want to erase all of the data on your local docker server. This is useful if you've been modifying image tags while developing. + +```bash +docker system prune -a +docker volume prune +``` + +If you are working on python connectors, you may also need to reset the `virtualenv` and re-install the connector's dependencies. + +```bash +# Assuming you have a virtualenv loaded into your shell +deactivate + +# From the connector's directory +# remove the venv directory entirely +rm -rf .venv + +# make and activate a new venv +python3 -m venv .venv +source .venv/bin/activate +pip install -e ".[dev]" +``` + ## Troubleshooting ### `gradlew Could not target platform: 'Java SE 14' using tool chain: 'JDK 8 (1.8)'.` @@ -188,4 +216,3 @@ For example: ```text env JAVA_HOME=/usr/lib/jvm/java-14-openjdk ./gradlew :airbyte-integrations:connectors:your-connector-dir:build ``` -