Feature/v0.2.0 (#152)

* Fixes legacy logger

* Initial pass at removing legacy logging

* Update log file output to use json format

* Updates server logging to send events

* Removes outdated comment

* Support partial parsing (#151)

* Support partial parsing

* Updates todo

* Core integration playground (#129)

* possible way of intergrating all of the dbt commands

* somewhat working version of a generalized framework

* working version of run, a lot of refactor and better core interface needed

* using some new interface

* remove unused function

* using state for run task

* some clean up

* Resolves merge conflicts (#145)

* Core integration updates (#148)

* Updates state_id usage

* Moves task logic to StateController

* removes hardcoded command

* Initiates logmanager in async function

* Removes old async logic and reinstates python logger for dbt-server

Co-authored-by: Rachel <[email protected]>
Co-authored-by: Rachel Daniel <[email protected]>

* Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency (#149)

* Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt to resolve error handling issue with underlying FastAPI dependency

* Add changelog entry

* Accept project path in addition to state_id (#154)

* possible way of intergrating all of the dbt commands

* somewhat working version of a generalized framework

* working version of run, a lot of refactor and better core interface needed

* using some new interface

* remove unused function

* using state for run task

* some clean up

* Core integration updates (#148)

* Updates state_id usage

* Moves task logic to StateController

* removes hardcoded command

* Removes old async logic and reinstates python logger for dbt-server

* Beginning logic to accept a project path

* Adds project_path storing and cacheing

* Removes prints and fixes caching issue

* removes unused task functions

* adds changie entry

* removes dup code from rebase error

* removes dup code from rebase error

* removes dup code from rebase error

* Adds tests for dbt_entry and preliminary state tests

* Removes unused file

* Copies minimal project to tempdir to avoid writing files

Co-authored-by: Chenyu Li <[email protected]>
Co-authored-by: Chenyu Li <[email protected]>

* Fixes broken tests (#156)

* Fix profile for async endpoint (#157)

* Updates async endpoint to use set_profile_name function

* Adds checkfirst flag to avoid table exists error

* Fixes profile name and potential fix for sqlalchemy error

* Adds profile back to command args

* Fixes whitespace

* Adds status endpoint

* Fixes shutdown and removes middleware

* Fixes response_model as called out by community member on main branch

* Sync dbt endpoint (#161)

* Adds sync endpoint and fixes linting

* Adds test for sync dbt entry endpoint

* Fixes formatting

* Adds changie entry

* Add task status callback (#164)

* Add the requests library to the requirements

* Replace each specific task update method with a generic method so that it can be called cleanly upstream

* Update this class to use camel casing

* Add new update task status method that sets the task status in the local DB as well as calling the callback if there is one

* Accept a callback url and pass it to the async command method

* Call the new update task status method where the crud methods were previously called

* Move requests from the dev requirements to requirements

* Return the state ID in addition to the other task fields in the async response

* Remove commented out code

* Specify to retry post requests since it isn't enabled by default

* Update dbt_server/views.py

Co-authored-by: Rachel <[email protected]>

* Rename DBTCommandArgs to DbtCommandArgs

* Add a change log entry

---------

Co-authored-by: Rachel <[email protected]>

* make server working with dbt-core main (#167)

* Control server write locations (#166)

* Updates db path to working dir instead of app root

* Solidifies locations that the dbt-server writes to

* Changes back to app root after dbt command run

* Fixes comment

* Accept a task ID as part of the request and, if present, use it when creating the async task. If not present, create a task ID and use it (#168)

* Adds error handling for json conversion

* Include all exceptions in error handling. (#169)

* Fix bug of not chdir back (#175)

* Fix tests. (#173)

* Fix tests.

* Fix wrong package

* Remove adaptor requirements and skip tests without dependency.

* Fix wrong package name

* Update actions (#176)

* Resolves merge conflicts

* Cherry-pick gone awry

* spaces

* Allows images to build on PR

* Removes conditional on test, tailors to branch (#181)

* Removes conditional on test, tailors to branch

* Adds changie entry

* Comments out unused matrix

* fixes formatting

* fixes formatting

* Testing installations in one line

* Undoes consolidaiton to single line

* Adds quotes to head installs

* Adds 1.5.0b1 to github action

* Adds prerelease flag

* RUNTIME-733 Add smoke tests (#170)

* Add smoke test and check in test dbt project jaffle shop.

* nits

* nits

* Add changie

* Update smoke test.

* Start dbt-server inside smoke test.

* Fix format.

* Fixes linting

* Removes conditionals in github actions

* Removes additional branch-- to be managed separately

---------

Co-authored-by: Chenyu Li <[email protected]>
Co-authored-by: Jennifer Miller <[email protected]>
Co-authored-by: Chenyu Li <[email protected]>
Co-authored-by: jp-dbt <[email protected]>
Co-authored-by: dichenqiandbt <[email protected]>

.changes/unreleased/Features-20230117-083848.yaml

@@ -0,0 +1,10 @@
+kind: Features
+body: With this PR, dbt-server users/clients can pass a project path directly through
+  /parse rather than sending a dictionary of file contents to /push. If a user does
+  this, they can then call other endpoints such as /async/dbt and /compile without
+  a state_id, and the server will default to using that project path.
+time: 2023-01-17T08:38:48.019196-06:00
+custom:
+  Author: racheldaniel
+  Issue: "155"
+  PR: "154"

.changes/unreleased/Features-20230130-160207.yaml

@@ -0,0 +1,9 @@
+kind: Features
+body: This PR adds a new synchronous endpoint, which will block and return command
+  results rather than return a task_id. These tasks are not added to the db, and do
+  not output logs
+time: 2023-01-30T16:02:07.497161-06:00
+custom:
+  Author: racheldaniel
+  Issue: "162"
+  PR: "161"

.changes/unreleased/Features-20230206-120426.yaml

@@ -0,0 +1,7 @@
+kind: Features
+body: Add new task status callback functionality to the async dbt endpoint
+time: 2023-02-06T12:04:26.954999-05:00
+custom:
+  Author: jp-dbt
+  Issue: "165"
+  PR: "164"

.changes/unreleased/Features-20230221-183401.yaml

@@ -0,0 +1,7 @@
+kind: Features
+body: Add smoke test.
+time: 2023-02-21T18:34:01.400318-08:00
+custom:
+  Author: dichenqiandbt
+  Issue: "727"
+  PR: "170"

.changes/unreleased/Under the Hood-20221221-112702.yaml

@@ -0,0 +1,8 @@
+kind: Under the Hood
+body: Upgrade FastAPI version in requirements.txt and add httpx to dev-requirements.txt
+  to resolve error handling issue with underlying FastAPI dependency
+time: 2022-12-21T11:27:02.990803-08:00
+custom:
+  Author: jenniferjsmmiller
+  Issue: "599"
+  PR: "149"

.changes/unreleased/Under the Hood-20230222-124703.yaml

@@ -0,0 +1,8 @@
+kind: Under the Hood
+body: Update github actions to test by dbt branch and deploy different images based
+  on branch (0.1.latest)
+time: 2023-02-22T12:47:03.781432-06:00
+custom:
+  Author: racheldaniel
+  Issue: "172"
+  PR: "171"

.changes/unreleased/Under the Hood-20230227-113905.yaml

@@ -0,0 +1,7 @@
+kind: Under the Hood
+body: Only run tests for github actions on appropriate branch
+time: 2023-02-27T11:39:05.03617-06:00
+custom:
+  Author: racheldaniel
+  Issue: "182"
+  PR: "181"

.github/workflows/main.yml

@@ -35,6 +35,13 @@ jobs:
     name: test code
     runs-on: ubuntu-latest
     timeout-minutes: 3
+    strategy:
+      matrix:
+        dbt-core:
+          - version: "1.5.0-pre"
+            package: "dbt-core~=1.5.0b1"
+            prerelease: true
+
     steps:
       - name: checkout repo
         uses: actions/checkout@v3
@@ -43,43 +50,29 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: "3.8"
-
-      - name: run tests
+      - name: run tests - releases
         run: |
           pip install -r requirements.txt -r dev-requirements.txt
-          pip install dbt-core dbt-postgres
+          pip install ${{ (matrix.dbt-core.prerelease && '--pre') || '' }} ${{ matrix.dbt-core.package }} dbt-postgres dbt-snowflake
+          pytest
+    
+      - name: run tests - head
+        run: |
+          pip install -r requirements.txt -r dev-requirements.txt
+          pip install "https://github.com/dbt-labs/dbt-core/archive/HEAD.tar.gz#egg=dbt-core&subdirectory=core"
+          pip install "https://github.com/dbt-labs/dbt-core/archive/HEAD.tar.gz#egg=dbt-postgres&subdirectory=plugins/postgres"
+          pip install "https://github.com/dbt-labs/dbt-snowflake/archive/HEAD.tar.gz#egg=dbt-snowflake"
           pytest
-
   build-push:
     name: build and push dbt server images
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         dbt-core:
-          - version: "1.0.0"
-            package: "dbt-core~=1.0.0"
-          - version: "1.0.1"
-            package: "dbt-core~=1.0.1"
-          - version: "1.1.0-pre"
-            package: "dbt-core~=1.1.0b1"
-          - version: "1.1.0-latest"
-            package: "dbt-core~=1.1.1"
-          - version: "1.2.0-pre"
-            package: "dbt-core~=1.2.0b1"
-            prerelease: true
-          - version: "1.2.0-latest"
-            package: "dbt-core~=1.2.0"
-          - version: "1.3.0-pre"
-            package: "dbt-core~=1.3.0b1"
-            prerelease: true
-          - version: "1.3.0-latest"
-            package: "dbt-core~=1.3.0"
-          - version: "1.4.0-pre"
-            package: "dbt-core~=1.4.0b1"
+          - version: "1.5.0-pre"
+            package: "dbt-core~=1.5.0b1"
             prerelease: true
-          - version: "1.4.0-latest"
-            package: "dbt-core~=1.4.0"
         dbt-database-adapter:
           - name: snowflake
             package: dbt-snowflake

.gitignore

@@ -16,3 +16,5 @@ sql_app.db
 /env
 /venv
 dbt-core-server-exploration/
+dbt.log
+.DS_Store

dbt_server/crud.py

@@ -24,26 +24,15 @@ def create_task(db: Session, task: schemas.Task):
     return db_task
 
 
-def set_task_running(db: Session, task: schemas.Task):
+def set_task_state(
+    db: Session, task: schemas.Task, state: models.TaskState, error: str
+):
     db_task = get_task(db, task.task_id)
-    db_task.state = models.TaskState.RUNNING
-    db.commit()
-    db.refresh(db_task)
-    return db_task
-
-
-def set_task_done(db: Session, task: schemas.Task):
-    db_task = get_task(db, task.task_id)
-    db_task.state = models.TaskState.FINISHED
-    db.commit()
-    db.refresh(db_task)
-    return db_task
+    db_task.state = state
 
+    if error:
+        db_task.error = error
 
-def set_task_errored(db: Session, task: schemas.Task, error: str):
-    db_task = get_task(db, task.task_id)
-    db_task.state = models.TaskState.ERROR
-    db_task.error = error
     db.commit()
     db.refresh(db_task)
     return db_task

dbt_server/database.py

@@ -1,8 +1,10 @@
 from sqlalchemy import create_engine
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import sessionmaker
+from dbt_server.services.filesystem_service import get_db_path
 
-SQLALCHEMY_DATABASE_URL = "sqlite:///./sql_app.db"
+
+SQLALCHEMY_DATABASE_URL = f"sqlite:///{get_db_path()}"
 
 engine = create_engine(
     SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}

dbt_server/helpers.py

@@ -1,10 +1,5 @@
 import os
 from dbt_server.exceptions import InternalException
-from pydantic import BaseModel
-
-
-class Args(BaseModel):
-    profile: str = None
 
 
 def extract_compiled_code_from_node(result_node_dict):
@@ -23,13 +18,12 @@ def extract_compiled_code_from_node(result_node_dict):
     return compiled_code
 
 
-def set_profile_name(args=None):
-    # If no profile name is passed in args, we will attempt to set it from env vars
-    # If no profile is set, dbt will default to reading from dbt_project.yml
+def get_profile_name(args=None):
+    # If no profile name is passed in args, we will attempt to get it from env vars
+    # If profile is None, dbt will default to reading from dbt_project.yml
     if args and hasattr(args, "profile") and args.profile:
-        return args
-    if os.getenv("DBT_PROFILE_NAME"):
-        if args is None:
-            args = Args()
-        args.profile = os.getenv("DBT_PROFILE_NAME")
-    return args
+        return args.profile
+    env_profile_name = os.getenv("DBT_PROFILE_NAME")
+    if env_profile_name:
+        return env_profile_name
+    return None

dbt_server/logging.py

@@ -5,20 +5,24 @@
 from datetime import datetime
 from typing import Optional
 
-try:
-    from dbt.events.functions import STDOUT_LOG, FILE_LOG
-except (ModuleNotFoundError, ImportError):
-    STDOUT_LOG = None
-    FILE_LOG = None
-
+from dbt.events.eventmgr import EventLevel
+from dbt.events.base_types import BaseEvent
 from pythonjsonlogger import jsonlogger
-from dbt_server.models import TaskState
 
+from dbt_server.models import TaskState
 
 ACCOUNT_ID = os.environ.get("ACCOUNT_ID")
 ENVIRONMENT_ID = os.environ.get("ENVIRONMENT_ID")
 WORKSPACE_ID = os.environ.get("WORKSPACE_ID")
 
+dbt_event_to_python_root_log = {
+    EventLevel.DEBUG: logging.root.debug,
+    EventLevel.TEST: logging.root.debug,
+    EventLevel.INFO: logging.root.info,
+    EventLevel.WARN: logging.root.warn,
+    EventLevel.ERROR: logging.root.error,
+}
+
 
 class CustomJsonFormatter(jsonlogger.JsonFormatter):
     def add_fields(self, log_record, record, message_dict):
@@ -37,9 +41,9 @@ def add_fields(self, log_record, record, message_dict):
             log_record["workspaceID"] = WORKSPACE_ID
 
 
-# setup json logging
+# setup json logging for stdout and datadog
 logger = logging.getLogger()
-logger.setLevel(logging.INFO)
+logger.setLevel(logging.DEBUG)
 stdout = logging.StreamHandler()
 if os.environ.get("APPLICATION_ENVIRONMENT") in ("dev", None):
     formatter = logging.Formatter(
@@ -55,16 +59,11 @@ def add_fields(self, log_record, record, message_dict):
     )
 stdout.setFormatter(formatter)
 logger.addHandler(stdout)
-dbt_server_logger = logging.getLogger("dbt-server")
-dbt_server_logger.setLevel(logging.DEBUG)
-GLOBAL_LOGGER = dbt_server_logger
 
-# remove handlers from these loggers, so
-# that they propagate up to the root logger
-# for json formatting
-if STDOUT_LOG and FILE_LOG:
-    STDOUT_LOG.handlers = []
-    FILE_LOG.handlers = []
+# Use standard python logger for all dbt-server logs-- these will be sent to
+# stdout but will not be written to task log files
+DBT_SERVER_LOGGER = logging.getLogger("dbt-server")
+DBT_SERVER_LOGGER.setLevel(logging.DEBUG)
 
 # make sure uvicorn is deferring to the root
 # logger to format logs
@@ -91,6 +90,21 @@ def configure_uvicorn_access_log():
     ual.handlers = []
 
 
+# Push event messages to root python logger for formatting
+def log_event_to_console(event: BaseEvent):
+    logging_method = dbt_event_to_python_root_log[event.log_level()]
+    if logging_method == logging.root.debug:
+        # Only log debug level for dbt-server logs
+        return
+    logging_method(event.info.msg)
+
+
+# TODO: Core is still working on a way to add a callback to the eventlogger using the
+# newer format. We will still need to do this for events emitted by core
+# EVENT_MANAGER.callbacks.append(log_event_to_console)
+
+
+# TODO: This should be some type of event. We may also choose to send events for all task state updates.
 @dataclass
 class ServerLog:
     state: TaskState

dbt_server/models.py

@@ -1,6 +1,5 @@
 from sqlalchemy import Column, String
 from enum import Enum
-
 from .database import Base
 
 

dbt_server/server.py

@@ -8,12 +8,18 @@
 from dbt_server.database import engine
 from dbt_server.services import dbt_service, filesystem_service
 from dbt_server.views import app
-from dbt_server.logging import GLOBAL_LOGGER as logger, configure_uvicorn_access_log
+from dbt_server.logging import DBT_SERVER_LOGGER as logger, configure_uvicorn_access_log
 from dbt_server.state import LAST_PARSED
 from dbt_server.exceptions import StateNotFoundException
+from sqlalchemy.exc import OperationalError
 
+# The default checkfirst=True should handle this, however we still
+# see a table exists error from time to time
+try:
+    models.Base.metadata.create_all(bind=engine, checkfirst=True)
+except OperationalError as err:
+    logger.debug(f"Handled error when creating database: {str(err)}")
 
-models.Base.metadata.create_all(bind=engine)
 dbt_service.disable_tracking()
 
 
@@ -25,23 +31,25 @@ class ConfigArgs(BaseModel):
 def startup_cache_initialize():
     """
     Initialize the manifest cache at startup. The cache will only be populated if there is
-    a latest-state-id.txt file pointing to a state folder with a pre-compiled manifest.
-    If any step fails (the latest-state-id.txt file is missing, there's no compiled manifest,
-    or it can't be deserialized) then continue without caching.
+    a latest-state-id.txt file or latest-project-path.txt file pointing to a state or project folder
+    with a pre-compiled manifest. If any step fails (the latest-state-id.txt file is missing,
+    there's no compiled manifest, or it can't be deserialized) then continue without caching.
     """
 
     # If an exception is raised in this method, the dbt-server will fail to start up.
     # Be careful here :)
-
     latest_state_id = filesystem_service.get_latest_state_id(None)
-    if latest_state_id is None:
-        logger.info("[STARTUP] No latest state found - not loading manifest into cache")
+    latest_project_path = filesystem_service.get_latest_project_path()
+    root_path = filesystem_service.get_root_path(latest_state_id, latest_project_path)
+
+    if root_path is None:
+        logger.info(
+            "[STARTUP] No latest state or project found - not loading manifest into cache"
+        )
         return
 
-    manifest_path = filesystem_service.get_path(latest_state_id, "manifest.msgpack")
-    logger.info(
-        f"[STARTUP] Loading manifest from file system (state_id={latest_state_id})"
-    )
+    manifest_path = filesystem_service.get_path(root_path, "manifest.msgpack")
+    logger.info(f"[STARTUP] Loading manifest from file system (path={root_path})")
 
     try:
         manifest = dbt_service.deserialize_manifest(manifest_path)
@@ -50,7 +58,7 @@ def startup_cache_initialize():
         return
     except (StateNotFoundException):
         logger.error(
-            f"[STARTUP] Specified latest state not found - not loading manifest (state_id={latest_state_id})"
+            f"[STARTUP] Specified root path not found - not loading manifest (path={root_path})"
         )
         return
 
@@ -59,15 +67,14 @@ def startup_cache_initialize():
         target_name = None
     config_args = ConfigArgs(target=target_name)
 
-    source_path = filesystem_service.get_root_path(latest_state_id)
     manifest_size = filesystem_service.get_size(manifest_path)
-    config = dbt_service.create_dbt_config(source_path, config_args)
+    config = dbt_service.create_dbt_config(root_path, config_args)
 
     LAST_PARSED.set_last_parsed_manifest(
-        latest_state_id, manifest, manifest_size, config
+        latest_state_id, latest_project_path, root_path, manifest, manifest_size, config
     )
 
-    logger.info(f"[STARTUP] Cached manifest in memory (state_id={latest_state_id})")
+    logger.info(f"[STARTUP] Cached manifest in memory (path={root_path})")
 
 
 @tracer.wrap