Merge pull request #1465 from fishtown-analytics/dev/merge-in-0.13.1

merge 0.13.1

CHANGELOG.md

@@ -1,3 +1,28 @@
+## dbt 0.13.1 (May 13, 2019)
+
+### Overview
+This is a bugfix release.
+
+### Bugfixes
+- Add "MaterializedView" relation type to the Snowflake adapter ([#1430](https://github.com/fishtown-analytics/dbt/issues/1430), [#1432](https://github.com/fishtown-analytics/dbt/pull/1432)) ([@adriank-convoy](https://github.com/adriank-convoy))
+- Quote databases properly ([#1396](https://github.com/fishtown-analytics/dbt/issues/1396), [#1402](https://github.com/fishtown-analytics/dbt/pull/1402))
+- Use "ilike" instead of "=" for database equality when listing schemas ([#1411](https://github.com/fishtown-analytics/dbt/issues/1411), [#1412](https://github.com/fishtown-analytics/dbt/pull/1412))
+- Pass the model name along in get_relations ([#1384](https://github.com/fishtown-analytics/dbt/issues/1384), [#1388](https://github.com/fishtown-analytics/dbt/pull/1388))
+- Add logging to dbt clean ([#1261](https://github.com/fishtown-analytics/dbt/issues/1261), [#1383](https://github.com/fishtown-analytics/dbt/pull/1383), [#1391](https://github.com/fishtown-analytics/dbt/pull/1391)) ([@emilieschario](https://github.com/emilieschario))
+
+### dbt Docs
+- Search by columns ([dbt-docs#23](https://github.com/fishtown-analytics/dbt-docs/pull/23)) ([rmgpinto](https://github.com/rmgpinto))
+- Support @ selector ([dbt-docs#27](https://github.com/fishtown-analytics/dbt-docs/pull/27))
+- Fix number formatting on Snowflake and BQ in table stats ([dbt-docs#28](https://github.com/fishtown-analytics/dbt-docs/pull/28))
+
+### Contributors:
+Thanks for your contributions to dbt!
+
+- [@emilieschario](https://github.com/emilieschario)
+- [@adriank-convoy](https://github.com/adriank-convoy)
+- [@rmgpinto](https://github.com/rmgpinto)
+
+
 ## dbt 0.13.0 - Stephen Girard (March 21, 2019)
 
 ### Overview

CONTRIBUTING.md

@@ -12,10 +12,29 @@ Before you can develop dbt effectively, you should set up the following:
 
 We strongly recommend setting up [pyenv](https://github.com/pyenv/pyenv) and its [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) plugin. This setup will make it much easier for you to manage multiple Python projects in the medium to long term.
 
+There is more documentation in each of those links on how to get set up, but the commands you'll need to run will be:
+```
+brew install pyenv
+echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
+exec "$SHELL"
+brew install pyenv-virtualenv
+eval "$(pyenv init -)"
+eval "$(pyenv virtualenv-init -)"
+```
+
 ### python
 
 By default, `pyenv` has only one python version installed and it's the `system` python - the one that comes with your OS. You don't want that. Instead, use `pyenv install 3.6.5` to install a more recent version. dbt supports up to Python 3.6 at the time of writing (and will soon support Python 3.7)
 
+If you get the following error:
+```
+import pip
+zipimport.ZipImportError: can't decompress data; zlib not available
+make: *** [install] Error 1
+```
+
+You can solve it by running `brew install zlib`, then try `pyenv install 3.6.5` again.
+
 To get a full (very long!) list of versions available, you can do `pyenv install -l` and look for the versions defined by numbers alone - the others are variants of Python and outside the scope of this document.
 
 ### docker and docker-compose
@@ -42,6 +61,15 @@ git clone [email protected]:fishtown-analytics/dbt.git
 
 But it really does not matter where you put it as long as you remember it.
 
+
+### Installing postgres locally
+
+For testing, and later in the examples in this document, you may want to have `psql` available so you can poke around in the database and see what happened. We recommend that you use [homebrew](https://brew.sh/) for that on macOS, and your package manager on Linux. You can install any version of the postgres client that you'd like. So on macOS, with homebrew setup:
+
+```
+brew install postgresql
+```
+
 ### Setting up your virtualenv
 
 Set up a fresh virtualenv with pyenv-virtualenv for dbt:
@@ -55,15 +83,6 @@ pyenv activate
 
 This makes a new virtualenv based on python 3.6.5 named `dbt36`, and tells pyenv that when you're in the `dbt` directory it should automatically use that virtualenv.
 
-
-### Installing postgres locally
-
-For testing, and later in the examples in this document, you may want to have `psql` available so you can poke around in the database and see what happened. We recommend that you use [homebrew](https://brew.sh/) for that on macOS, and your package manager on Linux. You can install any version of the postgres client that you'd like. So on macOS, with homebrew setup:
-
-```
-brew install postgresql
-```
-
 ## Testing
 
 Getting the dbt integration tests set up in your local environment will be very helpful as you start to make changes to your local version of dbt. The section that follows outlines some helpful tips for setting up the test environment.
@@ -130,7 +149,7 @@ If you open a PR as a non-contributor, these tests won't run automatically. Some
 
 Sometimes, you're going to have to pretend to be an end user to reproduce bugs and stuff. So that means manually setting up some stuff that the test harness takes care of for you.
 
-### installation
+### Installation
 
 First make sure that you setup your `virtualenv` as described in section _Setting up your environment_.
 
@@ -161,7 +180,7 @@ talk:
     target: default
 ```
 
-There's a sample you can look at in the `dbt` folder (`sample.profiles.yml`) but it's got a lot of extra and as a developer, you really probably only want to test against your local postgres container. The basic idea is that there are multiple 'profiles' (`talk`, in this case) and within those each profile has one or more 'targets' (`default`, in this case), and each profile has a default target. You can specify what profile you want to use with the `--profile` flag, and which target with the `--target` flag. If you want to be really snazzy, dbt project files actually specify their target, and if you match up your dbt project `profile` key with your `profiles.yml` profile names you don't have to use `--profile` (and if you like your profile's default target, no need for `--target` either).
+There's a sample you can look at in the `dbt` [docs](https://docs.getdbt.com/reference#profile) but it's got a lot of extra and as a developer, you really probably only want to test against your local postgres container. The basic idea is that there are multiple 'profiles' (`talk`, in this case) and within those each profile has one or more 'targets' (`default`, in this case), and each profile has a default target. You can specify what profile you want to use with the `--profile` flag, and which target with the `--target` flag. If you want to be really snazzy, dbt project files actually specify their target, and if you match up your dbt project `profile` key with your `profiles.yml` profile names you don't have to use `--profile` (and if you like your profile's default target, no need for `--target` either).
 
 ## Example
 

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src="/etc/dbt-horizontal.png" alt="dbt logo"/>
+  <img src="https://github.com/fishtown-analytics/dbt/blob/master/etc/dbt-horizontal.png?raw=true" alt="dbt logo"/>
 </p>
 <p align="center">
   <a href="https://codeclimate.com/github/fishtown-analytics/dbt">
@@ -20,7 +20,7 @@
 
 dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.
 
-![dbt architecture](/etc/dbt-arch.png?raw=true)
+![dbt architecture](https://github.com/fishtown-analytics/dbt/blob/master/etc/dbt-arch.png?raw=true)
 
 dbt can be used to [aggregate pageviews into sessions](https://github.com/fishtown-analytics/snowplow), calculate [ad spend ROI](https://github.com/fishtown-analytics/facebook-ads), or report on [email campaign performance](https://github.com/fishtown-analytics/mailchimp).
 
@@ -30,7 +30,7 @@ Analysts using dbt can transform their data by simply writing select statements,
 
 These select statements, or "models", form a dbt project. Models frequently build on top of one another – dbt makes it easy to [manage relationships](https://docs.getdbt.com/reference#ref) between models, and [visualize these relationships](https://docs.getdbt.com/docs/documentation), as well as assure the quality of your transformations through [testing](https://docs.getdbt.com/docs/testing).
 
-![dbt dag](/etc/dbt-dag.png?raw=true)
+![dbt dag](https://github.com/fishtown-analytics/dbt/blob/master/etc/dbt-dag.png?raw=true)
 
 ## Getting started
 

core/dbt/adapters/base/impl.py

@@ -608,7 +608,9 @@ def list_relations(self, database, schema):
         information_schema = self.Relation.create(
             database=database,
             schema=schema,
-            model_name='').information_schema()
+            model_name='',
+            quote_policy=self.config.quoting
+        ).information_schema()
 
         # we can't build the relations cache because we don't have a
         # manifest so we can't run any operations.
@@ -628,7 +630,7 @@ def _make_match_kwargs(self, database, schema, identifier):
         if schema is not None and quoting['schema'] is False:
             schema = schema.lower()
 
-        if database is not None and quoting['schema'] is False:
+        if database is not None and quoting['database'] is False:
             database = database.lower()
 
         return filter_null_values({

core/dbt/adapters/base/relation.py

@@ -10,11 +10,13 @@ class BaseRelation(APIObject):
     Table = "table"
     View = "view"
     CTE = "cte"
+    MaterializedView = "materializedview"
 
     RelationTypes = [
         Table,
         View,
-        CTE
+        CTE,
+        MaterializedView
     ]
 
     DEFAULTS = {

core/dbt/adapters/sql/impl.py

@@ -183,6 +183,7 @@ def list_relations_without_caching(self, information_schema, schema):
 
         relations = []
         quote_policy = {
+            'database': True,
             'schema': True,
             'identifier': True
         }
@@ -209,7 +210,8 @@ def list_schemas(self, database):
 
     def check_schema_exists(self, database, schema):
         information_schema = self.Relation.create(
-            database=database, schema=schema
+            database=database, schema=schema,
+            quote_policy=self.config.quoting
         ).information_schema()
 
         kwargs = {'information_schema': information_schema, 'schema': schema}

core/dbt/include/global_project/macros/adapters/common.sql

@@ -192,7 +192,7 @@
   {% call statement('list_schemas', fetch_result=True, auto_begin=False) %}
     select distinct schema_name
     from {{ information_schema_name(database) }}.schemata
-    where catalog_name='{{ database }}'
+    where catalog_name ilike '{{ database }}'
   {% endcall %}
   {{ return(load_result('list_schemas').table) }}
 {% endmacro %}

core/dbt/task/clean.py

@@ -3,6 +3,7 @@
 import shutil
 
 from dbt.task.base import ProjectOnlyTask
+from dbt.logger import GLOBAL_LOGGER as logger
 
 
 class CleanTask(ProjectOnlyTask):
@@ -14,15 +15,24 @@ def __is_project_path(self, path):
         ) == proj_path
 
     def __is_protected_path(self, path):
+        """
+        This function identifies protected paths, so as not to clean them.
+        """
         abs_path = os.path.abspath(path)
         protected_paths = self.config.source_paths + \
             self.config.test_paths + ['.']
-
         protected_abs_paths = [os.path.abspath for p in protected_paths]
         return abs_path in set(protected_abs_paths) or \
             self.__is_project_path(abs_path)
 
     def run(self):
+        """
+        This function takes all the paths in the target file
+        and cleans the project paths that are not protected.
+        """
         for path in self.config.clean_targets:
+            logger.info("Checking {}/*".format(path))
             if not self.__is_protected_path(path):
                 shutil.rmtree(path, True)
+                logger.info(" Cleaned {}/*".format(path))
+        logger.info("Finished cleaning all paths.")

plugins/bigquery/README.md

@@ -0,0 +1,32 @@
+<p align="center">
+  <img src="https://github.com/fishtown-analytics/dbt/blob/master/etc/dbt-horizontal.png?raw=true" alt="dbt logo"/>
+</p>
+
+**[dbt](https://www.getdbt.com/)** (data build tool) enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
+
+dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.
+
+## dbt-bigquery
+
+The `dbt-bigquery` package contains all of the code required to make dbt operate on a BigQuery database. For
+more information on using dbt with BigQuery, consult [the docs](https://docs.getdbt.com/docs/profile-bigquery).
+
+
+## Find out more
+
+-   Check out the [Introduction to dbt](https://dbt.readme.io/docs/introduction).
+-   Read the [dbt Viewpoint](https://dbt.readme.io/docs/viewpoint).
+
+## Join thousands of analysts in the dbt community
+
+-   Join the [chat](http://slack.getdbt.com/) on Slack.
+-   Find community posts on [dbt Discourse](https://discourse.getdbt.com).
+
+## Reporting bugs and contributing code
+
+-   Want to report a bug or request a feature? Let us know on [Slack](http://slack.getdbt.com/), or open [an issue](https://github.com/fishtown-analytics/dbt/issues/new).
+-   Want to help us build dbt? Check out the [Contributing Getting Started Guide](/CONTRIBUTING.md)
+
+## Code of Conduct
+
+Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [PyPA Code of Conduct](https://www.pypa.io/en/latest/code-of-conduct/).

plugins/bigquery/setup.py

@@ -1,17 +1,22 @@
 #!/usr/bin/env python
 from setuptools import find_packages
 from distutils.core import setup
+import os
 
 package_name = "dbt-bigquery"
 package_version = "0.14.0a1"
 description = """The bigquery adapter plugin for dbt (data build tool)"""
 
+this_directory = os.path.abspath(os.path.dirname(__file__))
+with open(os.path.join(this_directory, 'README.md')) as f:
+    long_description = f.read()
 
 setup(
     name=package_name,
     version=package_version,
     description=description,
-    long_description=description,
+    long_description=long_description,
+    long_description_content_type='text/markdown',
     author="Fishtown Analytics",
     author_email="[email protected]",
     url="https://github.com/fishtown-analytics/dbt",

plugins/postgres/README.md

@@ -0,0 +1,32 @@
+<p align="center">
+  <img src="https://github.com/fishtown-analytics/dbt/blob/master/etc/dbt-horizontal.png?raw=true" alt="dbt logo"/>
+</p>
+
+**[dbt](https://www.getdbt.com/)** (data build tool) enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
+
+dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.
+
+## dbt-postgres
+
+The `dbt-postgres` package contains all of the code required to make dbt operate on a Postgres database. For
+more information on using dbt with Postgres, consult [the docs](https://docs.getdbt.com/docs/profile-postgres).
+
+
+## Find out more
+
+-   Check out the [Introduction to dbt](https://dbt.readme.io/docs/introduction).
+-   Read the [dbt Viewpoint](https://dbt.readme.io/docs/viewpoint).
+
+## Join thousands of analysts in the dbt community
+
+-   Join the [chat](http://slack.getdbt.com/) on Slack.
+-   Find community posts on [dbt Discourse](https://discourse.getdbt.com).
+
+## Reporting bugs and contributing code
+
+-   Want to report a bug or request a feature? Let us know on [Slack](http://slack.getdbt.com/), or open [an issue](https://github.com/fishtown-analytics/dbt/issues/new).
+-   Want to help us build dbt? Check out the [Contributing Getting Started Guide](/CONTRIBUTING.md)
+
+## Code of Conduct
+
+Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [PyPA Code of Conduct](https://www.pypa.io/en/latest/code-of-conduct/).

plugins/postgres/dbt/adapters/postgres/impl.py

@@ -20,7 +20,10 @@ def date_function(cls):
 
     @available
     def verify_database(self, database):
-        database = database.strip('"')
+        if database.startswith('"'):
+            database = database.strip('"')
+        else:
+            database = database.lower()
         expected = self.config.credentials.database
         if database != expected:
             raise dbt.exceptions.NotImplementedException(
@@ -31,6 +34,11 @@ def verify_database(self, database):
         return ''
 
     def _link_cached_database_relations(self, schemas):
+        """
+
+        :param Set[str] schemas: The set of schemas that should have links
+            added.
+        """
         database = self.config.credentials.database
         table = self.execute_macro(GET_RELATIONS_MACRO_NAME)
 
@@ -66,8 +74,11 @@ def _get_cache_schemas(self, manifest, exec_only=False):
 
     def _link_cached_relations(self, manifest):
         schemas = set()
-        for db, schema in manifest.get_used_schemas():
-            self.verify_database(db)
+        # only link executable nodes
+        info_schema_name_map = self._get_cache_schemas(manifest,
+                                                       exec_only=True)
+        for db, schema in info_schema_name_map.search():
+            self.verify_database(db.database)
             schemas.add(schema)
 
         self._link_cached_database_relations(schemas)

plugins/postgres/setup.py

@@ -1,16 +1,22 @@
 #!/usr/bin/env python
 from setuptools import find_packages
 from distutils.core import setup
+import os
 
 package_name = "dbt-postgres"
 package_version = "0.14.0a1"
 description = """The postgres adpter plugin for dbt (data build tool)"""
 
+this_directory = os.path.abspath(os.path.dirname(__file__))
+with open(os.path.join(this_directory, 'README.md')) as f:
+    long_description = f.read()
+
 setup(
     name=package_name,
     version=package_version,
     description=description,
     long_description=description,
+    long_description_content_type='text/markdown',
     author="Fishtown Analytics",
     author_email="[email protected]",
     url="https://github.com/fishtown-analytics/dbt",