-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Raise CompilationException on duplicate model #568
Raise CompilationException on duplicate model #568
Conversation
Extend tests
I had to place the check for duplicate in parser, because this is where all duplicates get discarded. If we do this, and move the check to compiler we will be able to distinguish between enabled/disabled models. I followed instructions left in TODO and used CompilationException instead of creating a new one. |
This is great, thanks @mturzanska. I need to dig into this code a little more, but one requirement we have here is that multiple models can have the same name, but only one of them can be |
@drewbanin thanks for having a look! |
@mturzanska thanks for your patience! Yeah, I think that makes sense. Disabled nodes should not be returned from So maybe another approach is to call |
@drewbanin are you referring to this line: https://github.com/fishtown-analytics/dbt/blob/development/dbt/parser.py#L276 ? i think that's a good way to do it. save the node, if it's disabled skip, otherwise add it to |
@drewbanin @cmcarthur Agreed. I'll work on it either tomorrow or on Sunday. |
Extend tests for duplicate model
Rename model-1 to model-disabled
af5e42b
to
a7f7cfe
Compare
@mturzanska this is really close! I think I led you astray above though -- the code in We iterate over all of the projects and load the models here. Maybe it makes sense to override the load_all method for the |
@drewbanin right, I totally missed that. I think one could either
(Just to be sure by |
@mturzanska yes, sorry about that, definitely |
@drewbanin I've added a check for duplicate models across packages and extended tests |
great, thanks @mturzanska. I'll check this out today! |
@mturzanska great job with the tests -- this looks good to me! @cmcarthur do you want to give this a look too? |
'Found models with the same name: \n- %s\n- %s' % ( | ||
model, existing_name)) | ||
names_models[name] = model | ||
return to_return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this looks good. @drewbanin we should expand on this error message later on. specifically it'd be nice to tell the user what to do (disable one of these in your dbt_project.yml, or rename one of the models)
raise dbt.exceptions.CompilationException( | ||
'Found models with the same name:\n- %s\n- %s' % ( | ||
existing_node.get('original_file_path'), | ||
node.get('original_file_path'))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need this check in addition to the one in ModelLoader
? if not let's just rip this out to DRY this up
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we still need this check because to_return
is a dictionary and keys for duplicate models are non-unique. Without this check, only one of the models would be left in the dictionary after this stage and user would not be notified about the existing duplicate.
@mturzanska this looks great. i left a couple comments, but nothing major |
@mturzanska this is ready to be merged, right? |
Yep. I talked with @cmcarthur about his comments at the DataEngConf so all settled. |
@mturzanska great, thanks. Going to do some quick tests on this with a couple of funky existing projects. If all is good, will merge this weekend! |
* only load hooks and archives once (#540) * sets schema for node before parsing raw sql (#541) * Fix/env vars (#543) * fix for bad env_var exception * overwrite target with compiled values * fixes env vars, adds test. Auto-compile profile/target args * improvements for code that runs in hooks (#544) * improvements for code that runs in hooks * fix error message note * typo * Update CHANGELOG.md * bump version (#546) * add scope to service account json creds initializer (#547) * bump 0.9.0a3 --> 0.9.0a4 (#548) * Fix README links (#554) * Update README.md * handle empty profiles.yml file (#555) * return empty string (instead of None) to avoid polluting rendered sql (#566) * tojson was added in jinja 2.9 (#563) * tojson was added in jinja 2.9 * requirements * fix package-defined schema test macros (#562) * fix package-defined schema test macros * create a dummy Relation in parsing * fix for bq quoting (#565) * bump snowflake, remove pyasn1 (#570) * bump snowflake, remove pyasn1 * change requirements.txt * allow macros to return non-text values (#571) * revert jinja version, implement tojson hack (#572) * bump to 090a5 * update changelog * bump (#574) * 090 docs (#575) * 090 docs * Update CHANGELOG.md * Update CHANGELOG.md * Raise CompilationException on duplicate model (#568) * Raise CompilationException on duplicate model Extend tests * Ignore disabled models in parse_sql_nodes Extend tests for duplicate model * Fix preexisting models * Use double quotes consistently Rename model-1 to model-disabled * Fix unit tests * Raise exception on duplicate model across packages Extend tests * Make run_started_at timezone aware (#553) (#556) * Make run_started_at timezone aware Set run_started_at timezone to UTC Enable timezone change in models Extend requirements Extend tests * Address comments from code review Create modules namespace to context Move pytz to modules Add new dependencies to setup.py * Add warning for missing constraints. Fixes #592 (#600) * Add warning for missing constraints. Fixes #592 * fix unit tests * fix schema tests used in, or defined in packages (#599) * fix schema tests used in, or defined in packages * don't hardcode dbt test namespace * fix/actually run tests * rm junk * run hooks in correct order, fixes #590 (#601) * run hooks in correct order, fixes #590 * add tests * fix tests * pep8 * change req for snowflake to fix crypto install issue (#612) From cffi callback <function _verify_callback at 0x06BF2978>: Traceback (most recent call last): File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper _lib.X509_up_ref(x509) AttributeError: module 'lib' has no attribute 'X509_up_ref' From cffi callback <function _verify_callback at 0x06B8CF60>: * Update python version in Makefile from 3.5 to 3.6 (#613) * Fix/snowflake custom schema (#626) * Fixes already opened transaction issue For #602 * Fixes #621 * Create schema in archival flow (#625) * Fix for pre-hooks outside of transactions (#623) * Fix for pre-hooks outside of transactions #576 * improve tests * Fixes already opened transaction issue (#622) For #602 * Accept string for postgres port number (#583) (#624) * Accept string for postgres port number (#583) * s/str/basestring/g * print correct run time (include hooks) (#607) * add support for late binding views (Redshift) (#614) * add support for late binding views (Redshift) * fix bind logic * wip for get_columns_in_table * fix get_columns_in_table * fix for default value in bind config * pep8 * skip tests that depend on nonexistent or disabled models (#617) * skip tests that depend on nonexistent or disabled models * pep8, Fixes #616 * refactor * fix for adapter macro called within packages (#630) * fix for adapter macro called within packages * better error message * Update CHANGELOG.md (#632) * Update CHANGELOG.md * Update CHANGELOG.md * Bump version: 0.9.0 → 0.9.1 * more helpful exception for registry funcs * Rework deps to support local & git * pylint and cleanup * make modules directory first * Refactor registry client for cleanliness and better error handling * init converter script * create modules directory only if non-existent * Only check the hub registry for registry packages * Incorporate changes from Drew's branch Diff of original changes: https://github.com/fishtown-analytics/dbt/pull/591/files * lint * include a portion of the actual name in destination directory * Install dependencies using actual name; better exceptions * Error if two dependencies have same name * Process dependencies one level at a time Included in this change is a refactor of the deps run function for clarity. Also I changed the resolve_version function to update the object in place. I prefer the immutability of this function as it was, but the rest of the code doesn't really operate that way. And I ran into some bugs due to this discrepancy. * update var name * Provide support for repositories in project yml * Download files in a temp directory The downloads directory causes problems with the run command because this directory is not a dbt project. Need to download it elsewhere. * pin some versions * pep8-ify * some PR feedback changes around logging * PR feedback round 2 * Fix for redshift varchar bug (#647) * Fix for redshift varchar bug * pep8 on a sql string, smh * Set global variable overrides on the command line with --vars (#640) * Set global variable overrides on the command line with --vars * pep8 * integration tests for cli vars * Seed rewrite (#618) * loader for seed data files * Functioning rework of seed task * Make CompilerRunner fns private and impl. SeedRunner.compile Trying to distinguish between the public/private interface for this class. And the SeedRunner doesn't need the functionality in the compile function, it just needs a compile function to exist for use in the compilation process. * Test changes and fixes * make the DB setup script usable locally * convert simple copy test to use seeed * Fixes to get Snowflake working * New seed flag and make it non-destructive by default * Convert update SQL script to another seed * cleanup * implement bigquery csv load * context handling of StringIO * Better typing * strip seeder and csvkit dependency * update bigquery to use new data typing and to fix unicode issue * update seed test * fix abstract functions in base adapter * support time type * try pinning crypto, pyopenssl versions * remove unnecessary version pins * insert all at once, rather than one query per row * do not quote field names on creation * bad * quiet down parsedatetime logger * pep8 * UI updates + node conformity for seed nodes * add seed to list of resource types, cleanup * show option for CSVs * typo * pep8 * move agate import to avoid strange warnings * deprecation warning for --drop-existing * quote column names in seed files * revert quoting change (breaks Snowflake). Hush warnings * use hub url * Show installed version, silence semver regex warnings * sort versions to make tests deterministic. Prefer higher versions * pep8, fix comparison functions for py3 * make compare function return value in {-1, 0, 1} * fix for deleting git dirs on windows? * use system client rmdir instead of shutil directly * debug logging to identify appveyor issue * less restrictive error retry * rm debug logging
* semver resolution * cleanup * remove unnecessary comment * add test for multiples on both sides * add resolve_to_specific_version * local registry * hacking out deps * Buck pkg mgmt (#645) * only load hooks and archives once (#540) * sets schema for node before parsing raw sql (#541) * Fix/env vars (#543) * fix for bad env_var exception * overwrite target with compiled values * fixes env vars, adds test. Auto-compile profile/target args * improvements for code that runs in hooks (#544) * improvements for code that runs in hooks * fix error message note * typo * Update CHANGELOG.md * bump version (#546) * add scope to service account json creds initializer (#547) * bump 0.9.0a3 --> 0.9.0a4 (#548) * Fix README links (#554) * Update README.md * handle empty profiles.yml file (#555) * return empty string (instead of None) to avoid polluting rendered sql (#566) * tojson was added in jinja 2.9 (#563) * tojson was added in jinja 2.9 * requirements * fix package-defined schema test macros (#562) * fix package-defined schema test macros * create a dummy Relation in parsing * fix for bq quoting (#565) * bump snowflake, remove pyasn1 (#570) * bump snowflake, remove pyasn1 * change requirements.txt * allow macros to return non-text values (#571) * revert jinja version, implement tojson hack (#572) * bump to 090a5 * update changelog * bump (#574) * 090 docs (#575) * 090 docs * Update CHANGELOG.md * Update CHANGELOG.md * Raise CompilationException on duplicate model (#568) * Raise CompilationException on duplicate model Extend tests * Ignore disabled models in parse_sql_nodes Extend tests for duplicate model * Fix preexisting models * Use double quotes consistently Rename model-1 to model-disabled * Fix unit tests * Raise exception on duplicate model across packages Extend tests * Make run_started_at timezone aware (#553) (#556) * Make run_started_at timezone aware Set run_started_at timezone to UTC Enable timezone change in models Extend requirements Extend tests * Address comments from code review Create modules namespace to context Move pytz to modules Add new dependencies to setup.py * Add warning for missing constraints. Fixes #592 (#600) * Add warning for missing constraints. Fixes #592 * fix unit tests * fix schema tests used in, or defined in packages (#599) * fix schema tests used in, or defined in packages * don't hardcode dbt test namespace * fix/actually run tests * rm junk * run hooks in correct order, fixes #590 (#601) * run hooks in correct order, fixes #590 * add tests * fix tests * pep8 * change req for snowflake to fix crypto install issue (#612) From cffi callback <function _verify_callback at 0x06BF2978>: Traceback (most recent call last): File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper _lib.X509_up_ref(x509) AttributeError: module 'lib' has no attribute 'X509_up_ref' From cffi callback <function _verify_callback at 0x06B8CF60>: * Update python version in Makefile from 3.5 to 3.6 (#613) * Fix/snowflake custom schema (#626) * Fixes already opened transaction issue For #602 * Fixes #621 * Create schema in archival flow (#625) * Fix for pre-hooks outside of transactions (#623) * Fix for pre-hooks outside of transactions #576 * improve tests * Fixes already opened transaction issue (#622) For #602 * Accept string for postgres port number (#583) (#624) * Accept string for postgres port number (#583) * s/str/basestring/g * print correct run time (include hooks) (#607) * add support for late binding views (Redshift) (#614) * add support for late binding views (Redshift) * fix bind logic * wip for get_columns_in_table * fix get_columns_in_table * fix for default value in bind config * pep8 * skip tests that depend on nonexistent or disabled models (#617) * skip tests that depend on nonexistent or disabled models * pep8, Fixes #616 * refactor * fix for adapter macro called within packages (#630) * fix for adapter macro called within packages * better error message * Update CHANGELOG.md (#632) * Update CHANGELOG.md * Update CHANGELOG.md * Bump version: 0.9.0 → 0.9.1 * more helpful exception for registry funcs * Rework deps to support local & git * pylint and cleanup * make modules directory first * Refactor registry client for cleanliness and better error handling * init converter script * create modules directory only if non-existent * Only check the hub registry for registry packages * Incorporate changes from Drew's branch Diff of original changes: https://github.com/fishtown-analytics/dbt/pull/591/files * lint * include a portion of the actual name in destination directory * Install dependencies using actual name; better exceptions * Error if two dependencies have same name * Process dependencies one level at a time Included in this change is a refactor of the deps run function for clarity. Also I changed the resolve_version function to update the object in place. I prefer the immutability of this function as it was, but the rest of the code doesn't really operate that way. And I ran into some bugs due to this discrepancy. * update var name * Provide support for repositories in project yml * Download files in a temp directory The downloads directory causes problems with the run command because this directory is not a dbt project. Need to download it elsewhere. * pin some versions * pep8-ify * some PR feedback changes around logging * PR feedback round 2 * Fix for redshift varchar bug (#647) * Fix for redshift varchar bug * pep8 on a sql string, smh * Set global variable overrides on the command line with --vars (#640) * Set global variable overrides on the command line with --vars * pep8 * integration tests for cli vars * Seed rewrite (#618) * loader for seed data files * Functioning rework of seed task * Make CompilerRunner fns private and impl. SeedRunner.compile Trying to distinguish between the public/private interface for this class. And the SeedRunner doesn't need the functionality in the compile function, it just needs a compile function to exist for use in the compilation process. * Test changes and fixes * make the DB setup script usable locally * convert simple copy test to use seeed * Fixes to get Snowflake working * New seed flag and make it non-destructive by default * Convert update SQL script to another seed * cleanup * implement bigquery csv load * context handling of StringIO * Better typing * strip seeder and csvkit dependency * update bigquery to use new data typing and to fix unicode issue * update seed test * fix abstract functions in base adapter * support time type * try pinning crypto, pyopenssl versions * remove unnecessary version pins * insert all at once, rather than one query per row * do not quote field names on creation * bad * quiet down parsedatetime logger * pep8 * UI updates + node conformity for seed nodes * add seed to list of resource types, cleanup * show option for CSVs * typo * pep8 * move agate import to avoid strange warnings * deprecation warning for --drop-existing * quote column names in seed files * revert quoting change (breaks Snowflake). Hush warnings * use hub url * Show installed version, silence semver regex warnings * sort versions to make tests deterministic. Prefer higher versions * pep8, fix comparison functions for py3 * make compare function return value in {-1, 0, 1} * fix for deleting git dirs on windows? * use system client rmdir instead of shutil directly * debug logging to identify appveyor issue * less restrictive error retry * rm debug logging * s/version/revision for git packages * more s/version/revision, deprecation cleanup * remove unused semver codepath * plus symlinks!!! * get rid of reference to removed function
* semver resolution * cleanup * remove unnecessary comment * add test for multiples on both sides * add resolve_to_specific_version * local registry * hacking out deps * Buck pkg mgmt (#645) * only load hooks and archives once (#540) * sets schema for node before parsing raw sql (#541) * Fix/env vars (#543) * fix for bad env_var exception * overwrite target with compiled values * fixes env vars, adds test. Auto-compile profile/target args * improvements for code that runs in hooks (#544) * improvements for code that runs in hooks * fix error message note * typo * Update CHANGELOG.md * bump version (#546) * add scope to service account json creds initializer (#547) * bump 0.9.0a3 --> 0.9.0a4 (#548) * Fix README links (#554) * Update README.md * handle empty profiles.yml file (#555) * return empty string (instead of None) to avoid polluting rendered sql (#566) * tojson was added in jinja 2.9 (#563) * tojson was added in jinja 2.9 * requirements * fix package-defined schema test macros (#562) * fix package-defined schema test macros * create a dummy Relation in parsing * fix for bq quoting (#565) * bump snowflake, remove pyasn1 (#570) * bump snowflake, remove pyasn1 * change requirements.txt * allow macros to return non-text values (#571) * revert jinja version, implement tojson hack (#572) * bump to 090a5 * update changelog * bump (#574) * 090 docs (#575) * 090 docs * Update CHANGELOG.md * Update CHANGELOG.md * Raise CompilationException on duplicate model (#568) * Raise CompilationException on duplicate model Extend tests * Ignore disabled models in parse_sql_nodes Extend tests for duplicate model * Fix preexisting models * Use double quotes consistently Rename model-1 to model-disabled * Fix unit tests * Raise exception on duplicate model across packages Extend tests * Make run_started_at timezone aware (#553) (#556) * Make run_started_at timezone aware Set run_started_at timezone to UTC Enable timezone change in models Extend requirements Extend tests * Address comments from code review Create modules namespace to context Move pytz to modules Add new dependencies to setup.py * Add warning for missing constraints. Fixes #592 (#600) * Add warning for missing constraints. Fixes #592 * fix unit tests * fix schema tests used in, or defined in packages (#599) * fix schema tests used in, or defined in packages * don't hardcode dbt test namespace * fix/actually run tests * rm junk * run hooks in correct order, fixes #590 (#601) * run hooks in correct order, fixes #590 * add tests * fix tests * pep8 * change req for snowflake to fix crypto install issue (#612) From cffi callback <function _verify_callback at 0x06BF2978>: Traceback (most recent call last): File "c:\projects\dbt\.tox\pywin\lib\site-packages\OpenSSL\SSL.py", line 313, in wrapper _lib.X509_up_ref(x509) AttributeError: module 'lib' has no attribute 'X509_up_ref' From cffi callback <function _verify_callback at 0x06B8CF60>: * Update python version in Makefile from 3.5 to 3.6 (#613) * Fix/snowflake custom schema (#626) * Fixes already opened transaction issue For #602 * Fixes #621 * Create schema in archival flow (#625) * Fix for pre-hooks outside of transactions (#623) * Fix for pre-hooks outside of transactions #576 * improve tests * Fixes already opened transaction issue (#622) For #602 * Accept string for postgres port number (#583) (#624) * Accept string for postgres port number (#583) * s/str/basestring/g * print correct run time (include hooks) (#607) * add support for late binding views (Redshift) (#614) * add support for late binding views (Redshift) * fix bind logic * wip for get_columns_in_table * fix get_columns_in_table * fix for default value in bind config * pep8 * skip tests that depend on nonexistent or disabled models (#617) * skip tests that depend on nonexistent or disabled models * pep8, Fixes #616 * refactor * fix for adapter macro called within packages (#630) * fix for adapter macro called within packages * better error message * Update CHANGELOG.md (#632) * Update CHANGELOG.md * Update CHANGELOG.md * Bump version: 0.9.0 → 0.9.1 * more helpful exception for registry funcs * Rework deps to support local & git * pylint and cleanup * make modules directory first * Refactor registry client for cleanliness and better error handling * init converter script * create modules directory only if non-existent * Only check the hub registry for registry packages * Incorporate changes from Drew's branch Diff of original changes: https://github.com/fishtown-analytics/dbt/pull/591/files * lint * include a portion of the actual name in destination directory * Install dependencies using actual name; better exceptions * Error if two dependencies have same name * Process dependencies one level at a time Included in this change is a refactor of the deps run function for clarity. Also I changed the resolve_version function to update the object in place. I prefer the immutability of this function as it was, but the rest of the code doesn't really operate that way. And I ran into some bugs due to this discrepancy. * update var name * Provide support for repositories in project yml * Download files in a temp directory The downloads directory causes problems with the run command because this directory is not a dbt project. Need to download it elsewhere. * pin some versions * pep8-ify * some PR feedback changes around logging * PR feedback round 2 * Fix for redshift varchar bug (#647) * Fix for redshift varchar bug * pep8 on a sql string, smh * Set global variable overrides on the command line with --vars (#640) * Set global variable overrides on the command line with --vars * pep8 * integration tests for cli vars * Seed rewrite (#618) * loader for seed data files * Functioning rework of seed task * Make CompilerRunner fns private and impl. SeedRunner.compile Trying to distinguish between the public/private interface for this class. And the SeedRunner doesn't need the functionality in the compile function, it just needs a compile function to exist for use in the compilation process. * Test changes and fixes * make the DB setup script usable locally * convert simple copy test to use seeed * Fixes to get Snowflake working * New seed flag and make it non-destructive by default * Convert update SQL script to another seed * cleanup * implement bigquery csv load * context handling of StringIO * Better typing * strip seeder and csvkit dependency * update bigquery to use new data typing and to fix unicode issue * update seed test * fix abstract functions in base adapter * support time type * try pinning crypto, pyopenssl versions * remove unnecessary version pins * insert all at once, rather than one query per row * do not quote field names on creation * bad * quiet down parsedatetime logger * pep8 * UI updates + node conformity for seed nodes * add seed to list of resource types, cleanup * show option for CSVs * typo * pep8 * move agate import to avoid strange warnings * deprecation warning for --drop-existing * quote column names in seed files * revert quoting change (breaks Snowflake). Hush warnings * use hub url * Show installed version, silence semver regex warnings * sort versions to make tests deterministic. Prefer higher versions * pep8, fix comparison functions for py3 * make compare function return value in {-1, 0, 1} * fix for deleting git dirs on windows? * use system client rmdir instead of shutil directly * debug logging to identify appveyor issue * less restrictive error retry * rm debug logging * s/version/revision for git packages * more s/version/revision, deprecation cleanup * remove unused semver codepath * plus symlinks!!! * get rid of reference to removed function automatic commit by git-black, original commits: 5fbcd12
No description provided.