Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correctly handle an empty dbt_project.yml in dbt debug #2120

Merged
merged 1 commit into from
Feb 18, 2020
Merged

Correctly handle an empty dbt_project.yml in dbt debug #2120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions core/dbt/config/project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,12 @@ def _raw_project_from(project_root: str) -> Dict[str, Any]:
)

project_dict = _load_yaml(project_yaml_filepath)
Copy link
Contributor Author

@nchammas nchammas Feb 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we should just validate the complete project schema here, perhaps using something like StrictYAML.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty cool! I'll have to look into it. Sadly it looks like they don't support anchors, but maybe we'd live without that. We do validate the parsed project using hologram, but I would love a smaller, more close-to-the-input schema definition for your yaml files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are anchors?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

foo: &my_anchor
  a: 1
  b: [2, 3]
  c: false
bar: *my_anchor

Is equivalent to:

foo:
  a: 1
  b: [2, 3]
  c: false
bar:
  a: 1
  b: [2, 3]
  c: false

There's also a field-override syntax, which is very convenient.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, StrictYAML removes a lot of YAML features like that.

I'm not familiar with Hologram, but I have heard of Cerberus as an option for flexible data validation.


if not isinstance(project_dict, dict):
raise DbtProjectError(
'dbt_project.yml does not parse to a dictionary'
)

return project_dict


Expand Down
17 changes: 9 additions & 8 deletions core/dbt/task/debug.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,10 +182,10 @@ def _choose_profile_names(self) -> Optional[List[str]]:
project_profile: Optional[str] = None
if os.path.exists(self.project_path):
try:
project_profile = load_yaml_text(
dbt.clients.system.load_file_contents(self.project_path)
).get('profile')
except dbt.exceptions.Exception:
project_profile = Project.partial_load(
nchammas marked this conversation as resolved.
Show resolved Hide resolved
os.path.dirname(self.project_path)
).profile_name
except dbt.exceptions.DbtProjectError:
pass

args_profile: Optional[str] = getattr(self.args, 'profile', None)
Expand All @@ -196,19 +196,20 @@ def _choose_profile_names(self) -> Optional[List[str]]:
pass
# try to guess

profiles = []
if self.raw_profile_data:
profiles = [k for k in self.raw_profile_data if k != 'config']
if len(profiles) == 0:
if project_profile is None:
self.messages.append('Could not load dbt_project.yml')
elif len(profiles) == 0:
self.messages.append('The profiles.yml has no profiles')
elif len(profiles) == 1:
self.messages.append(ONLY_PROFILE_MESSAGE.format(profiles[0]))
return profiles
else:
self.messages.append(MULTIPLE_PROFILE_MESSAGE.format(
'\n'.join(' - {}'.format(o) for o in profiles)
))
return profiles
return None
return profiles

def _choose_target_name(self, profile_name: str):
has_raw_profile = (
Expand Down
12 changes: 12 additions & 0 deletions test/integration/049_dbt_debug_test/test_debug.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,18 @@ def models(self):
def capsys(self, capsys):
self.capsys = capsys

@use_profile('postgres')
def test_postgres_empty_project(self):
with open('dbt_project.yml', 'w') as f:
pass
self.run_dbt(['debug', '--profile', 'test'])
splitout = self.capsys.readouterr().out.split('\n')
for line in splitout:
if line.strip().startswith('dbt_project.yml file'):
self.assertIn('ERROR invalid', line)
elif line.strip().startswith('profiles.yml file'):
self.assertNotIn('ERROR invalid', line)

@use_profile('postgres')
def test_postgres_badproject(self):
# load a special project that is an error
Expand Down