Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

289 saas connector headers #327

Merged
merged 20 commits into from
Mar 28, 2022
Merged

289 saas connector headers #327

merged 20 commits into from
Mar 28, 2022

Conversation

galvana
Copy link
Collaborator

@galvana galvana commented Mar 24, 2022

Purpose

Adding the ability to specify headers in requests of a SaaS config.

Changes

  • Added headers and query_params to request in the saas_config.
  • Removed the type param from request_params, each value in a request param is free to be used anywhere in the placeholders of path, headers, query_params, and body.
  • Cleaned up param value mapping in saas_query_config by refactoring the logic to be able to use a shared map_param_values function.
  • Updated documentation and Postman collection.

Checklist

  • Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram)
  • Good unit test/integration test coverage
  • This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
  • The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket

Fixes #289

Base automatically changed from 276-saas-connector-add-the-ability-for-request_params-to-reference-connector_param-values to main March 25, 2022 17:27
@seanpreston seanpreston added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Mar 25, 2022
Copy link
Contributor

@pattisdr pattisdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice @galvana, besides adding header support, this saas config iteration adds several other improvements. The query_params spec is especially more readable than before.

data/saas/config/mailchimp_config.yml Show resolved Hide resolved
docs/fidesops/docs/guides/saas_config.md Outdated Show resolved Hide resolved
src/fidesops/schemas/saas/saas_config.py Outdated Show resolved Hide resolved
src/fidesops/service/connectors/saas_query_config.py Outdated Show resolved Hide resolved
Comment on lines 72 to 78
if value:
placeholders = re.findall("<(.+?)>", value)
for placeholder in placeholders:
value = value.replace(
f"<{placeholder}>", str(param_values[placeholder])
)
return value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be validation when creating/updating the saas config to make sure placeholders have corresponding values in request_params? I'd like to be as helpful as possible when the saas config is being created.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll and a logger statement for now and create a separate ticket for validating the different places the placeholders can be used.

src/fidesops/service/connectors/saas_query_config.py Outdated Show resolved Hide resolved
docs/fidesops/docs/guides/saas_config.md Outdated Show resolved Hide resolved
docs/fidesops/docs/guides/saas_config.md Outdated Show resolved Hide resolved
- `pagination` An optional strategy used to get the next set of results from APIs with resources spanning multiple pages. Details can be found under [SaaS Pagination](saas_pagination.md).

## Request params in more detail
The `request_params` list is what provides the values to our various placeholders in the path, headers, query params and body. Values can be `identities` such as email or phone number, `references` to fields in other collections, or `connector_params` which are defined as part of configuring a SaaS connector. Whenever a placeholder is encountered, the placeholder name is looked up in the list of `request_params` and corresponding value is used instead. Here is an example of placeholders being used in various locations:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that query_params are separated, the name of request_params isn't intuitive without these docs, it's really where we're specifying request values. Just a personal opinion.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eastandwestwind brought up the same point this morning, I'm open to suggestions for a better name, some ideas:

  • param_value_map
  • param_values
  • request_param_values

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah good suggestions, I like param_values, shorter than request_param_values

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the efforts to change this

@galvana galvana closed this Mar 25, 2022
@galvana galvana reopened this Mar 25, 2022
@galvana galvana added the run unsafe ci checks Triggers running of unsafe CI checks label Mar 26, 2022
prepared_request: SaaSRequestParams = SaaSRequestParams(
method=HTTPMethod.GET, path=test_request_path, params={}, body=None
method=test_request.method, path=test_request.path
)
self.client().send(prepared_request)
return ConnectionTestStatus.succeeded
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to these changes, in testing this, if the user hasn't set their secrets, self.secrets would be None, and therefore we couldn't do f"{self.client_config.protocol}://{self.secrets[host_key]}" and would get a weird typeerror, i'd expect them to be given more detailed information that their secrets weren't configured.

Copy link
Contributor

@pattisdr pattisdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, thanks for being thorough here, I don't want to hold up the merge because of the unrelated piece ^

@pattisdr pattisdr merged commit 11eb3d8 into main Mar 28, 2022
@pattisdr pattisdr deleted the 289-saas-connector-headers branch March 28, 2022 18:51
sanders41 pushed a commit that referenced this pull request Sep 22, 2022
Adding the ability to specify headers in requests of a SaaS config, plus changes to specifying query_params and param_values.

Co-authored-by: Adrian Galvan <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
run unsafe ci checks Triggers running of unsafe CI checks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[SaaS Connector] Add the ability for request headers to reference connector_param values
3 participants