Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-17610: Multiple apps support and data fetching #94

Merged

Conversation

karanpanchal-crest
Copy link
Contributor

@karanpanchal-crest karanpanchal-crest commented Apr 6, 2022

Description of change

  • Allows user to configure multiple app_ids to extract data
  • Fetches data for all app_ids if not provided (the current version only fetches data for default app_id which cannot be changed)

Manual QA steps

  • Verify that if app_ids are not available in config then we get all apps data and if app_ids are given then we get those apps data

Risks

Rollback steps

  • revert this branch

tests/unittests/test_app_ids_configurable.py Outdated Show resolved Hide resolved
tests/unittests/test_app_ids_configurable.py Outdated Show resolved Hide resolved
tests/unittests/test_app_ids_configurable.py Outdated Show resolved Hide resolved
tap_pendo/__init__.py Outdated Show resolved Hide resolved
tap_pendo/__init__.py Outdated Show resolved Hide resolved
tests/tap_tester/test_all_fields.py Outdated Show resolved Hide resolved
Copy link
Contributor

@RushiT0122 RushiT0122 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karanpanchal-crest CCi job has failed, please look into it.

@somethingmorerelevant somethingmorerelevant force-pushed the TDL-17610-multiple-apps-support-and-data-fetching branch from d0df678 to a935175 Compare March 21, 2023 12:46
Copy link
Member

@somethingmorerelevant somethingmorerelevant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed and tested these changes

@RushiT0122 RushiT0122 force-pushed the TDL-17610-multiple-apps-support-and-data-fetching branch from bdab5d0 to ab08c89 Compare April 12, 2023 09:08
tests/tap_tester/test_app_ids.py Outdated Show resolved Hide resolved
expected_streams = {"features", "pages", "events", "guides", "track_types", "track_events"}
self.run_test(expected_streams=expected_streams, app_ids="-323232", is_multi_apps=False)
self.run_test(expected_streams=expected_streams, app_ids=None, is_multi_apps=True)
self.run_test(expected_streams={"visitors", "visitor_history"}, app_ids=None, is_multi_apps=True)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the visitors and visitor_history the only streams that support this test? I'm leaning towards using other streams here if possible. The logic being that this suite may be cumbersome to maintain since it's noted that data for these streams must be manually generated every 3 months. It would be nice not to lose all coverage and have the whole suite fail when that occurs and just have certain relevant tests fail (all streams, all fields, etc).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only "accounts", "visitors", "metadata_accounts" and "metadata_visitors" streams are not supported for this test and to ensure maximum test coverage within the three-hour maximum execution time in cci, we handle visitor_history stream in a specific manner. Note earlier we used to skip this stream due same limitation.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

tests/tap_tester/test_app_ids.py Outdated Show resolved Hide resolved
@RushiT0122 RushiT0122 requested a review from bhtowles April 12, 2023 18:10
@RushiT0122
Copy link
Contributor

@bhtowles please re-review.

expected_streams = {"features", "pages", "events", "guides", "track_types", "track_events"}
self.run_test(expected_streams=expected_streams, app_ids="-323232", is_multi_apps=False)
self.run_test(expected_streams=expected_streams, app_ids=None, is_multi_apps=True)
self.run_test(expected_streams={"visitors", "visitor_history"}, app_ids=None, is_multi_apps=True)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

@RushiT0122 RushiT0122 requested a review from kspeer825 July 20, 2023 11:47
@somethingmorerelevant somethingmorerelevant merged commit 050e17a into master Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants