[EPIC] Source Google Analytics to Beta

[sherifnada]

The main purpose of this issue is to track release progress for the connector.

According to checklist:

• MLP improvements:

  • Source Google Analytics: Implement Advanced Reports #11689

• Engineering Acceptance Criteria:

  • Source Google Analytics: increase unit test and acceptance test coverage #11477

• All outstanding bug reports blocking GA:

  • Source GA: DEADLINE_EXCEEDED exception #4536
  • https://github.com/airbytehq/alpha-beta-issues/issues/45
  • https://github.com/airbytehq/alpha-beta-issues/issues/48
  • https://github.com/airbytehq/alpha-beta-issues/issues/33
  • Google Analytics source: expose “isDataGolden” flag #12013

• Label connector as Beta in cloud:

  • https://github.com/airbytehq/airbyte-cloud/pull/1590

• Resolve issues:

  • Google Analytics source: expose “isDataGolden” flag #12013

Acceptance criteria

• Create a checklist document that should rbe placed on the Google Drive
• Checklist filled up by engineering team
• Collect and list all tech debt-related issues in the section below
• Create/Revise MLP document and place it here
• Move checklist to the unblockers team
• Reviewed/Updated by unblockers team
• Move checklist to the product team for review
• Reviewed/Updated by product team
• Update connector documentation according to the comments in the checklist
• Certification completed

[misteryeo]

@davydov-d Reviewed the checklist again - didn't see a Docs PR to review, can you share?

[ChristopheDuong]

@sherifnada @misteryeo , should this issue #12013 be included in this epic too?

[davydov-d]

@davydov-d Reviewed the checklist again - didn't see a Docs PR to review, can you share?

@misteryeo Sure, but it's merged already.. I was not aware you review PRs as well
https://github.com/airbytehq/airbyte/pull/12150/files

[misteryeo]

@misteryeo Sure, but it's merged already.. I was not aware you review PRs as well
https://github.com/airbytehq/airbyte/pull/12150/files

@davydov-d Oh I only asked because we were discussing the 'Window in days' field and whether 'data chunk' should be used. On that field specifically, I'm still unclear what exactly 'Window in days' means even with the description - what does 'stream slice' mean? Are there external docs that we can link to so we can make this clearer?

@sherifnada @misteryeo , should this issue #12013 be included in this epic too?

Yes, thanks for bringing this up @ChristopheDuong - was reviewing Thalia's response. Saw there's some chatter happening over there but @davydov-d let's make sure to implement this as part of this verification process.

[davydov-d]

@misteryeo Sure, but it's merged already.. I was not aware you review PRs as well
https://github.com/airbytehq/airbyte/pull/12150/files

@davydov-d Oh I only asked because we were discussing the 'Window in days' field and whether 'data chunk' should be used. On that field specifically, I'm still unclear what exactly 'Window in days' means even with the description - what does 'stream slice' mean? Are there external docs that we can link to so we can make this clearer?

No, stream slice is our internal technical term. As a rule, under the hood the connector makes a series of requests to a resource, each of which retrieves a batch/chunk/slice of data during the sync. The window in days defines a date range filter for every request. For example, when start_date = 2020-01-01 and window_in_days=30, request filters will look like
2020-01-01..2020-01-30
2020-01-31..2020-02-29
2020-03-01..2020-03-30
How would you like this parameter to be descripted? I guess it should be something laconic, not that large I've written here ☝️

@sherifnada @misteryeo , should this issue #12013 be included in this epic too?

Yes, thanks for bringing this up @ChristopheDuong - was reviewing Thalia's response. Saw there's some chatter happening over there but @davydov-d let's make sure to implement this as part of this verification process.

ok

[misteryeo]

And to confirm @davydov-d - there's no external documentation referencing this field? This is an Airbyte created field?

[davydov-d]

And to confirm @davydov-d - there's no external documenta

well, we can reference this one https://developers.google.com/analytics/devguides/reporting/core/v4/rest/v4/DateRange
window_in_days is a number of days between startDate and endDate

[misteryeo]

@davydov-d but the field itself, is describing sampling - correct? https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py#L25

So what the field is doing, is determining how big of a sample size of data a user would want to sync each time based on a range of days from the start_date. The larger this day range, the large the sample and the slower it will be but the more accurate it will be. Does this seem correct?

Clarifying because this is opposite of what's currently described in the field description today:

Window in days (Optional) - The amount of days each stream slice would consist of beginning from start_date. Bigger the value - faster the fetch. (Min=1, as for a Day; Max=364, as for a Year). (e.g. 30, 60, 90, 120, 200, 364)

[davydov-d]

@davydov-d but the field itself, is describing sampling - correct? https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py#L25

So what the field is doing, is determining how big of a sample size of data a user would want to sync each time based on a range of days from the start_date. The larger this day range, the large the sample and the slower it will be but the more accurate it will be. Does this seem correct?

Clarifying because this is opposite of what's currently described in the field description today:

Window in days (Optional) - The amount of days each stream slice would consist of beginning from start_date. Bigger the value - faster the fetch. (Min=1, as for a Day; Max=364, as for a Year). (e.g. 30, 60, 90, 120, 200, 364)

well, yes but no :)
According to the doc, sampling level is defined by the samplingLevel parameter, which we do not expose in the input config(should we?), so the DEFAULT level is used meaning speed and accuracy of the response are balanced. The response speed depends on both accuracy and the date range. So, in case we have a large date range, we can speed up the response time by decreasing the date range, leaving the accuracy approximately the same. On the other hand, it also means increasing the number of requests. In other words, the window_in_days field indirectly affects sampling(only after it kicks in) but that's not the primary goal of this field.

I'm not so sure about Bigger the value - faster the fetch. Bigger the value - slower the single response, fewer the number of requests. Not confident about the total speed of the whole sync, so I would omit this phrase

[misteryeo]

Gotcha, let's update the description to - does that resonate with you @davydov-d:

Data request window (Optional) - The amount of data batched by the number of days. The bigger the value, the bigger the batch size and the lower the API requests made.

[davydov-d]

Gotcha, let's update the description to - does that resonate with you @davydov-d:

Data request window (Optional) - The amount of data batched by the number of days. The bigger the value, the bigger the batch size and the lower the API requests made.

sounds good
@misteryeo have a look please #12385
are we done with the checklist here?

[sherifnada]

@davydov-d yes please consider the checklist done

[davydov-d]

Successfully certified as Beta: https://docs.google.com/spreadsheets/d/1BRkxXPpAulWBZYOXs5W5ni9sAto06wtBuUKNBGBf0XI/edit#gid=90626852