Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws s3 bug fixes #2200

Merged
merged 1 commit into from
Jul 19, 2024
Merged

aws s3 bug fixes #2200

merged 1 commit into from
Jul 19, 2024

Conversation

joe-ayoub-segment
Copy link
Contributor

Fixes some bugs on S3 CSV Destination

Testing

Testing done locally

@joe-ayoub-segment joe-ayoub-segment requested a review from a team as a code owner July 19, 2024 14:16
@joe-ayoub-segment joe-ayoub-segment self-assigned this Jul 19, 2024
@joe-ayoub-segment joe-ayoub-segment merged commit f0e5ac2 into main Jul 19, 2024
11 checks passed
@joe-ayoub-segment joe-ayoub-segment deleted the aws-s3-bug-fixes branch July 19, 2024 14:19
nick-Ag added a commit that referenced this pull request Jul 23, 2024
@nick-Ag
Squashed commit of the following:
21b4f81 
commit 199765a87a153b1aadd3e8598f1ef16de28d938b
Merge: 3ea1444 f0e5ac2
Author: Nick Aguilar <[email protected]>
Date:   Mon Jul 22 17:16:35 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit 3ea1444
Author: Nick Aguilar <[email protected]>
Date:   Mon Jul 22 17:15:11 2024 -0700

    Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

commit f0e5ac2
Author: Joe Ayoub <[email protected]>
Date:   Fri Jul 19 16:19:18 2024 +0200

    aws s3 bug fixes (#2200)

commit 0a1285a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:08:29 2024 -0700

    Generates types

commit 8f5f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:04:14 2024 -0700

    WIP on an inexplicably not passing unit test

commit fc4f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 14:05:48 2024 -0700

    Checks for existence of user selected audience. Saves audience name and ID when returning from hook

commit a2a642c
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:58:03 2024 -0700

    WIP - verifying selected audience exists and saving it's ID as output

commit a022704
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:47:32 2024 -0700

    Removes description param from creating an audience per PRD

commit 49add30
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:44:07 2024 -0700

    Pulls down 200 most recent audiences in dynamic field response

commit c2f3e32
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:35:50 2024 -0700

    Removes pagination related code - that work will be saved for later :)

commit 6078690
Merge: b831f7b c9ecf66
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:31:29 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit b831f7b
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:14:15 2024 -0700

    Sets paging return such that the previous page was the current page

commit 365690e
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:09:14 2024 -0700

    Can pass dynamicFieldContext & paging into local server request

commit 037534c
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 14:37:23 2024 -0700

    Fixes build

commit 1e9d616
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:55:05 2024 -0700

    WIP on a pagination mechanism for dynamic fields

commit 151d26c
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:14:03 2024 -0700

    Adds a dynamic audience field, pulls down all customaudiences as choices

commit 27e4903
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 15:29:46 2024 -0700

    Adds required customer_file_source field

commit 60d2291
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:26:51 2024 -0700

    WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

commit 5ba6c09
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:05:23 2024 -0700

    Introduces shared adAccountId top-level setting and audienceSetting

commit 0a20b76
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:02:16 2024 -0700

    Removes delete action, add action. Introduces single sync action
nick-Ag pushed a commit that referenced this pull request Jul 23, 2024
@joe-ayoub-segment @nick-Ag
aws s3 bug fixes (#2200)
95b4593
nick-Ag added a commit that referenced this pull request Jul 25, 2024
@nick-Ag
Squashed commit of the following:
b963b94 
commit 185d425
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 10:59:42 2024 -0700

    Updates hook logic to consider the operation the user selected, if create is selected we will attempt to create even if an existing audience ID exists on the mapping

commit a54e845
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 24 14:05:05 2024 -0700

    Generates types

commit 8a7a08e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 13:11:32 2024 -0700

    Introduces a selector for users to choose whether they want to create a new rule or select an existing one

commit d7b8c1e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:39:58 2024 -0700

    Fixes unit tests

commit b9f0391
Merge: 95b4593 b6fc3f2
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:27:04 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit 95b4593
Author: Joe Ayoub <[email protected]>
Date:   Fri Jul 19 16:19:18 2024 +0200

    aws s3 bug fixes (#2200)

commit 945c7eb
Author: Pooya Jaferian <[email protected]>
Date:   Thu Jul 18 17:02:38 2024 -0700

    Adding support for FieldDisplayModes & disabling certain input methods (#2187)

    * update types

    * added a note on default displayMode

commit 3ea1444
Author: Nick Aguilar <[email protected]>
Date:   Mon Jul 22 17:15:11 2024 -0700

    Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

commit 0a1285a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:08:29 2024 -0700

    Generates types

commit 8f5f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:04:14 2024 -0700

    WIP on an inexplicably not passing unit test

commit fc4f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 14:05:48 2024 -0700

    Checks for existence of user selected audience. Saves audience name and ID when returning from hook

commit a2a642c
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:58:03 2024 -0700

    WIP - verifying selected audience exists and saving it's ID as output

commit a022704
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:47:32 2024 -0700

    Removes description param from creating an audience per PRD

commit 49add30
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:44:07 2024 -0700

    Pulls down 200 most recent audiences in dynamic field response

commit c2f3e32
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:35:50 2024 -0700

    Removes pagination related code - that work will be saved for later :)

commit 6078690
Merge: b831f7b c9ecf66
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:31:29 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit b831f7b
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:14:15 2024 -0700

    Sets paging return such that the previous page was the current page

commit 365690e
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:09:14 2024 -0700

    Can pass dynamicFieldContext & paging into local server request

commit 037534c
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 14:37:23 2024 -0700

    Fixes build

commit 1e9d616
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:55:05 2024 -0700

    WIP on a pagination mechanism for dynamic fields

commit 151d26c
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:14:03 2024 -0700

    Adds a dynamic audience field, pulls down all customaudiences as choices

commit 27e4903
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 15:29:46 2024 -0700

    Adds required customer_file_source field

commit 60d2291
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:26:51 2024 -0700

    WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

commit 5ba6c09
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:05:23 2024 -0700

    Introduces shared adAccountId top-level setting and audienceSetting

commit 0a20b76
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:02:16 2024 -0700

    Removes delete action, add action. Introduces single sync action
nick-Ag added a commit that referenced this pull request Jul 29, 2024
@nick-Ag
Squashed commit of the following:
12c1c79 
commit a174a3c
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 15:15:56 2024 -0700

    Adds a unit test for multiple payloads. Generates types

commit fa0131d
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 14:58:38 2024 -0700

    Updates generateData to be a shared function so it can be exported and unit tested directly. Fixes minor issues and the first unit test is passing

commit 8ec6462
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 14:14:55 2024 -0700

    Major implementation of data & schema generation algorithm. The schema is hardcoded to always have the same order, irrespective to whether the payloads contain any of the keys in the hardcoded schema, this is to avoid having to generate a schema based on the present keys across all payloads. This is the method the classic ADS version of this destination takes as well. To generate the 2d data array all payloads are iterated thru, and each present value is inserted into the correct index of a row array which has the same length as the pre-defined schemas array. Each of these row arrays is sequentially appended to the data array. Includes 1 unit test, untested

commit 051cc65
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 15:35:55 2024 -0700

    A broken unit test, which I will fix with the next commit

commit 0e98649
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:59:13 2024 -0700

    Implements delete operation

commit f747dfd
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:50:07 2024 -0700

    Style updates and console.log removals

commit 48ff851
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:46:41 2024 -0700

    Fixes URL when syncing audience. Removes console.logs

commit 2e1f7ac
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:37:31 2024 -0700

    Updates usage of adAccountId setting to retlAdAccountId. Fixes type error

commit 8c6111a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 17:03:00 2024 -0700

    WIP on syncing users - request likely failing due to bad permissions

commit 2cf86bf
Merge: 1bac8c3 bfc9b86
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 12:07:22 2024 -0700

    Merge remote-tracking branch 'origin/main' into fbca-retl-hook

commit 1bac8c3
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 12:06:03 2024 -0700

    Removes stray change to aws destination

commit 185d425
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 10:59:42 2024 -0700

    Updates hook logic to consider the operation the user selected, if create is selected we will attempt to create even if an existing audience ID exists on the mapping

commit a54e845
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 24 14:05:05 2024 -0700

    Generates types

commit 8a7a08e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 13:11:32 2024 -0700

    Introduces a selector for users to choose whether they want to create a new rule or select an existing one

commit d7b8c1e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:39:58 2024 -0700

    Fixes unit tests

commit b9f0391
Merge: 95b4593 b6fc3f2
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:27:04 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit 95b4593
Author: Joe Ayoub <[email protected]>
Date:   Fri Jul 19 16:19:18 2024 +0200

    aws s3 bug fixes (#2200)

commit 945c7eb
Author: Pooya Jaferian <[email protected]>
Date:   Thu Jul 18 17:02:38 2024 -0700

    Adding support for FieldDisplayModes & disabling certain input methods (#2187)

    * update types

    * added a note on default displayMode

commit 3ea1444
Author: Nick Aguilar <[email protected]>
Date:   Mon Jul 22 17:15:11 2024 -0700

    Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

commit 0a1285a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:08:29 2024 -0700

    Generates types

commit 8f5f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:04:14 2024 -0700

    WIP on an inexplicably not passing unit test

commit fc4f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 14:05:48 2024 -0700

    Checks for existence of user selected audience. Saves audience name and ID when returning from hook

commit a2a642c
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:58:03 2024 -0700

    WIP - verifying selected audience exists and saving it's ID as output

commit a022704
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:47:32 2024 -0700

    Removes description param from creating an audience per PRD

commit 49add30
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:44:07 2024 -0700

    Pulls down 200 most recent audiences in dynamic field response

commit c2f3e32
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:35:50 2024 -0700

    Removes pagination related code - that work will be saved for later :)

commit 6078690
Merge: b831f7b c9ecf66
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:31:29 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit b831f7b
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:14:15 2024 -0700

    Sets paging return such that the previous page was the current page

commit 365690e
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:09:14 2024 -0700

    Can pass dynamicFieldContext & paging into local server request

commit 037534c
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 14:37:23 2024 -0700

    Fixes build

commit 1e9d616
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:55:05 2024 -0700

    WIP on a pagination mechanism for dynamic fields

commit 151d26c
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:14:03 2024 -0700

    Adds a dynamic audience field, pulls down all customaudiences as choices

commit 27e4903
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 15:29:46 2024 -0700

    Adds required customer_file_source field

commit 60d2291
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:26:51 2024 -0700

    WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

commit 5ba6c09
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:05:23 2024 -0700

    Introduces shared adAccountId top-level setting and audienceSetting

commit 0a20b76
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:02:16 2024 -0700

    Removes delete action, add action. Introduces single sync action
nick-Ag added a commit that referenced this pull request Jul 29, 2024
@nick-Ag
Squashed commit of the following:
261fe1a 
commit a174a3c
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 15:15:56 2024 -0700

    Adds a unit test for multiple payloads. Generates types

commit fa0131d
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 14:58:38 2024 -0700

    Updates generateData to be a shared function so it can be exported and unit tested directly. Fixes minor issues and the first unit test is passing

commit 8ec6462
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 14:14:55 2024 -0700

    Major implementation of data & schema generation algorithm. The schema is hardcoded to always have the same order, irrespective to whether the payloads contain any of the keys in the hardcoded schema, this is to avoid having to generate a schema based on the present keys across all payloads. This is the method the classic ADS version of this destination takes as well. To generate the 2d data array all payloads are iterated thru, and each present value is inserted into the correct index of a row array which has the same length as the pre-defined schemas array. Each of these row arrays is sequentially appended to the data array. Includes 1 unit test, untested

commit 051cc65
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 15:35:55 2024 -0700

    A broken unit test, which I will fix with the next commit

commit 0e98649
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:59:13 2024 -0700

    Implements delete operation

commit f747dfd
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:50:07 2024 -0700

    Style updates and console.log removals

commit 48ff851
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:46:41 2024 -0700

    Fixes URL when syncing audience. Removes console.logs

commit 2e1f7ac
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:37:31 2024 -0700

    Updates usage of adAccountId setting to retlAdAccountId. Fixes type error

commit 8c6111a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 17:03:00 2024 -0700

    WIP on syncing users - request likely failing due to bad permissions

commit 2cf86bf
Merge: 1bac8c3 bfc9b86
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 12:07:22 2024 -0700

    Merge remote-tracking branch 'origin/main' into fbca-retl-hook

commit 1bac8c3
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 12:06:03 2024 -0700

    Removes stray change to aws destination

commit 185d425
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 10:59:42 2024 -0700

    Updates hook logic to consider the operation the user selected, if create is selected we will attempt to create even if an existing audience ID exists on the mapping

commit a54e845
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 24 14:05:05 2024 -0700

    Generates types

commit 8a7a08e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 13:11:32 2024 -0700

    Introduces a selector for users to choose whether they want to create a new rule or select an existing one

commit d7b8c1e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:39:58 2024 -0700

    Fixes unit tests

commit b9f0391
Merge: 95b4593 b6fc3f2
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:27:04 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit 95b4593
Author: Joe Ayoub <[email protected]>
Date:   Fri Jul 19 16:19:18 2024 +0200

    aws s3 bug fixes (#2200)

commit 945c7eb
Author: Pooya Jaferian <[email protected]>
Date:   Thu Jul 18 17:02:38 2024 -0700

    Adding support for FieldDisplayModes & disabling certain input methods (#2187)

    * update types

    * added a note on default displayMode

commit 3ea1444
Author: Nick Aguilar <[email protected]>
Date:   Mon Jul 22 17:15:11 2024 -0700

    Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

commit 0a1285a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:08:29 2024 -0700

    Generates types

commit 8f5f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:04:14 2024 -0700

    WIP on an inexplicably not passing unit test

commit fc4f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 14:05:48 2024 -0700

    Checks for existence of user selected audience. Saves audience name and ID when returning from hook

commit a2a642c
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:58:03 2024 -0700

    WIP - verifying selected audience exists and saving it's ID as output

commit a022704
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:47:32 2024 -0700

    Removes description param from creating an audience per PRD

commit 49add30
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:44:07 2024 -0700

    Pulls down 200 most recent audiences in dynamic field response

commit c2f3e32
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:35:50 2024 -0700

    Removes pagination related code - that work will be saved for later :)

commit 6078690
Merge: b831f7b c9ecf66
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:31:29 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit b831f7b
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:14:15 2024 -0700

    Sets paging return such that the previous page was the current page

commit 365690e
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:09:14 2024 -0700

    Can pass dynamicFieldContext & paging into local server request

commit 037534c
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 14:37:23 2024 -0700

    Fixes build

commit 1e9d616
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:55:05 2024 -0700

    WIP on a pagination mechanism for dynamic fields

commit 151d26c
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:14:03 2024 -0700

    Adds a dynamic audience field, pulls down all customaudiences as choices

commit 27e4903
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 15:29:46 2024 -0700

    Adds required customer_file_source field

commit 60d2291
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:26:51 2024 -0700

    WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

commit 5ba6c09
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:05:23 2024 -0700

    Introduces shared adAccountId top-level setting and audienceSetting

commit 0a20b76
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:02:16 2024 -0700

    Removes delete action, add action. Introduces single sync action
nick-Ag added a commit that referenced this pull request Jul 30, 2024
@nick-Ag
Squashed commit of the following:
4639c78 
commit 907e77d
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 30 14:13:12 2024 -0700

    Adds batching configuration with max batch size of 10,000. Throws an error if sync mode is not included and the destination does no operation

commit a174a3c
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 15:15:56 2024 -0700

    Adds a unit test for multiple payloads. Generates types

commit fa0131d
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 14:58:38 2024 -0700

    Updates generateData to be a shared function so it can be exported and unit tested directly. Fixes minor issues and the first unit test is passing

commit 8ec6462
Author: Nick Aguilar <[email protected]>
Date:   Sun Jul 28 14:14:55 2024 -0700

    Major implementation of data & schema generation algorithm. The schema is hardcoded to always have the same order, irrespective to whether the payloads contain any of the keys in the hardcoded schema, this is to avoid having to generate a schema based on the present keys across all payloads. This is the method the classic ADS version of this destination takes as well. To generate the 2d data array all payloads are iterated thru, and each present value is inserted into the correct index of a row array which has the same length as the pre-defined schemas array. Each of these row arrays is sequentially appended to the data array. Includes 1 unit test, untested

commit 051cc65
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 15:35:55 2024 -0700

    A broken unit test, which I will fix with the next commit

commit 0e98649
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:59:13 2024 -0700

    Implements delete operation

commit f747dfd
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:50:07 2024 -0700

    Style updates and console.log removals

commit 48ff851
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:46:41 2024 -0700

    Fixes URL when syncing audience. Removes console.logs

commit 2e1f7ac
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 14:37:31 2024 -0700

    Updates usage of adAccountId setting to retlAdAccountId. Fixes type error

commit 8c6111a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 17:03:00 2024 -0700

    WIP on syncing users - request likely failing due to bad permissions

commit 2cf86bf
Merge: 1bac8c3 bfc9b86
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 12:07:22 2024 -0700

    Merge remote-tracking branch 'origin/main' into fbca-retl-hook

commit 1bac8c3
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 12:06:03 2024 -0700

    Removes stray change to aws destination

commit 185d425
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 25 10:59:42 2024 -0700

    Updates hook logic to consider the operation the user selected, if create is selected we will attempt to create even if an existing audience ID exists on the mapping

commit a54e845
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 24 14:05:05 2024 -0700

    Generates types

commit 8a7a08e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 13:11:32 2024 -0700

    Introduces a selector for users to choose whether they want to create a new rule or select an existing one

commit d7b8c1e
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:39:58 2024 -0700

    Fixes unit tests

commit b9f0391
Merge: 95b4593 b6fc3f2
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 23 10:27:04 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit 95b4593
Author: Joe Ayoub <[email protected]>
Date:   Fri Jul 19 16:19:18 2024 +0200

    aws s3 bug fixes (#2200)

commit 945c7eb
Author: Pooya Jaferian <[email protected]>
Date:   Thu Jul 18 17:02:38 2024 -0700

    Adding support for FieldDisplayModes & disabling certain input methods (#2187)

    * update types

    * added a note on default displayMode

commit 3ea1444
Author: Nick Aguilar <[email protected]>
Date:   Mon Jul 22 17:15:11 2024 -0700

    Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

commit 0a1285a
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:08:29 2024 -0700

    Generates types

commit 8f5f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 15:04:14 2024 -0700

    WIP on an inexplicably not passing unit test

commit fc4f5ed
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 14:05:48 2024 -0700

    Checks for existence of user selected audience. Saves audience name and ID when returning from hook

commit a2a642c
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:58:03 2024 -0700

    WIP - verifying selected audience exists and saving it's ID as output

commit a022704
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:47:32 2024 -0700

    Removes description param from creating an audience per PRD

commit 49add30
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:44:07 2024 -0700

    Pulls down 200 most recent audiences in dynamic field response

commit c2f3e32
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:35:50 2024 -0700

    Removes pagination related code - that work will be saved for later :)

commit 6078690
Merge: b831f7b c9ecf66
Author: Nick Aguilar <[email protected]>
Date:   Thu Jul 18 12:31:29 2024 -0700

    Merge branch 'main' into fbca-retl-hook

commit b831f7b
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:14:15 2024 -0700

    Sets paging return such that the previous page was the current page

commit 365690e
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 15:09:14 2024 -0700

    Can pass dynamicFieldContext & paging into local server request

commit 037534c
Author: Nick Aguilar <[email protected]>
Date:   Wed Jul 17 14:37:23 2024 -0700

    Fixes build

commit 1e9d616
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:55:05 2024 -0700

    WIP on a pagination mechanism for dynamic fields

commit 151d26c
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 16:14:03 2024 -0700

    Adds a dynamic audience field, pulls down all customaudiences as choices

commit 27e4903
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 15:29:46 2024 -0700

    Adds required customer_file_source field

commit 60d2291
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:26:51 2024 -0700

    WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

commit 5ba6c09
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:05:23 2024 -0700

    Introduces shared adAccountId top-level setting and audienceSetting

commit 0a20b76
Author: Nick Aguilar <[email protected]>
Date:   Tue Jul 16 12:02:16 2024 -0700

    Removes delete action, add action. Introduces single sync action
marinhero pushed a commit that referenced this pull request Aug 2, 2024
@joe-ayoub-segment @marinhero
aws s3 bug fixes (#2200)
716fcdb
nick-Ag added a commit that referenced this pull request Aug 5, 2024
@nick-Ag @pooyaj @joe-ayoub-segment
[FB Custom Audiences] Audience Selection/Creation RETL Hook (#2197)
9807e5c 
* Removes delete action, add action. Introduces single sync action

* Introduces shared adAccountId top-level setting and audienceSetting

* WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

* Adds required customer_file_source field

* Adds a dynamic audience field, pulls down all customaudiences as choices

* WIP on a pagination mechanism for dynamic fields

* Fixes build

* Can pass dynamicFieldContext & paging into local server request

* Sets paging return such that the previous page was the current page

* Removes pagination related code - that work will be saved for later :)

* Pulls down 200 most recent audiences in dynamic field response

* Removes description param from creating an audience per PRD

* WIP - verifying selected audience exists and saving it's ID as output

* Checks for existence of user selected audience. Saves audience name and ID when returning from hook

* WIP on an inexplicably not passing unit test

* Generates types

* Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

* Adding support for FieldDisplayModes & disabling certain input methods (#2187)

* update types

* added a note on default displayMode

* aws s3 bug fixes (#2200)

* Fixes unit tests

* Introduces a selector for users to choose whether they want to create a new rule or select an existing one

* Generates types

* Updates hook logic to consider the operation the user selected, if create is selected we will attempt to create even if an existing audience ID exists on the mapping

* Removes stray change to aws destination

* [Facebook Custom Audiences] Sync users to an audience (#2233)

* WIP on syncing users - request likely failing due to bad permissions

* Updates usage of adAccountId setting to retlAdAccountId. Fixes type error

* Fixes URL when syncing audience. Removes console.logs

* Style updates and console.log removals

* Implements delete operation

* A broken unit test, which I will fix with the next commit

* Major implementation of data & schema generation algorithm. The schema is hardcoded to always have the same order, irrespective to whether the payloads contain any of the keys in the hardcoded schema, this is to avoid having to generate a schema based on the present keys across all payloads. This is the method the classic ADS version of this destination takes as well. To generate the 2d data array all payloads are iterated thru, and each present value is inserted into the correct index of a row array which has the same length as the pre-defined schemas array. Each of these row arrays is sequentially appended to the data array. Includes 1 unit test, untested

* Updates generateData to be a shared function so it can be exported and unit tested directly. Fixes minor issues and the first unit test is passing

* Adds a unit test for multiple payloads. Generates types

* Adds batching configuration with max batch size of 10,000. Throws an error if sync mode is not included and the destination does no operation

* Updates data generation so batching fields don't trigger an error and are instead ignored. Updates unit tests, generates types

* Fixes the referencing of hookOutput fields

* Removes a try catch that was discarding errors

* Updates sync action unit test

* Adds Data normalization to the destination. Includes new unit tests

* Updates a unit test with normalization

* Removes address object field in favor of individual fields for each item (city, state, etc)

* Simplifies normalization functions

* Removes unused currency code constants

* Removes add and update sync modes. This destination will only support upsert and delete for now

---------

Co-authored-by: Pooya Jaferian <[email protected]>
Co-authored-by: Joe Ayoub <[email protected]>
harsh-joshi99 pushed a commit that referenced this pull request Aug 16, 2024
@joe-ayoub-segment @harsh-joshi99
aws s3 bug fixes (#2200)
83b3bfe
harsh-joshi99 pushed a commit that referenced this pull request Aug 16, 2024
@nick-Ag @pooyaj
@joe-ayoub-segment @harsh-joshi99
[FB Custom Audiences] Audience Selection/Creation RETL Hook (#2197)
c8907ba 
* Removes delete action, add action. Introduces single sync action

* Introduces shared adAccountId top-level setting and audienceSetting

* WIP - first draft of a FacebookClient class, including an untested createAudience hook for retl

* Adds required customer_file_source field

* Adds a dynamic audience field, pulls down all customaudiences as choices

* WIP on a pagination mechanism for dynamic fields

* Fixes build

* Can pass dynamicFieldContext & paging into local server request

* Sets paging return such that the previous page was the current page

* Removes pagination related code - that work will be saved for later :)

* Pulls down 200 most recent audiences in dynamic field response

* Removes description param from creating an audience per PRD

* WIP - verifying selected audience exists and saving it's ID as output

* Checks for existence of user selected audience. Saves audience name and ID when returning from hook

* WIP on an inexplicably not passing unit test

* Generates types

* Splits off engage and retl AdAccountId settings, fun fact: these cannot share a slug

* Adding support for FieldDisplayModes & disabling certain input methods (#2187)

* update types

* added a note on default displayMode

* aws s3 bug fixes (#2200)

* Fixes unit tests

* Introduces a selector for users to choose whether they want to create a new rule or select an existing one

* Generates types

* Updates hook logic to consider the operation the user selected, if create is selected we will attempt to create even if an existing audience ID exists on the mapping

* Removes stray change to aws destination

* [Facebook Custom Audiences] Sync users to an audience (#2233)

* WIP on syncing users - request likely failing due to bad permissions

* Updates usage of adAccountId setting to retlAdAccountId. Fixes type error

* Fixes URL when syncing audience. Removes console.logs

* Style updates and console.log removals

* Implements delete operation

* A broken unit test, which I will fix with the next commit

* Major implementation of data & schema generation algorithm. The schema is hardcoded to always have the same order, irrespective to whether the payloads contain any of the keys in the hardcoded schema, this is to avoid having to generate a schema based on the present keys across all payloads. This is the method the classic ADS version of this destination takes as well. To generate the 2d data array all payloads are iterated thru, and each present value is inserted into the correct index of a row array which has the same length as the pre-defined schemas array. Each of these row arrays is sequentially appended to the data array. Includes 1 unit test, untested

* Updates generateData to be a shared function so it can be exported and unit tested directly. Fixes minor issues and the first unit test is passing

* Adds a unit test for multiple payloads. Generates types

* Adds batching configuration with max batch size of 10,000. Throws an error if sync mode is not included and the destination does no operation

* Updates data generation so batching fields don't trigger an error and are instead ignored. Updates unit tests, generates types

* Fixes the referencing of hookOutput fields

* Removes a try catch that was discarding errors

* Updates sync action unit test

* Adds Data normalization to the destination. Includes new unit tests

* Updates a unit test with normalization

* Removes address object field in favor of individual fields for each item (city, state, etc)

* Simplifies normalization functions

* Removes unused currency code constants

* Removes add and update sync modes. This destination will only support upsert and delete for now

---------

Co-authored-by: Pooya Jaferian <[email protected]>
Co-authored-by: Joe Ayoub <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@varadarajan-tw varadarajan-tw varadarajan-tw approved these changes

Assignees

@joe-ayoub-segment joe-ayoub-segment

Labels
deploy:push mode:cloud partner-eng-review (cloud) Ready for Release team:segment-core
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@joe-ayoub-segment @varadarajan-tw