Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(types): dedupe unique and greatest strategy needs id #11945

Merged
merged 4 commits into from
May 16, 2024

Conversation

joscha
Copy link
Contributor

@joscha joscha commented May 14, 2024

WHY

When using the dedupe strategy unique and omitting the id field in the metadata, events are not emitted correctly. They seem to be swallowed and also don't show up in the event interface.

Copy link

vercel bot commented May 14, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Ignored Deployments
Name Status Preview Comments Updated (UTC)
docs-v2 ⬜️ Ignored (Inspect) Visit Preview May 14, 2024 11:11pm
pipedream-docs-redirect-do-not-edit ⬜️ Ignored (Inspect) May 14, 2024 11:11pm

@dylburger dylburger added the User submitted Submitted by a user label May 14, 2024
@dylburger dylburger added the triaged For maintainers: This issue has been triaged by a Pipedream employee label May 14, 2024
@@ -310,15 +310,23 @@ export interface EmitMetadata {
ts?: number;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly the timestamp is a prerequisite for the greatest strategy. Let me know if that is the case and I can make the change.

types/src/index.ts Outdated Show resolved Hide resolved
types/src/index.ts Outdated Show resolved Hide resolved
@joscha joscha force-pushed the dedupe-needs-id branch from 8eaa445 to aa2392f Compare May 14, 2024 22:33
@joscha joscha force-pushed the dedupe-needs-id branch from aa2392f to ea09407 Compare May 14, 2024 22:39
@joscha
Copy link
Contributor Author

joscha commented May 14, 2024

I seem to be getting:

Base commit: 9a188c64ca1d53ebef2c20d38c7e7815ad276f45
Head commit: ea09407ac541889b[6](https://github.com/PipedreamHQ/pipedream/actions/runs/9087144710/job/24974295576?pr=11945#step:4:7)eed03786120b8f1c5d6b409
Error: The head commit for this pull_request event is not ahead of the base commit. Please submit an issue on this action's GitHub repo.

from the spellcheck step (see https://github.com/PipedreamHQ/pipedream/actions/runs/9087144710/job/24974295576). I thought it was because of an outdated master reference, but even after a rebase I can't seem to make it go away.

Maybe the rebase:

git log 9a188c64ca1d53ebef2c20d38c7e7815ad276f45
fatal: bad object 9a188c64ca1d53ebef2c20d38c7e7815ad276f45

is this actual issue? The base commit is nowhere to be found.
Seems to be related to jitterbit/get-changed-files#54, jitterbit/get-changed-files#4, jitterbit/get-changed-files#7 and jitterbit/get-changed-files#11

The repository with the action has not been updated in 4 years. Suggested "fix" is:

       - uses: jitterbit/get-changed-files@v1
         id: files
         continue-on-error: true

pushed 70e85af to this PR with the suggested workaround.

Copy link
Contributor Author

@joscha joscha May 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for #11945 (comment) a possibly better way would be to use a fork of the original unmaintained action in https://github.com/Ana06/get-changed-files - it's a drop in replacement with a fix for the issue and a few other improvements. Let me know if you prefer the fork.

Copy link
Contributor Author

@joscha joscha May 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opened an adjacent pull request in #11949
will rebase this PR on top of it, if it gets merged before and remove the continue-on-error flags.

@lcaresia
Copy link
Collaborator

@michelle0927 Can you make a review in this PR too? just for sure

@lcaresia lcaresia requested a review from michelle0927 May 15, 2024 13:32
@vunguyenhung
Copy link
Collaborator

Merge this on behalf of user

@vunguyenhung vunguyenhung merged commit fabecdd into PipedreamHQ:master May 16, 2024
7 checks passed
@joscha joscha deleted the dedupe-needs-id branch May 16, 2024 06:23
dannyroosevelt added a commit that referenced this pull request May 24, 2024
* Fixing typo in GH error message (#11816)

* Fixing typo in GH error message

* Incrementing versions

* Bump tqdm from 4.66.1 to 4.66.3 in /blog/pi (#11818)

Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.66.1 to 4.66.3.
- [Release notes](https://github.com/tqdm/tqdm/releases)
- [Commits](tqdm/tqdm@v4.66.1...v4.66.3)

---
updated-dependencies:
- dependency-name: tqdm
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tqdm from 4.66.1 to 4.66.3 in /packages/component_code_gen (#11819)

Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.66.1 to 4.66.3.
- [Release notes](https://github.com/tqdm/tqdm/releases)
- [Commits](tqdm/tqdm@v4.66.1...v4.66.3)

---
updated-dependencies:
- dependency-name: tqdm
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Adding HIPAA docs, removing deprecated migration plan (#11729)

* New Components - vryno (#11628)

* vryno init

* [Components] vryno #11624
Actions
 - Create Unique Lead

* pnpm update

* remove console.log

* some adjusts

* New Components - orimon (#11802)

* orimon init

* [Components] orimon #11796
Actions
- Send Message

* pnpm update

* bug fix (#11812)

* New Components - relavate (#11803)

* relavate init

* new components

* pnpm-lock.yaml

* fix typo

* Telnyx - Ability to retrieve a message (#11710)

* Added the ability to list phone numbers

* Adjusted component/action versions

* eslint

* Updated constants file

* constants.mjs

* version

* Added Retrieve Message action

* Fixed error message details

* versions

---------

Co-authored-by: michelle0927 <[email protected]>
Co-authored-by: michelle0927 <[email protected]>

* Adding newest YT intro video

* Reusing Gitlab components for Developer App (#11603)

* New app/package

* Creating actions reusing Gitlab actions

* Package update

* pnpm

* Mapping propDefinitions to new app

* Adjusting actions' propDefinitions

* Adjusting sources to use mapped propDefinitions

* Removing unused file

* Fixing import error for sources

* Updating list projects

* Gitlab version bumps

* package newline

* Description update

* Making username required

* Fix labels and parentId optional

* Fixing 'new milstone'

* Adjusting 'new review request'

* Update issue: fixing labels on request

* Replacing node-fetch with axios

* Splittin group path and group id

* Adding error treatment for source

* Bump jinja2 from 3.1.3 to 3.1.4 in /blog/pi (#11825)

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WooCommerce App Marketplace page (#11826)

* Adding troubleshooting and getting started steps

* correcting hierarchy

* example use case docs

* Migrated to latest rest api version (#11707)

* Added actions (#11808)

* New Components - tripadvisor_content_api (#11403)

* tripadvisor_content_api init

* Added actons

* Update components/tripadvisor_content_api/actions/location-reviews/location-reviews.mjs

Co-authored-by: Luan Cazarine <[email protected]>

* Update components/tripadvisor_content_api/actions/location-search/location-search.mjs

Co-authored-by: Luan Cazarine <[email protected]>

* Update components/tripadvisor_content_api/tripadvisor_content_api.app.mjs

Co-authored-by: Luan Cazarine <[email protected]>

* Update components/tripadvisor_content_api/tripadvisor_content_api.app.mjs

Co-authored-by: Luan Cazarine <[email protected]>

* Update components/tripadvisor_content_api/tripadvisor_content_api.app.mjs

Co-authored-by: Luan Cazarine <[email protected]>

* Update components/tripadvisor_content_api/tripadvisor_content_api.app.mjs

Co-authored-by: Luan Cazarine <[email protected]>

* Bump package.json version

* Remove redundant prop

* Remove redundant prop

* Revert "Remove redundant prop"

This reverts commit 80b81c4.

* Revert "Remove redundant prop"

This reverts commit bca8cf1.

* Fixing file name

---------

Co-authored-by: Luan Cazarine <[email protected]>
Co-authored-by: Leo Vu <[email protected]>
Co-authored-by: Leo Vu <[email protected]>

* Adding app scaffolding for fly_io

* Adding mongodb connecting and troubleshooting (#11835)

* Tested component with queries (#11828)

* Firestore Firebase - parse boolean values in Update Document & Create Document (#11801)

* parse boolean values

* versions

* Remove 0 and 1 case for boolean

* Fix typos

---------

Co-authored-by: Leo Vu <[email protected]>
Co-authored-by: Leo Vu <[email protected]>

* Merge branch 'issue-11579' into 9395-action-zoom-app-zoom-phones-get-call-logs-action (#11781)

* AWS readme (#11841)

* Google Drive Triggers text/usability improvements (#11652)

* Description update

* Adding descriptions for update types

* description adjustment

* Adjustments

* Text improvements

* text update

* Reverting specific drive change

* Version bumps

* Creating common file interval deduping

* Applying dedupe changes for file modification sources

* Version bumps

* Version bumps

* Moving constants to common folder

* adjusting 'changes' schema

* 'Share File' improvements

* Sharing file adjustments

* Adding advanced role options

* Version adjustment

* Updating component code to use PD Proxy for requests. (#11842)

* Updating component code to use PD Proxy for requests.

* Version bump.

* Tested components (#11820)

* GDrive Component description updates (#11844)

* SendGrid readme (#11850)

* New Components - bilflo (#11840)

* bilflo init

* [Components] bilflo #11732
Actions
 - Create Client
 - Assign Contract Job To Invoice
 - Create Contract Job

* pnpm update

* add datetime format description

* New Components - cradl_ai (#11830)

* cradl_ai init

* new components

* pnpm-lock.yaml

* Klayvio Readme (#11852)

* Zendesk API (#11853)

* 9395 action zoom app zoom phones get call logs action (#11854)

* Merge branch 'issue-11579' into 9395-action-zoom-app-zoom-phones-get-call-logs-action

* bump version

* Auto-generated documentation for apps (#11800)

* Auto-generated documentation for apps

* Overledger (`overledger`)
* Chaser (`chaser`)
* SARE (`sare`)
* Encodian (`encodian`)
* ByteNite (`bytenite`)
* Sigma (`sigma`)
* Thoughtly (`thoughtly`)
* Plate Recognizer (`platerecognizer`)
* Flipando (`flipando`)
* Relavate (`relavate`)
* TimeTonic (`timetonic`)
* Cradl AI (`cradl_ai`)
* Bilflo (`bilflo`)
* Navigatr (`navigatr`)
* ChatFly (`chatfly`)
* Perplexity (`perplexity`)
* FracTEL (`fractel`)
* y.gy (`y_gy`)
* SkyCiv (`skyciv`)
* Zip Archive API (`zip_archive_api`)
* Connectwise PSA (`connectwise_psa`)
* Botpress (`botpress`)
* Orimon (`orimon`)
* Soax (`scrapein_`)
* Twenty (`twenty`)

* Delete components/bytenite/README.md

hallucination

* Delete components/flipando/README.md

* Delete components/navigatr/README.md

* Delete components/orimon/README.md

* Delete components/thoughtly/README.md

incorrect

* Delete components/twenty/README.md

---------

Co-authored-by: Pipedream Component Development <106282845+pipedream-component-development@users.noreply.github.com>
Co-authored-by: Michael Lim <[email protected]>

* ESLint PR check improvement (#11846)

* adding eslintignore extensions

* Adding quiet to suppress non-error warnings

* New Components - timetonic (#11813)

* timetonic init

* new components

* pnpm-lock.yaml

* handle link fields

* upload files

* parse boolean values

* versions & form-data dependency

* pnpm-lock.yaml

* updates

* Slack readme

* Component republishes (#11898)

* Google Sheets version bumps

* Google Drive & Gitlab Dev App bumps

* Zoom Admin version bumps

* Google sheets import fix

* Auto-generated documentation for apps (#11897)

* Auto-generated documentation for apps

* Fly.io (`fly_io`)
* Flipando (`flipando`)
* Orimon (`orimon`)

* Delete components/orimon/README.md

* Delete components/flipando/README.md

---------

Co-authored-by: Pipedream Component Development <106282845+pipedream-component-development@users.noreply.github.com>
Co-authored-by: Michael Lim <[email protected]>

* Updating Perplexity to latest models. (#11904)

* Updating Perplexity to latest models.

* Updating constants.

* Updating package.

* Adding app scaffolding for pro_ledger

* Adding app scaffolding for wiza

* Adding app scaffolding for summit

* Adding app scaffolding for leadiq

* docs: add tap info for brew installation (#11895)

Without tapping it first you get the following error:
```
Warning: 'pipedreamhq/pd-cli/pipedream' formula is unreadable: No available formula with the name "pipedreamhq/pd-cli/pipedream".
Please tap it and then try again: brew tap pipedreamhq/pd-cli
```

* Hubspot - New Event source (#11855)

* update description

* add configuration error

* Google Sheet Sources/Triggers Improvements (#11711)

* add test events

* move constants.mjs to common folder

* update deduping to support longer ids

* update versions

* combine shared & non-shared drive sources

* new-comment source

* add new-comment test-event

* versions

* versions

* Google Sheets Actions Improvements (#11782)

* add doc links

* replace sheetName w/ worksheetId, ConfigurationErrors, updates

* use alert prop for rows description

* combine find-row & find-row-vlookup

* combine delete-row & delete-rows

* versions

* fix version

* bump package version

* parse array inputs

* combine get-values & get-values-in-range

* update action names

* headersDisplay prop

* update cells description

* update clear-row to clear-rows

* versions

* versions

* [Components] Hotmart #8023 (#11626)

* Added actions

* Fixed requested changes

* Fixed requested changes

* Adding app scaffolding for pitchlane

* Adding app scaffolding for appwrite

* Bump actions/checkout from 4.1.4 to 4.1.5 (#11928)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.4 to 4.1.5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4.1.4...v4.1.5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pnpm/action-setup from 3.0.0 to 4.0.0 (#11929)

Bumps [pnpm/action-setup](https://github.com/pnpm/action-setup) from 3.0.0 to 4.0.0.
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](pnpm/action-setup@v3.0.0...v4.0.0)

---
updated-dependencies:
- dependency-name: pnpm/action-setup
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* New Components - twenty (#11823)

* twenty init

* [Components] twenty #11736
Sources
 - New Record (Instant)

Actions
 - Create Update Delete Record

* pnpm update

* some adjusts

* some adjusts

* Adding app scaffolding for pidj

* Adding app scaffolding for keycloak

* Adding app scaffolding for insighto_ai

* Adding app scaffolding for kodagpt

* HIPAA on Enterprise (#11941)

* Update eslintignore to ignore *.mdx (#11948)

* [ACTION] Unsplash - Search Photos, Get Photo (#11939)

* Tested components

* pnpm-lock.yaml

* Added tested components (#11845)

* Tested components (#11893)

* Adding app scaffolding for boloforms

* New Components - chatfly (#11931)

* chatfly init

* [Components] chatfly #11730
Actions
 - SendMessage

* pnpm update

* Update components/chatfly/chatfly.app.mjs

* fix description

---------

Co-authored-by: Leo Vu <[email protected]>

* New Components - thoughtly (#11933)

* thoughtly init

* [Components] thoughtly #11728
Sources
 - New Response (Instant)

Actions
 - Trigger Call
 - Create Contact

* pnpm update

* fix import

* add doc links

---------

Co-authored-by: michelle0927 <[email protected]>

* Adding app partners page to docs and fixing some typos. (#11947)

* Adding app partners page to docs and fixing some typos.

* Update docs-v2/pages/apps/app-partners.mdx

Co-authored-by: Danny Roosevelt <[email protected]>

* Update docs-v2/pages/apps/app-partners.mdx

Co-authored-by: Danny Roosevelt <[email protected]>

---------

Co-authored-by: Danny Roosevelt <[email protected]>

* Adding app scaffolding for upstash_redis

* chore: replace unmaintained jitterbit/get-changed-files@v1 (#11949)

* [Components] leadiq (#11923)

* Tested component

* pnpm-lock.yaml

* fix(types): dedupe unique and greatest strategy needs id (#11945)

* Added actions (#11951)

* New Components - fractel (#11807)

* fractel init

* [Components] fractel #11791
Actions
- Call Phone
- Send SMS/MMS

* pnpm update

* some adjusts

* New Components - platerecognizer (#11944)

* platerecognizer init

* [Components] platerecognizer #11727
Actions
 - Run Recognition

* pnpm update

* [Docs] Data stores atomic operations (#11961)

* add section for data stores atomic operations

* link to integrated apps

* chore: remove temporary continue-on-error fix for changed files (#11959)

* Tested components (#11958)

* [BUG] Postmark - New Inbound Email Received #11950 (#11952)

* [BUG] Postmark - New Inbound Email Received #11950
Sources
 - New Inbound Email Received

* update package.json

* bump version

* fix prop

* New Components - boloforms (#11955)

* boloforms init

* [Components] boloforms #11953
Sources
 - New Form Response (Instant)
 - New Template Response (Instant)
 - New Signature Completed (Instant)

Ations
 - Send Form
 - Send Template For Signature

* pnpm update

* some adjusts

* Tested components (#11921)

* Bump actions/checkout from 4.1.5 to 4.1.6 (#12002)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.5 to 4.1.6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4.1.5...v4.1.6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* ServiceNow docs (#11899)

* ServiceNow readme instructions

* Proofreading pass

* Notion readme (#11900)

* Update README.md (#12006)

* Update README.md (#12007)

* first commit for byoa docs

* more BYOA docs

* arcade embed

* Update oauth-clients.mdx

* Update oauth-clients.mdx

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dylan J. Sather <[email protected]>
Co-authored-by: Luan Cazarine <[email protected]>
Co-authored-by: michelle0927 <[email protected]>
Co-authored-by: Marcelo Jabali <[email protected]>
Co-authored-by: michelle0927 <[email protected]>
Co-authored-by: Dylan Pierce <[email protected]>
Co-authored-by: Guilherme Falcão <[email protected]>
Co-authored-by: Jorge Cortes <[email protected]>
Co-authored-by: Lucas Caresia <[email protected]>
Co-authored-by: Leo Vu <[email protected]>
Co-authored-by: Leo Vu <[email protected]>
Co-authored-by: danhsiung <[email protected]>
Co-authored-by: Michael Lim <[email protected]>
Co-authored-by: js07 <[email protected]>
Co-authored-by: Pipedream Component Development <106282845+pipedream-component-development@users.noreply.github.com>
Co-authored-by: Joscha Feth <[email protected]>
Co-authored-by: Andrew Chuang <[email protected]>
Co-authored-by: Tod Sacerdoti <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triaged For maintainers: This issue has been triaged by a Pipedream employee User submitted Submitted by a user
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants