Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(RSS Feed Trigger Node): Save last item's date instead of last execution date #8572

Merged
merged 1 commit into from
Feb 8, 2024

Conversation

sweenu
Copy link
Contributor

@sweenu sweenu commented Feb 7, 2024

Summary

Some RSS feed won't update exactly at the time show on the last item. In order to have a more reliable node, we stop checking for the last time we polled the feed to know which items are new, but instead we store and check the last item's publication date.

Related tickets and issues

Related to #8473.

Review / Merge checklist

  • PR title and summary are descriptive. Remember, the title automatically goes into the changelog. Use (no-changelog) otherwise. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.

    A bug is not considered fixed, unless a test is added to prevent it from happening again.
    A feature is not complete without tests.

I saw no tests were present for this node. I didn't add one. I did test the behavior locally, though.

@CLAassistant
Copy link

CLAassistant commented Feb 7, 2024

CLA assistant check
All committers have signed the CLA.

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2024

@sweenu that looks like it is it, Have you given it a test? It may be worth having something in there to set the lastItemDate to the lastRun if it exists as well so we don't make things worse for existing users.

@sweenu
Copy link
Contributor Author

sweenu commented Feb 7, 2024

Locally, yes, it works as expected. Okay, let me make the modifications.

@n8n-assistant n8n-assistant bot added community Authored by a community member node/improvement New feature or request labels Feb 7, 2024
@sweenu
Copy link
Contributor Author

sweenu commented Feb 7, 2024

@Joffcom Do you still want to populate pollData.lastTimeChecked or do we just check it to make the transition?

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2024

@sweenu I think we should just use it once to set the new option if it is not empty and if the workflow is not being ran manually. Then we can delete it and we should be all good.

@sweenu
Copy link
Contributor Author

sweenu commented Feb 7, 2024

Done. I tested again with the same workflow and also a brand new one.
I should also probably change the name of startDate. How about dateToCheck?

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2024

That looks good, I am just creating a docker image and I will leave it running overnight to see if anything odd happens

@sweenu sweenu force-pushed the master branch 2 times, most recently from 57eab7d to 064100b Compare February 7, 2024 15:57
@sweenu
Copy link
Contributor Author

sweenu commented Feb 7, 2024

I've got no clue what the problem is with the linting, I might need your help:
45:22 error Replace `·` with `⏎↹↹↹` prettier/prettier

I ran pnpm lint -- --fix but it hasn't done anything.

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2024

@sweenu it wants to you to change it to

const dateToCheck =
	(pollData.lastItemDate as string) || (pollData.lastTimeChecked as string) || now;

I would push the change but you used the master branch instead of making a new one.

@sweenu
Copy link
Contributor Author

sweenu commented Feb 7, 2024

Sorry, bad habit. Thanks for your help. Pushed the fix.

@Joffcom
Copy link
Member

Joffcom commented Feb 7, 2024

The test image seems to be running well so far I have had a couple of posts from Hackernews come through and none of the old events have fired off. I think we will be able to get this merged tomorrow, Fantastic work 🙌🏻

@Joffcom
Copy link
Member

Joffcom commented Feb 8, 2024

Hey @sweenu,

Testing overnight has looked very good and no issues so I think we can get this merged and make this node better for everyone.

@Joffcom Joffcom merged commit a822588 into n8n-io:master Feb 8, 2024
8 checks passed
andyjoyous added a commit to eyaljoyous/n8n that referenced this pull request Feb 9, 2024
* refactor(core): Replace promisify-d node calls with native promises (no-changelog) (n8n-io#8464)

* fix(editor): Send template id as a number in telemetry events (n8n-io#8484)

* refactor(editor): Prevent router.replace from computed property (no-changelog) (n8n-io#8489)

Signed-off-by: Oleg Ivaniv <[email protected]>

* feat(core): Remove `own` execution-process mode (n8n-io#8490)

* feat(editor): Implement loading and error states for dynamically loaded components in node parameter list (n8n-io#8477)

* fix(AwsS3 Node): Fix handling of bucket with dot in name (n8n-io#8475)

* refactor(core): Modernize credentials controllers and services (no-changelog) (n8n-io#8488)

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>

* fix(core): Don't report executions that have been paused as failed to rudderstack and log streams (n8n-io#8501)

* fix: Properly iterate over credentials with expressions (n8n-io#8502)

* refactor(core): Move all code related to `onServerStarted` into `InternalHooks` (no-changelog) (n8n-io#8500)

* feat(editor): Send template id as string in all telemetry events (n8n-io#8498)

* fix(core): Handle possibly invalid `updatedAt` timestamps in source-control (n8n-io#8485)

* fix(core): Forward authorization header when on same domain (n8n-io#8507)

* fix(core): Improve handling of wrapped errors (n8n-io#8510)

* 🚀 Release 1.27.0 (n8n-io#8512)

Co-authored-by: ivov <[email protected]>

* ci: Skip running `postInstall` scripts for all packages except sqlite3 (no-changelog) (n8n-io#8514)

* ci: Fix DB tests (no-changelog) (n8n-io#8513)

* fix(core): Fix new graceful shutdown env being always overridden by deprecated env (n8n-io#8503)

* fix(HTTP Request Node): Support form data when using pagination (n8n-io#8497)

* fix(HTTP Request Node): Require parameter with filled name and value to avoid infinite loop (n8n-io#8454)

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Elias Meire <[email protected]>

* fix(Microsoft Excel 365 Node): Upsert append new rows at the end of used range, option to append at the end of selected range (n8n-io#8461)

* fix(Slack Node): Attachments fix (n8n-io#8471)

Co-authored-by: Elias Meire <[email protected]>

* feat: Azure Open AI chat model & embeddings (n8n-io#8522)

Signed-off-by: Oleg Ivaniv <[email protected]>

* fix(core): Fix test runs of triggers that rely on static data (n8n-io#8524)

* ci: Replace `pnpm/action-setup` action with corepack (no-changelog) (n8n-io#8504)

* fix(MongoDB Node): Fix "Maximum call stack size exceeded" error on too many rows (n8n-io#8530)

* fix: Allow Date/Luxon objects and additional formats in DateTime validation (n8n-io#8525)

* fix(editor): Prune values that are not in the schema in the ResourceMapper component (n8n-io#8478)

* fix(core): Fix PermissionChecker.check, and add additional unit tests (n8n-io#8528)

* fix(core): Fix DropRoleMapping migration (n8n-io#8521)

* fix(core): Ensure AxiosError status always gets copied over to NodeApiError (n8n-io#8509)

* fix(Embeddings OpenAI Node): Fix dynamic models fetching (n8n-io#8533)

Signed-off-by: Oleg Ivaniv <[email protected]>

* fix(core): Fix workflow tagging failure due to unique constraint check (n8n-io#8505)

* ci: Fix release-create-pr.yml (no-changelog)

* fix(core): Upgrade nodemailer to address an exploit (n8n-io#8535)

* feat(RabbitMQ Trigger Node): Add options to configure assert of exchanges and queues (n8n-io#8430)

* feat(editor): Add delete and disable button to nodes on hover (n8n-io#8482)

* feat(Email Trigger (IMAP) Node): Upgrade mailparser (n8n-io#8539)

* refactor(core): Streamline flows in multi-main mode (no-changelog) (n8n-io#8446)

* fix: Remove ts-node from overrides and typeorm script (no-changelog) (n8n-io#8547)

* fix: Update BaseChatModel import checks for MistralAI compatibility (n8n-io#8527)

Signed-off-by: Oleg Ivaniv <[email protected]>

* docs: Add encryption key check to breaking changes list (n8n-io#8551)

* refactor(core): Lock `webhook` process out of multi-main setup (no-changelog) (n8n-io#8549)

* refactor(core): Continue breaking dependency cycles (no-changelog) (n8n-io#8545)

* fix(core): Use trx manager instead of repository for tags overwrite (n8n-io#8557)

* build: Fix outdated import to fix build (no-changelog) (n8n-io#8558)

* refactor(core): Couple of refactors on WorkflowRunner and ActiveExecutions (no-changelog) (n8n-io#8487)

* feat: Add assignment component with drag and drop to Set node (n8n-io#8283)

Co-authored-by: Giulio Andreini <[email protected]>

* ci: Update validate-n8n-pull-request-title action (no-changelog) (n8n-io#8553)

* fix(core): Use hostname from URL instead of Host header for SNI (n8n-io#8562)

* fix(Microsoft Outlook Node): Download executes more than once per incoming item (n8n-io#8566)

* 🚀 Release 1.28.0 (n8n-io#8569)

Co-authored-by: ivov <[email protected]>

* ci(core): Avoid slow bcrypt calls in tests (no-changelog) (n8n-io#8570)

* fix(core): Upgrade rudderstack sdk to address npm postInstall issues (n8n-io#8568)

* feat: Upgrade typeorm, sqlite3, and pg/pg-promise (n8n-io#8579)

* build: Add GitHub issue form for reporting bugs (n8n-io#8585)

* fix(Google Sheets Trigger Node): First non-header row is ignored when using on row added event (n8n-io#8580)

* fix(RSS Feed Trigger Node): Save last item's date instead of last execution date (n8n-io#8572)

* feat(core): Migrate to n8n's typeorm fork (n8n-io#8590)

* refactor: Add lint rule for unsafe property access with lodash get/set (no-changelog) (n8n-io#8587)

* fix(HTTP Request Node): Errorneous binary object without content-disposition response header (n8n-io#8583)

Co-authored-by: Marcus <[email protected]>

* fix(core): Custom workflow tool tweaks  (n8n-io#8561)

* Fixes imports and enables workers view

---------

Signed-off-by: Oleg Ivaniv <[email protected]>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Milorad FIlipović <[email protected]>
Co-authored-by: oleg <[email protected]>
Co-authored-by: Iván Ovejero <[email protected]>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Danny Martini <[email protected]>
Co-authored-by: Omar Ajoue <[email protected]>
Co-authored-by: Elias Meire <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: ivov <[email protected]>
Co-authored-by: Michael Kret <[email protected]>
Co-authored-by: Andrea Ascari <[email protected]>
Co-authored-by: Giulio Andreini <[email protected]>
Co-authored-by: Cornelius Suermann <[email protected]>
Co-authored-by: Bruno Inec <[email protected]>
Co-authored-by: Marcus <[email protected]>
andyjoyous added a commit to eyaljoyous/n8n that referenced this pull request Feb 10, 2024
* Andy/joy 277 mergeupdate n8n 128 (#54)

* refactor(core): Replace promisify-d node calls with native promises (no-changelog) (n8n-io#8464)

* fix(editor): Send template id as a number in telemetry events (n8n-io#8484)

* refactor(editor): Prevent router.replace from computed property (no-changelog) (n8n-io#8489)

Signed-off-by: Oleg Ivaniv <[email protected]>

* feat(core): Remove `own` execution-process mode (n8n-io#8490)

* feat(editor): Implement loading and error states for dynamically loaded components in node parameter list (n8n-io#8477)

* fix(AwsS3 Node): Fix handling of bucket with dot in name (n8n-io#8475)

* refactor(core): Modernize credentials controllers and services (no-changelog) (n8n-io#8488)

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>

* fix(core): Don't report executions that have been paused as failed to rudderstack and log streams (n8n-io#8501)

* fix: Properly iterate over credentials with expressions (n8n-io#8502)

* refactor(core): Move all code related to `onServerStarted` into `InternalHooks` (no-changelog) (n8n-io#8500)

* feat(editor): Send template id as string in all telemetry events (n8n-io#8498)

* fix(core): Handle possibly invalid `updatedAt` timestamps in source-control (n8n-io#8485)

* fix(core): Forward authorization header when on same domain (n8n-io#8507)

* fix(core): Improve handling of wrapped errors (n8n-io#8510)

* 🚀 Release 1.27.0 (n8n-io#8512)

Co-authored-by: ivov <[email protected]>

* ci: Skip running `postInstall` scripts for all packages except sqlite3 (no-changelog) (n8n-io#8514)

* ci: Fix DB tests (no-changelog) (n8n-io#8513)

* fix(core): Fix new graceful shutdown env being always overridden by deprecated env (n8n-io#8503)

* fix(HTTP Request Node): Support form data when using pagination (n8n-io#8497)

* fix(HTTP Request Node): Require parameter with filled name and value to avoid infinite loop (n8n-io#8454)

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Elias Meire <[email protected]>

* fix(Microsoft Excel 365 Node): Upsert append new rows at the end of used range, option to append at the end of selected range (n8n-io#8461)

* fix(Slack Node): Attachments fix (n8n-io#8471)

Co-authored-by: Elias Meire <[email protected]>

* feat: Azure Open AI chat model & embeddings (n8n-io#8522)

Signed-off-by: Oleg Ivaniv <[email protected]>

* fix(core): Fix test runs of triggers that rely on static data (n8n-io#8524)

* ci: Replace `pnpm/action-setup` action with corepack (no-changelog) (n8n-io#8504)

* fix(MongoDB Node): Fix "Maximum call stack size exceeded" error on too many rows (n8n-io#8530)

* fix: Allow Date/Luxon objects and additional formats in DateTime validation (n8n-io#8525)

* fix(editor): Prune values that are not in the schema in the ResourceMapper component (n8n-io#8478)

* fix(core): Fix PermissionChecker.check, and add additional unit tests (n8n-io#8528)

* fix(core): Fix DropRoleMapping migration (n8n-io#8521)

* fix(core): Ensure AxiosError status always gets copied over to NodeApiError (n8n-io#8509)

* fix(Embeddings OpenAI Node): Fix dynamic models fetching (n8n-io#8533)

Signed-off-by: Oleg Ivaniv <[email protected]>

* fix(core): Fix workflow tagging failure due to unique constraint check (n8n-io#8505)

* ci: Fix release-create-pr.yml (no-changelog)

* fix(core): Upgrade nodemailer to address an exploit (n8n-io#8535)

* feat(RabbitMQ Trigger Node): Add options to configure assert of exchanges and queues (n8n-io#8430)

* feat(editor): Add delete and disable button to nodes on hover (n8n-io#8482)

* feat(Email Trigger (IMAP) Node): Upgrade mailparser (n8n-io#8539)

* refactor(core): Streamline flows in multi-main mode (no-changelog) (n8n-io#8446)

* fix: Remove ts-node from overrides and typeorm script (no-changelog) (n8n-io#8547)

* fix: Update BaseChatModel import checks for MistralAI compatibility (n8n-io#8527)

Signed-off-by: Oleg Ivaniv <[email protected]>

* docs: Add encryption key check to breaking changes list (n8n-io#8551)

* refactor(core): Lock `webhook` process out of multi-main setup (no-changelog) (n8n-io#8549)

* refactor(core): Continue breaking dependency cycles (no-changelog) (n8n-io#8545)

* fix(core): Use trx manager instead of repository for tags overwrite (n8n-io#8557)

* build: Fix outdated import to fix build (no-changelog) (n8n-io#8558)

* refactor(core): Couple of refactors on WorkflowRunner and ActiveExecutions (no-changelog) (n8n-io#8487)

* feat: Add assignment component with drag and drop to Set node (n8n-io#8283)

Co-authored-by: Giulio Andreini <[email protected]>

* ci: Update validate-n8n-pull-request-title action (no-changelog) (n8n-io#8553)

* fix(core): Use hostname from URL instead of Host header for SNI (n8n-io#8562)

* fix(Microsoft Outlook Node): Download executes more than once per incoming item (n8n-io#8566)

* 🚀 Release 1.28.0 (n8n-io#8569)

Co-authored-by: ivov <[email protected]>

* ci(core): Avoid slow bcrypt calls in tests (no-changelog) (n8n-io#8570)

* fix(core): Upgrade rudderstack sdk to address npm postInstall issues (n8n-io#8568)

* feat: Upgrade typeorm, sqlite3, and pg/pg-promise (n8n-io#8579)

* build: Add GitHub issue form for reporting bugs (n8n-io#8585)

* fix(Google Sheets Trigger Node): First non-header row is ignored when using on row added event (n8n-io#8580)

* fix(RSS Feed Trigger Node): Save last item's date instead of last execution date (n8n-io#8572)

* feat(core): Migrate to n8n's typeorm fork (n8n-io#8590)

* refactor: Add lint rule for unsafe property access with lodash get/set (no-changelog) (n8n-io#8587)

* fix(HTTP Request Node): Errorneous binary object without content-disposition response header (n8n-io#8583)

Co-authored-by: Marcus <[email protected]>

* fix(core): Custom workflow tool tweaks  (n8n-io#8561)

* Fixes imports and enables workers view

---------

Signed-off-by: Oleg Ivaniv <[email protected]>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Milorad FIlipović <[email protected]>
Co-authored-by: oleg <[email protected]>
Co-authored-by: Iván Ovejero <[email protected]>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Danny Martini <[email protected]>
Co-authored-by: Omar Ajoue <[email protected]>
Co-authored-by: Elias Meire <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: ivov <[email protected]>
Co-authored-by: Michael Kret <[email protected]>
Co-authored-by: Andrea Ascari <[email protected]>
Co-authored-by: Giulio Andreini <[email protected]>
Co-authored-by: Cornelius Suermann <[email protected]>
Co-authored-by: Bruno Inec <[email protected]>
Co-authored-by: Marcus <[email protected]>

* Enables advanced permissions

* Fixes workflow statistics update errors

---------

Signed-off-by: Oleg Ivaniv <[email protected]>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Milorad FIlipović <[email protected]>
Co-authored-by: oleg <[email protected]>
Co-authored-by: Iván Ovejero <[email protected]>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <[email protected]>
Co-authored-by: Danny Martini <[email protected]>
Co-authored-by: Omar Ajoue <[email protected]>
Co-authored-by: Elias Meire <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: ivov <[email protected]>
Co-authored-by: Michael Kret <[email protected]>
Co-authored-by: Andrea Ascari <[email protected]>
Co-authored-by: Giulio Andreini <[email protected]>
Co-authored-by: Cornelius Suermann <[email protected]>
Co-authored-by: Bruno Inec <[email protected]>
Co-authored-by: Marcus <[email protected]>
@github-actions github-actions bot mentioned this pull request Feb 15, 2024
@janober
Copy link
Member

janober commented Feb 15, 2024

Got released with [email protected]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community Authored by a community member node/improvement New feature or request Released
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants