Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

standard key expiration for all supported drivers #262

Closed
4 of 9 tasks
Hebilicious opened this issue Jul 5, 2023 · 3 comments
Closed
4 of 9 tasks

standard key expiration for all supported drivers #262

Hebilicious opened this issue Jul 5, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@Hebilicious
Copy link
Member

Hebilicious commented Jul 5, 2023

Describe the feature

Implement a ttl top level option for setting a ttl in seconds at the object level for all drivers.

await setItem(key, item, { ttl: 30 })

Additional information

  • Github driver supports global a global ttl option that isn't related to expiration
  • Redis driver supports global and setItem ttl options that are related to expiration

For unsupported / partial support drivers, a manual expiration can be implemented by attaching custom unstorage metadata to the storage itself.

Unsupported drivers

  • Azure App configuration
  • Azure Storage Table
  • fs
  • github
  • http (altough we could forward ttl to cache-control max age I don't think it makes sense)
  • planetscale
  • localstorage
  • session-storage
  • memory (We could implement with setTimeout)

Partial support drivers

  • R2: This can be set at the bucket level, but not at the object level
  • mongodb: requires extra indexes

TTL driver native support

  1. enhancement ttl
  2. enhancement good first issue
  3. cloudflare
  4. driver
  • Would you be willing to help implement this feature?
@Hebilicious Hebilicious added the enhancement New feature or request label Jul 5, 2023
@Hebilicious Hebilicious changed the title [feature] implement generic key expiration method [feature] implement standard key expiration syntax Jul 5, 2023
@Hebilicious Hebilicious changed the title [feature] implement standard key expiration syntax [feature] implement standard key expiration for all drivers Jul 16, 2023
@Hebilicious Hebilicious changed the title [feature] implement standard key expiration for all drivers standard key expiration for all drivers Jul 16, 2023
@Hebilicious Hebilicious pinned this issue Jul 16, 2023
@itpropro
Copy link
Member

itpropro commented Sep 22, 2023

I will implement this for the Azure Key Vault and the CosmosDB driver, but Azure Storage Table/Blob and Azure App Config are definitely unsupported natively. I would also consider mongoDb as unsupported, as creating ttl indexes has a lot of implications and edge cases to be considered.
For the general approach, I am not really sure what the benefit is for having the ttl in the metadata treated in another way than normal metadata. I would prefer to just support ttl in a standard way where it is natively supported and leave the rest to custom metadata fields defined by the user.
Otherwise we would have to create our own deletion mechanism that is triggered when a item is pulled from the source system and then checks if the ttl is expired. If yes, it would delete the item and behave like it didn't exist, if ttl is not expired, it would return it.
What is your input on this @pi0? I would be happy to draft a general ttl implementation for unsupported drivers, if you think it's in the scope of unstorage.

@Hebilicious
Copy link
Member Author

Hebilicious commented Oct 9, 2023

I will implement this for the Azure Key Vault and the CosmosDB driver, but Azure Storage Table/Blob and Azure App Config are definitely unsupported natively. I would also consider mongoDb as unsupported, as creating ttl indexes has a lot of implications and edge cases to be considered. For the general approach, I am not really sure what the benefit is for having the ttl in the metadata treated in another way than normal metadata. I would prefer to just support ttl in a standard way where it is natively supported and leave the rest to custom metadata fields defined by the user. Otherwise we would have to create our own deletion mechanism that is triggered when a item is pulled from the source system and then checks if the ttl is expired. If yes, it would delete the item and behave like it didn't exist, if ttl is not expired, it would return it. What is your input on this @pi0? I would be happy to draft a general ttl implementation for unsupported drivers, if you think it's in the scope of unstorage.

I agree with you, I listed both mongodb and r2 under possible but not straighforward, but maybe my wording is not strong enough, I'll rename this to partial support. In both cases it would be pretty hacky or as you said a custom deletion mechanism.
Regarding the general approach, this PR is to harmonize the drivers that support native TTL, ie to avoid contributors using a different key (exp, settings.ttl ...) or unit.
@itpropro Do you think Azure T/B and Azure AC are in unsupported, or in partial support ?

@Hebilicious Hebilicious changed the title standard key expiration for all drivers standard key expiration for all supported drivers Oct 9, 2023
@pi0
Copy link
Member

pi0 commented Oct 9, 2023

At this point i think there is really no point of having such tracker. There is no new shared feature to implement other than adding types for ttl? as optional param. PR welcome to add (native) ttl option to any driver can accept it.


Regarding the general approach, this PR is to harmonize the drivers that support native TTL, ie to avoid contributors using a different key (exp, settings.ttl ...) or unit.

I will make sure of that before merging any PRs if they used another key :)

@pi0 pi0 closed this as completed Oct 9, 2023
@pi0 pi0 unpinned this issue Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants