persistent cache

[pi0]

Support abstract persistent cache for both input source and converted images. (implies supporting ttl)

Related: #46

[btkostner]

For this issue, I would assume the best place to put the cache would be at this function https://github.com/unjs/ipx/blob/main/src/ipx.ts#L72.

• Add some sort of maxSize option to https://github.com/unjs/ipx/blob/main/src/ipx.ts#L45
• Check if we should reevaluate the source from https://github.com/unjs/ipx/blob/main/src/ipx.ts#L96 by mtime and maxAge
• If we don't need to reevaluate, create a hash from all of the image modification options
• Check if the file exists in the cache, and if so return the buffer of it
• Otherwise create the image, save it to the cache, and return the buffer

If nobody is working on this, I might be able to take a stab at it.

[ohbob]

were you able to do the cache?

[btkostner]

@ohbob I did not get to this, but it looks like @ascorbic did. Look at the attached draft PR for progress.

[multipliedtwice]

Hello, thank you for an awesome library. Do you plan to release the feature?

[guillaumeprevost]

Hi ! Any news on this feature ?

[AbdallahAlhaddad]

Hi @pi0
Any update on this feature?

[bernhardberger]

Is there any timeframe on this? If it's something that will happen within Q1/2024 we could possibly hold out switching away from IPX in our projects but it would be good to know for planning how far the roadmap this is.

[pi0]

@bernhardberger It is on my plan. BTW curious how is your setup is it on top of nuxt/niro/h3?

[bernhardberger]

tbh, no clue - the node process is running supervised by our hosting partner and the nitro server more or less directly exposed to the outside world. There's no reverse proxy involved (that we could have control over) that would do the caching. Our clients and also the higher ups internally want to avoid cloud based solutions at all cost, also because it has a lot of implications regarding compliance etc in the EU with our larger corporate customers.

We're trying to move away from monolithic solutions as it would open up more possibilities to share frontend code between different projects and lessen the need to have frontend devs that have backend specific framework knowledge (Laravel, TYPO3, WordPress etc.). Our projects are mostly brand/marketing stuff (MPA, Content driven websites and portals) so going headless is mostly a decision driven by workflow and less by having to have the ability to scale (in which case we'd do SSG). But estimating about 20-30 projects yearly IPX in it's current form is a deal breaker when we had like 10-20 projects per server and now we'd be stuck at like 2 before resources run out 🙈

//Edit: I misread. FE stack is Nuxt3 + some Nitro middlewares for routing+ h3-compression to make Lighthouse stop complaining. SEO stuff unfortunately is the highest priority.

PPS: IPX doesn't work properly with SWR in Nuxt3. Delivers Bytestream json instead of images. ISR works.

[andrevandal]

hey, any update?

maybe persist images like aws-samples/image-optimization could be a good improvement.

I'm already using Cloudflare cache to prevent my cpu meltdown, but it's not the ideal. :S

if we could store the generated images an use them instead of creating a new one, it should be amazing.

[bernhardberger]

this is still a major issue preventing us from running SWR on our site..

[lesion]

I solved with a middleware implementing this:

https://0xacab.org/rbo/site/-/blob/no-masters/server-middleware/ipx.js?ref_type=heads

hope it helps

[bloor]

Thank god for this thread. Otherwise I would have lost another 6weeks to realize my IPX Server has no actual physical storage of the generated files. You restart/rebuild server and everything is gone.
I cannot fathom how was THIS not a day1 feature. My head hurts. Back to spending $10 on bunnycdn for PROPER image optimization.

PS: also a big issue with current method is cloudflare cannot properly cache these "dynamic optimized urls".

[bernhardberger]

Thank god for this thread. Otherwise I would have lost another 6weeks to realize my IPX Server has no actual physical storage of the generated files. You restart/rebuild server and everything is gone.
I cannot fathom how was THIS not a day1 feature. My head hurts. Back to spending $10 on bunnycdn for PROPER image optimization.

PS: also a big issue with current method is cloudflare cannot properly cache these "dynamic optimized urls".

I guess because in 2024 noone seems to care about self hosting anymore.. except a few poor souls like me/us that have to work in environments where cloud hosting is not an option..

PS: Yes, that's another huge problem that the URLs aren't using unique hashes, so you can't really cache those images "forever." So you can't even properly use a CDN, caching proxy or even cache headers for those images..

Right now, I had to turn away from using IPX and had to implement image processing through server components using either flyimg or custom TYPO3/Statamic/Glide endpoints and hashed, secure URLs to prevent mass resize attacks and get our server resources back to normal..

[andrevandal]

hey guys, the one way was building it my self.
it's not hard, and you can implement your way of persisting cache.

I've built my own saving cache into s3 bucket and using varnish cache in front of the service.
I hope it helps someone: https://github.com/andrevandal/ipx-server

[scherii]

Hi @andrevandal, I think your repository is set to private. 🙃