Refactor: Add plex cache policy patterns #1279

glensc · 2022-12-11T21:21:32Z

Continue #1221, but for plex urls.

git grep --function-context -C2 @nocache

Docs:

https://requests-cache.readthedocs.io/en/stable/user_guide/expiration.html#url-patterns

glensc · 2022-12-11T21:23:43Z

An easy way to find the @nocache patterns is to run full sync and grep logs for:

grep -o 'Skipping cache write for URL: https://.*plex.direct.*' plextraktsync.log

glensc · 2022-12-11T22:19:40Z

calling self.plex.library would make this request:

<!-- request: GET https://*.plex.direct:32400/library, response: 200 (460 bytes), created: 2022-12-11 23:27:11 EET, expires: N/A (fresh) -->
<?xml version='1.0' encoding='utf8'?>
<MediaContainer size="3" allowSync="0" art="/:/resources/library-art.png" content="" identifier="com.plexapp.plugins.library" mediaTagPrefix="/system/bundle/media/flags/" mediaTagVersion="1667296919" title1="Plex Library" title2="">
<Directory key="sections" title="Library Sections" />
<Directory key="recentlyAdded" title="Recently Added Content" />
<Directory key="onDeck" title="On Deck Content" />
</MediaContainer>

this doesn't seem to contain anything useful, so I could be cached forever.

but I'm wondering what is mediaTagVersion="1667296919", plexapi sources say "Unknown" and don't get much matches from google either.

It seems to match timestamp Tue Nov 1 12:01:59 2022 +0200, it doesn't match library update timestamp, nor pms startup time

glensc · 2022-12-11T22:24:11Z

there's a problem, can't set rule for root URL (used for "version" and "updated_at" calls), as it will match everything

        # version, updated_at
        "*/": DO_NOT_CACHE,

from docs:

Patterns will match request base URLs without the protocol, so the pattern site.com/resource/ is equivalent to http*://site.com/resource/**

glensc · 2022-12-11T23:07:05Z

could use filter_fn to filter root url:

https://requests-cache.readthedocs.io/en/stable/user_guide/filtering.html#custom-cache-filtering

simonc56 · 2022-12-15T19:04:11Z

Aren't login urls needed too ?

*.trakt.tv/oauth
plex.tv/api/resources
plex.tv/api/users
...

glensc · 2022-12-16T11:46:35Z

Aren't login urls needed too ?
*.trakt.tv/oauth
plex.tv/api/resources
plex.tv/api/users
...

I haven't finished urls about plex, but sure you can add your commits here
do you have actual issue with trakt URLs, please submit PR to fix.

with trakt pull request, I went every method and looked what requests they make and converted @nocache to rule. so URLs that did not have @nocache before did not get rule added, so they should continue to behave like before.

there's slight possibility if some method under @nocache made multiple requests based on conditions, then those I might have missed.

simonc56 · 2022-12-18T14:59:24Z

plextraktsync/config/HttpCacheConfig.py

+        # fetch_item
+        "*/library/sections/*/all?*": DO_NOT_CACHE,
+        "*/library/sections/*/collections": DO_NOT_CACHE,
+        "*/library/metadata/*": DO_NOT_CACHE,


I think metadata.provider.plex.tv/library/metadata/* should be cached because there is no user data in response.
But metadata.provider.plex.tv/library/metadata/*/userState must remain not cached.

Added dc07f64, but perhaps should have still some cache expiry rather NEVER_EXPIRE? how much?

If we could have a random value, that'd be perfect.
It would allow to spread out the requests.
E.g. 3 month +random(0-30days)

b667c0e, 5f2243b.

but how to test that? enable all watchlist related options and like some online movie?

That's smart, but it's a value for the whole session.
At first run, all requests will have same expiration date. There is no spread out.

If impossible, i'd say NEVER_EXPIRE is good enough for metadata.provider.plex.tv/library/metadata/* .

there's no metadata.provider.plex.tv/library/metadata/*/userState, but rather https://metadata.provider.plex.tv/library/metadata/*?*includeUserState=1 and it includes view counts etc.

but I don't think we use those values, only test for guids

PR to turn off userState:

Allow overriding includeUserState for _toOnlineMetadata pkkid/python-plexapi#1071

Some other places were also uncacheable because session wasn't inherited:

Fix: Inherit session when creating internal PlexServer objects pkkid/python-plexapi#1069

randomize expire request:

Randomize expire requests-cache/requests-cache#768

glensc · 2022-12-20T04:55:00Z

Basically, all methods were manually verified and moved @nocache to URL patterns.

but could be that I didn't test all agent URLs, I'm still using Legacy Agent for shows due performance reasons.

PlexServerConnection.connect remains with @nocache because it's pattern would be "*", making it apply for all urls.

plextraktsync/config/HttpCacheConfig.py

PUT methods are not cached anyway

* Add plex cache policy patterns * Add plex rule for fetch_item * Add cache rule for item history * Use DO_NOT_CACHE constant * Add rule for version, updated_at * Add cache rule for library_sections call * Disable root url rule for now; * Add rule for system_device methpd * Add rule for system_account method * Reorder to account for first match * Add rule for ratings search * Add patterns for plex playlists * Add rules for metadata.provider.plex.tv/library/metadata patterns * Add pattern for find_by_title * Record fetch_items pattern * Record rule for find_by_id * Add LONG_EXPIRY for metadata.provider.plex.tv/library/metadata/* * Add rule for /has_sessions * Add rule for plex.tv/users/account * Add rule for metadata.provider.plex.tv/library/sections/watchlist/all * Update includeUserState pattern * Add rule for len(PlexLibrarySection) * Add pattern for reloads * Add rule for PlexLibrarySection iteration * Add rule for episodes fetch * Add rules for mark played, mark unplayed * Sort items

* Remove nocache for playlists * Remove nocache for has_sessions * Remove nocache for _plex_account * Remove nocache from watchlist * Remove nocache for mark played, mark unplayed * Remove nocache for reset_show

@nocache

* Cleanup @nocache for find_by_title * Cleanup @nocache for find_by_id * Cleanup @nocache for search * Cleanup @nocache for fetch_items * Cleanup @nocache for section length method

* Remove nocache for get uids * Remove nocache for episodes * Remove nocache for rating method

simonc56 · 2022-12-22T14:42:58Z

Heavily tested I assume 👼

glensc · 2022-12-22T14:57:48Z

Heavily tested I assume 👼

It's not git tagged yet. but fingers crossed!

it already solved one issue for me that inspect <season_id> cached the result and episode.media.part.exists leading to wrong cached value.

simonc56 · 2022-12-22T15:10:23Z

fingers crossed

Come on....
Don't tag before tests.

glensc · 2022-12-22T15:12:52Z

what do you mean? I've been testing it and running it for a while. and I haven't made tag, that's what I said, it's not tagged yet.

simonc56 · 2022-12-22T15:24:00Z

I thought you didn't test and only cross finger as a quality process.

glensc · 2022-12-22T17:12:53Z

I wrote how I tested:

Refactor: Add plex cache policy patterns #1279 (comment)

you didn't reply anything to it. so I assumed you were fine with this. and plexapi fixes got released too.

glensc requested a review from simonc56 December 11, 2022 21:21

glensc self-assigned this Dec 11, 2022

glensc force-pushed the cache-patterns-for-plex branch from c29b06c to d130b6c Compare December 11, 2022 22:59

glensc mentioned this pull request Dec 13, 2022

Feature: Add global --no-cache flag to disable trakt cache #1287

Merged

1 task

glensc force-pushed the cache-patterns-for-plex branch from d130b6c to c114a2b Compare December 15, 2022 14:53

simonc56 reviewed Dec 18, 2022

View reviewed changes

glensc force-pushed the cache-patterns-for-plex branch from 41e2fe9 to 29e507a Compare December 19, 2022 14:24

This was referenced Dec 20, 2022

Inspect: Pretty print xml #1296

Merged

Add typing to PlexApi.reset_show #1297

Merged

simonc56 reviewed Dec 20, 2022

View reviewed changes

plextraktsync/config/HttpCacheConfig.py Show resolved Hide resolved

glensc force-pushed the cache-patterns-for-plex branch from a47279c to c5f0580 Compare December 20, 2022 12:43

glensc added 8 commits December 22, 2022 15:44

PlexLibrarySection: Add __iter__ to PlexLibrarySection

f29a04a

PlexApi: Pass session property for MyPlexAccount

5cc7915

Remove nocache decorator for PUT methods

3d7d8dd

PUT methods are not cached anyway

HttpCacheConfig: Add LONG_EXPIRY constant

7730522

Cleanup PlexApi nocache decorators

c599fc4

* Remove nocache for playlists * Remove nocache for has_sessions * Remove nocache for _plex_account * Remove nocache from watchlist * Remove nocache for mark played, mark unplayed * Remove nocache for reset_show

Cleanup PlexLibrarySection nocache decorators

15803de

* Cleanup @nocache for find_by_title * Cleanup @nocache for find_by_id * Cleanup @nocache for search * Cleanup @nocache for fetch_items * Cleanup @nocache for section length method

Cleanup PlexLibraryItem nocache decorators

6ebd0b7

* Remove nocache for get uids * Remove nocache for episodes * Remove nocache for rating method

glensc force-pushed the cache-patterns-for-plex branch from c5f0580 to 6ebd0b7 Compare December 22, 2022 13:44

glensc marked this pull request as ready for review December 22, 2022 13:45

glensc merged commit 71a1e3a into Taxel:main Dec 22, 2022

glensc deleted the cache-patterns-for-plex branch December 22, 2022 13:46

simonc56 added the testing-needed label Dec 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: Add plex cache policy patterns #1279

Refactor: Add plex cache policy patterns #1279

glensc commented Dec 11, 2022 •

edited

Loading

glensc commented Dec 11, 2022

glensc commented Dec 11, 2022

glensc commented Dec 11, 2022 •

edited

Loading

glensc commented Dec 11, 2022

simonc56 commented Dec 15, 2022

glensc commented Dec 16, 2022

simonc56 Dec 18, 2022

glensc Dec 19, 2022

simonc56 Dec 19, 2022

glensc Dec 19, 2022

simonc56 Dec 19, 2022 •

edited

Loading

glensc Dec 20, 2022

glensc Dec 20, 2022

glensc Dec 20, 2022

glensc Jan 20, 2023

glensc commented Dec 20, 2022

simonc56 commented Dec 22, 2022

glensc commented Dec 22, 2022

simonc56 commented Dec 22, 2022

glensc commented Dec 22, 2022

simonc56 commented Dec 22, 2022

glensc commented Dec 22, 2022

Refactor: Add plex cache policy patterns #1279

Refactor: Add plex cache policy patterns #1279

Conversation

glensc commented Dec 11, 2022 • edited Loading

glensc commented Dec 11, 2022

glensc commented Dec 11, 2022

glensc commented Dec 11, 2022 • edited Loading

glensc commented Dec 11, 2022

simonc56 commented Dec 15, 2022

glensc commented Dec 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonc56 Dec 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glensc commented Dec 20, 2022

simonc56 commented Dec 22, 2022

glensc commented Dec 22, 2022

simonc56 commented Dec 22, 2022

glensc commented Dec 22, 2022

simonc56 commented Dec 22, 2022

glensc commented Dec 22, 2022

glensc commented Dec 11, 2022 •

edited

Loading

glensc commented Dec 11, 2022 •

edited

Loading

simonc56 Dec 19, 2022 •

edited

Loading