[Reader] Pagination for unfollowed tags doesn't work properly

[thomashorta]

Expected behavior

When opening an unfollowed tag/topic feed (by tapping a tag in a post / discover suggested tags card) and reaching the end of the list, it should load the results from newer to older, fetching more posts and adding to the end of the list. It should also fetch all posts eventually if the user keeps scrolling.

Actual behavior

When opening an unfollowed tag/topic feed (by tapping a tag in a post / discover suggested tags card)and reaching the end of the list, it fetches some posts and put them in the middle of list, sometimes also at the end, and it sometimes stops loading more posts entirely.

Steps to reproduce the behavior

1. Open the Reader on Jetpack App
2. Open a tag you do not follow yet (from some post or the discover tag suggestions card) - prefer very generic and widely used tags like love, science, education.
3. Scroll to the end of the list so it loads more posts
4. Keep doing it for a while
5. Notice that sometimes posts are loaded above the last post (shifting content around) and eventually posts stop loading.

Tested on OnePlus 5T, Android 10, JPAndroid / WPAndroid [version]

Some initial investigation

It looks like the code that loads more posts from the endpoint uses different date/score fields for the before field. This function retrieves the local database field that will be used to set the before param in the read/tags endpoint.

For followed tags, since the ReaderTag.mEndpoint field is set, the code uses date_tagged, but if it’s not followed, it uses the date_published (because of the tag.isTagTopic logic).

When posts are no longer loading it is likely because we used some reference date for before which is too recent and doesn't match the actual date (used by the endpoint) of the last item on the shown list, so we only receive items that are already in the list.

Ideally, we should take a look at the /rest/v1.2/read/tags/<tag-slug>/posts endpoint code as well because we are sending different reference dates in the before for different cases which will likely never work properly since before is probably always using a specific column, leading to the pagination issues we see.