MTNI-281 ⁃ Store only one page object per URL

[blackforestboi]

Gerben provided a nice deduplication framework, where we can flexibly define the rules on when and if to store duplicates.

I may be wrong about that, but It seems to me that we currently won't need to store a new object for every visit, as long as there is no versioning.
It also would make it simpler if we just had one version per page and could index all of them, instead of worrying which versions to index. We may want to store and search multiple versions later though.

---

Want to back this issue? ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Post a bounty on it! We accept bounties via ?utm_campaign=plugin&utm_content=tracker%2F59103681&utm_medium=issues&utm_source=github Bountysource.

[blackforestboi]

New thoughts:

• For now, we should only store one page object per url - the most recent. And add all other visits as visit objects.

Expected behaviour:
Whenever a page is visited/imported, that is already existent: replace old version of text/content AND freeze-dry with the most recent.
The freeze dry versions can easily get very big. one page can have anything between 100kb-2MB
So we should reduce the amount of freeze dry as much as possible.

Later we might also think of adding some more metrics to the decision to keep a freeze dry version in the first place. Maybe something related to how long a person stays on an article, or only when a user choses to store the freeze dry version manually.

[blackforestboi]

fixed with #69