TST Sync Error (Insufficient Storage)

[irvinm]

Short description

Firefox Sync looks to be trying to sync a TST object that is larger than the allowed 16384 Bytes.

Steps to reproduce

Not sure yet. It appears to be dependent on the number of tabs, but it could also be related to a TST helper addon which creates a new state for the tab. (#2104) I had a larger session of tabs (around 750) and started seeing this error. Once I closed some tabs (still 700+ tabs) this error went away. From the sync log error, the object being synced was only a little over the maximum.

Is there anything you can think of that I can explore to determine what information is being synchronized or if there is anything I can do? I have had 1000+ tabs in the past and don't recall hitting this limitation.

Expected result

No sync errors.

Actual result

Entire error sync log:

1593440190017	Sync.Engine.Extension-Storage	ERROR	Syncing [email protected]: request failed: Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded " "(16443 > 16384 Bytes.)(resource://services-common/kinto-http-client.js:454:13) JS Stack trace: [email protected]:454:13
[email protected]:571:23
1593440190018	Sync.ErrorHandler	ERROR	Some engines did not sync correctly.

Environment

• Platform (OS): Windows 10 x64 (1909)
• Version of Firefox: FF 77.0.1
• Version (or revision) of Tree Style Tab: TST 3.5.5

[piroor]

The helper addon looks to be unrelated to the problem. The error you saw is returned by the sync server hosted by Mozilla, and it means that TST stores too large data via browser.storage.sync. If you use very long user-defined styles, it may exceeds the maximum storage size for sync.

[piroor]

The error message you saw says that it exceeds 16KB, but it is mysterious for me why the sync server returns the error, because the API document https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/sync says that the maximum data size for sync is 100KB...

[piroor]

I've introduced some changes to drop some temporary config data from sync keys. These changes should reduce such a sync error...maybe.

[irvinm]

@piroor with my high level review of the issue, it looks like it is 100k for the total storage and 8k for "The maximum size (in bytes) of each individual item in sync storage, as measured by the JSON stringification of its value plus its key length." (link)

I also saw comments that Firefox based their specs on Chrome, but increased the size from 8k to 16k as they store it base64.

In my case, it looks like it isn't the total size that is exceeded, but just something at the individual item storage level.

Obviously the size is close to the limit, so maybe dropping the temporary config data might be enough.

[irvinm]

@piroor I know what is causing my issue. It is my TST CSS. With it in, I must be over the individual item limit, but when I clear it out sync works properly. I will try clearing out my documentation\comments in my CSS to see if that gets me below the threshold as I do have a fair amount of information as to where I got some of the customizations. MS Word tells me there are 10,101 characters (no spaces) and 11,441 characters (with spaces) ... the helper addon also adds some additional CSS.

It does look like Mozilla will likely add some client side checking at some point around sync quotas. (link)

[piroor]

Thank you for the research!
Firefox's browser.storage.sync doesn't have these constant properties, but the implementation looks to know those max sizes of data internally.
https://searchfox.org/mozilla-central/rev/31d8600b73dc85b4cdbabf45ac3f1a9c11700d8e/toolkit/components/extensions/schemas/storage.json#318
https://searchfox.org/mozilla-central/rev/31d8600b73dc85b4cdbabf45ac3f1a9c11700d8e/toolkit/components/extensions/test/xpcshell/head_storage.js#820
Those constants were simply imported from Chrome's definitions and actual max size for each item is different from them as we know. Until the bug 1311516 become fixed these imported definitions looks to give no effect.

I've added a note about the limitation. I hope it will help other people who see same troubles.
https://github.com/piroor/treestyletab/wiki/Code-snippets-for-custom-style-rules#for-version-20-and-later