Plugin to show database size (% of available space or in MiB)

[dlvoy]

This plugin shows pill with information about size of database,
can display both % of available (declared) max size, or absolute size in MiB

This is useful for long-time users on free tier mongoDB, to indicate when to clean-up DB and prevent issues when DB size quota is exceeded unnoticed.

When size reached default or configured warning/urgent threshold - it highlights yellow or red
Includes: icon (with sources), translation support, README entry & tests

for DBSIZE_IN_MIB=true

[sulkaharo]

Heh, our DB is a bit bigger than usual. :) This looks great! Given the frequency of database issues, can you change this so it's enabled by default?

[sulkaharo]

I'm not clear on why the assets/fonts/pluginicons-v1.0/ folder was included? I'd prefer to keep the repo as clean as possible, where is this needed?

[dlvoy]

@sulkaharo you have biffy mighty self hosted or pro DB plan :)

It is not possible to detect how much space you've got from provider, default 496MiB is for free mongodb/heroku tier - change DBSIZE_MAX to that value where your DB limits (100%) is, and other (DBSIZE_WARN & DBSIZE_URGENT - for warning levels.

[sulkaharo]

@dlvoy this is actually just the $5/month Digital Ocean droplet! Can you comment on the extra folder?

[dlvoy]

@sulkaharo problem with icons is - i wanted to add them to nsicons, but all i found in repo was
resulting icon font & raw binary files (TTF, WOFF) which i cannot easily extend with new icons.
Therefore i created new one and included all sources (assets/fonts) and documented (in assets/fonts/README.md) how to make additions to that icon set and update project with new icons.

That way, if someone will need to add yet another icon, he goes to that folder, use it on https://icomoon.io/app/ to add new icon, updates it and main.css with generated font - and pushes update to repo :)

[dlvoy]

I will update it to have it enabled as default :) 👍

[dlvoy]

@sulkaharo I've tried to set plugin as default with commit 6fb3008, but i am still not expert with all NS internals - could you check is it ok/enough?

[sulkaharo]

@dlvoy I get the readme and the json file for the fonts, but the font demo files in the pluginicons-v1.0 folder are not needed, are they? The readme makes no mention of that folder and none of the files in the folder are used by the Nightscout runtime and/or the font generation and so they probably shouldn't live in the repo.

[dlvoy]

@sulkaharo you have right - i left them because it is exact content of generated ZIP file from icomoon.
But with source/project JSON and README.md it should be all it is needed to update main.css.
I will delete them!

[sulkaharo]

One last comment - on testing setting the warning levels to higher than default, I realised it'd be much easier for most users if the warn and urgent thresholds were defined as a percentage of the max. This way even people who have a larger database would typically not need to set more than one variable (and not do math on what the warn/urgent values should be). :)

[dlvoy]

@sulkaharo Thanks for idea! I've modified plugin to be % configured, updated README.md accordingly and added two tests to check if new setting works

[jasoncalabrese]

Here is an overview of what got changed by this pull request:

Issues
======
- Added 12
           

Complexity increasing per file
==============================
- lib/plugins/dbsize.js  14
         

See the complete overview on Codacy

[andrew-dixon]

I've noticed on the dev branch that the option on the menu doesn't save if you want to switch off the database size pill, so it is not possible to switch it off.

[dlvoy]

@andrew-dixon confirmed, it seems that code forces to have it always set instead of have it set only by default, my fault, i'll fix it!

[dlvoy]

@sulkaharo can you help?
I want to have plugin dbsize shown by default (and checked on plugin list) but also allow to turn it off - but do not know how to do it:

• if i force that plugin it is always shown, and i cannot uncheck it
• if i remove it from list - it is not shown by default and user need to check it on the plugin list

forcing is done at lib/settings.js: adjustShownPlugins with settings.showPlugins += ' delta direction upbat dbsize';

[dlvoy]

I think i fixed it! at PR #5529

[andrew-dixon]

@dlvoy I've spotted another potential issue with the implementation. My database filled up and Nightscout stopped getting data but the pill said 25% as that was the "data size" but when I hovered over it, the "database file size" was 100%. I didn't notice this at first as I saw the 25% and assumed the issue must be with the connection to Dexcom or something and was trying other things to resolve the issue. Had I seen the 100% on the pill I would have probably got to the issue much quicker. So I'm not sure if the pill is outputting the wrong size or if needs to output both or maybe just whichever is greater?

[dlvoy]

@andrew-dixon i tried to explain that a little bit in README section, in NOTE. Because the way MongoDB allocates files, database file size gets 100% relatively quickly, when it jumps from 256MiB to 496MiB, so when used as size source it would show 100% and red pill when database is 51% full of data - for most of mLab free tier people - way too soon.

In configuration you can set DBSIZE_WARN lower, to be warned sooner, and i set warn level on relatively low 60% of capacity.

Unfortunately, there is no way to check fragmentation of free space inside sparse, so it may fail at 60% or work without issues at 95%. After mouse hover on pill you will see both numbers thou, and doing repairDatabase (like using mLab Tools -> Commands tab and dropdown, or with mongodb client, like Robo 3T) will rebuild database file and defragment free space.

[andrew-dixon]

@dlvoy I've actually just had this DB file size issue again tonight:

Once you hit that 100% on the file size, Nightscout stops being able to receive new data, and from what I've seen this always occurs first, e.g. the file size is always larger than the "data size".

After running a repair DB on mLab I end up like this:

With the file size still larger than the data size. So wouldn't it be better to use the file size and not the data size or at least use whichever is larger?

[dlvoy]

@andrew-dixon but that is not correct, when you hit 100% on file size you have not run out of space inside database! It only means that mongodb will not be able to allocate new extent. I run my NS instance with 100% of file size and it still can get new data since months:

As long as its internal organisation inside extents allows it - it will accept data. In your case, repairDatabase both defragmented data and allowed for smaller count of extents. And extents are allocated not linearly, so you will always have file size either 240MiB o 496MiB (i do not remember sizes for smaller tiers). As long data fit in 240MiB extent, it will use it, but then with brutto 250MiB of data + indexes + stuff your file size will jump to 496MiB but you will still have aprox. 49% free space left inside for data.

How much it can take - it depends, because records are also allocated in powers of-2 sizes and they left holes when deleted, so it produces fragmentation problem - like with old FAT file systems.
You can find details why is that here: https://blog.mlab.com/2014/01/how-big-is-your-mongodb/

And of course there there are configurations of mongodb engine with replication and stuff that do not suffer from it - it is more or less because we use it on small scale with development, free tier :)

[andrew-dixon]

@dlvoy Well, I can only go by my own experience, but both times my Nightscout has stopped working recently the "Database file size" has been at 100% and the "Data size" at about 26% and doing the repair database on mLab as "fixed" the issue and it has started working again, once the "Database file size" is no longer at 100%. Maybe it is something specific to mLab, but it certainly doesn't appear to work any more once you hit 100% on the "Database file size".

[andrew-dixon]

@dlvoy Just to let you know, that since migrating from mLab to Atlas (due to the mLab shutdown) the database size pill no longer appears, despite still being turned on under the "Show Plugins" section. I assume this is because it can't get the data required from Atlas in the same way it did from mLab.

[dlvoy]

@andrew-dixon indeed! db.stats() reports fileSize = 0 even for non-empty database (since it does not use mmapv1 storage engine)
I will try to fix this. Thanks for finding that out!

[inventor96]

@dlvoy I was just doing some work on virtual assistant integration, and saw that you added support for that in your plugin. In order for virtual assistants to be asked about something, they need to be configured to recognize it in speech, which is done in Alexa's "Interaction Model" and in Google's "Language Template". The work you did here simply tells Nightscout how to respond when such a request comes in. Did you update the Interaction Model and Language Template in another PR? If not, no worries, I can include that in the work I'm doing now.