Home screen loses AAPS data from >3 hours ago

[vickster1]

I’m using Nightscout dev 0.11.0, but was seeing this with 0.10.3 also. I use Dexcom g5 with xdrip+ & AndroidAPS with Dana RS pump.
I see the issue on desktop, iPad & Android phone.

I open my Nightscout page. Data from the last 3 hours is shown correctly, but older data just shows my standard basal profile & my cgm data - no TBRs or carbs/boluses. It also doesn’t show my sage & cage data if they were changed >3 hours ago.
Then it flicks to showing all the info, then flicks back & continues like this. It happens all the time.

[DigitalDan1]

I've seen exactly the same thing happen on Windows 10 Chrome and iPhone XS Max.

I'm using Nightscout 0.10.3, Dexcom G6, xDrip+, AndroidAPS (2.0) with Insight pump. I think it could be related to performance because it seems to be worse at certain times when it's generally pretty slow.

[sulkaharo]

Would you be willing to PM me your Nightscout URL so I can check if I can see errors in the browser console? Sounds like there's some problematic data somewhere that's be causing this - we're on OpenAPS and have no issues with the older data rendering.

[DigitalDan1]

@sulkaharo No errors in the console for me. I've been taking a closer look and tracked it down to the way the client gets new data from the server. When there's a reconnect (roughly every 5 mins for me on Chrome) or hitting reload, the server sends a first chunk of data that includes all the BG values for 48 hours and treatments and device statuses for only the last 3 hours. It then sends a second chunk half a second later with more treatments, etc.

Here's the relevant snippet from websocket.js:

        // send all data upon new connection
        if (lastData && lastData.splitRecent) {
          var split = lastData.splitRecent(Date.now(), times.hours(3).msecs, msecHistory, filterTreatments);
          if (message.status) {
            split.first.status = status(split.first.profiles);
          }
          //send out first chunk
          socket.emit('dataUpdate', split.first);

          //then send out the rest
          setTimeout(function sendTheRest() {
            split.rest.delta = true;
            socket.emit('dataUpdate', split.rest);
          }, 500);
        }

I've seen the second chunk return more than 5,000 treatment records, which takes the browser a long time to process. This is when the user sees just the last 3 hours (like the first screenshot from @vickster1).

Part of the problem for me was duplicate treatment records. Now I've deleted most of them it's a lot faster and I don't really see this problem.

I still think there's a change needed to ensure the browser gets 48 hours of all data, not just BG. Then the second chunk wouldn't be needed at all (by the nightscout browser view anyway).

I may take a closer look when I get time, but for now wanted to share what I'd found.

[sulkaharo]

Ok I took a peek into @vickster1's Nightscout and looks to me like the cause is AndroidAPS uploading unexpectedly large amount of profile switch treatments - the current codebase assumes this dataset is quite small (entered by hand occasionally) but there's actually almost 600 of these data objects loaded in that Nightscout instance. Most of the profile switches have a duration of 0 as well, which is a bit confusing. @MilosKozak or someone else from the AndroidAPS team - wanna help? The profile code needs to be massively optimized if we want to support Nightscout on the new Android APS behavior.

[sulkaharo]

Right ok I did actually find a bug in Nightscout - @vickster1 and @DigitalDan1 can you update to latest DEV and see if it improves the situation?

[vickster1]

That looks to have solved it - thank you Sulka