Viewer can be sluggish for one of the most common type of series we have (chest CT)

[wlongabaugh]

@fedorov @pieper If users hit our landing page and click on the featured CT scan, they are looking at a series with 280 slices. It takes a long time to load, so if they start scrolling, they are looking at a frozen "Loading..." page for awhile. Perhaps not the greatest of first impressions? The featured PET scan also has a ton of slices (262), though is not quite as frozen.

Best case scenario would be a snazzy image with fewer slices showing up as the first thing you see.

[pieper]

Yes, there is definitely something wrong with the frame fetching behavior. If possible, I'd prefer to fix the underlying issue, since 280 slices is not an unusual case.

Here's what I see:

1. open the console to the Network tab
2. open the study url https://dev-viewer.canceridc.dev/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571
3. grab the slice scroll tab on the right side and pull it to the bottom of the window to look at slice 280.

The result is a very long (~5 second) delay with the "Loading" message before the slice appears even though the network tab is very active.

At the beginning, slices are downloading within about 100ms, so if things were working correctly I should be able to see the any interactively selected slice within 100ms.

But instead it appears that as queue is being swamped with fetches related to slices that were triggered as I scrolled past, and they are not canceled even though I'm no longer on that slice. In the end so many slice fetches are queued up that some of the accesses take over 7 seconds.

@swederik do you agree this is an issue we can fix? I'm kind of curious to take a look myself, but I'm sure you know the code a lot better than I do.

[swederik]

There are definitely things we can do to improve it but I think there will be some tradeoffs (e.g. dropping in-progress requests, which would be a waste of data and server resources). I spoke to James and he's going to look into it and see if there's anything obviously wrong.

[pieper]

From what I could see there are a lot of pending requests generated when dragging the scrollbar that become "stale" when the scroll bar moves on. We should be able to identify and drop those. If we can really get an arbitrary slice in 100ms then there is no reason we should ever have more than 100ms latency between scrolling to a location and seeing the corresponding slice. Anything else is a bug IMHO.

[fedorov]

I was recording a video for the tutorial, and it took about 45 seconds to load MRP for a chest CT. I am going to cut that piece out, and add a message "45 seconds later".

[pieper]

It would be good to know if this is an issue with the client, the proxy, or the google healthcare api. My evidence points at the proxy (sorry @wlongabaugh).

There are 280 slices in the CT study linked from the main page, and if you watch the network tab image below you can see that the time takes anywhere from 79ms to almost 9 seconds per slice. At 100ms/slice, you should get 10 slices per second, or 28 seconds worst case non-overlapped access. So 45 seconds means that whatever we are doing is 2x worse.

If I use the sandbox to hit google dicomweb directly, I can load a 500 slice study in about 10 seconds, and if I look at the network tab worst case is about 400ms per slice, and most are below 50ms.

@wlongabaugh are you able to look at the proxy logs together with the network tab in the browser to see what's going on? Maybe a scaling issue over other overload of the proxy?

[s-paquette]

@wlongabaugh Is this fixed by #311?

[wlongabaugh]

@s-paquette Alas, no, since the selected series include the many-slice series in question here.

[wlongabaugh]

Data point one is that we only keep three instances running at all times, and a large influx of requests requires instances to be spun up. These are the instances coming online to handle the first tab series. A request that brings an instance online will take about 7-8 seconds to respond. At about $10/day per instance, it would cost $36/K year to keep ten instances at the ready. It might be cheaper on App Engine Flex, I don't know. This spin-up time is not something you will see hitting Google:

[wlongabaugh]

Data point two is that I agree the myriad OPTIONS calls were taking too long; the code to respond to that was later than it needed to be, and I am deploying that fix.

[wlongabaugh]

@pieper When I chose a large-slice series from sandbox-000, I am seeing about 1 second load times per slice (see below). I am also seeing, for some reasons, a lot less CORS OPTIONS calls to the server. I note that for the IDC featured series, I am seeing over 600 calls to bring down 280 slices, with half of those as OPTIONS calls. I have made them a little bit more efficient, but this is slowing things down.

[wlongabaugh]

I needed to set the value of "Access-Control-Max-Age" to something in the CORS OPTIONS response, which cut the number of requests to the server roughly in half. (This is deployed on dev.) Still looking at other possible optimizations.

[pieper]

Sounds like progress.

Data point one is that we only keep three instances running at all times, and a large influx of requests requires instances to be spun up.

Should we limit the client to only make max of 3 simultaneous requests?

[wlongabaugh]

Series is not going to be changed by MVP, this is now a proxy/viewer combination performance issue that is post-MVP.

[fedorov]

There were improvements to the viewer over the past few months, and performance is now considerably better.