kvs-watch: retrieve content blobs in chunks rather than all at once

[chu11]

While #6414 / #6444 solves an important initial problem of not re-fetching older unnecessary data, whenever there is a large amount of data to retrieve, it sends requests to the content store to retrieve all data.

The difference can be seen in something like

> flux run flux lptest 2000000 80`
<wait job done>
> flux job attach `flux job last`

In the first job, when stdout is being "watched", the very first lookup of the stdout will not be that large. And every subsequent append that is watched should be only 1 or a few lines of output.

In the follow up flux job attach, the job is already completed. So the very first lookup of the tree object for stdout will be all 2 million lines. kvs-watch will subsequently do a lookup of all 2 million lines. i.e. it's a for loop doing 2 million iterations and sending 2 million RPCs. That's not good.

Instead, only a subset should be retrieved in "windows". This allows data to be sent back to the caller much more quickly.

[chu11]

While this issue still exists, it appears that the larger performance issue is in job-info, where an entire lookup on the entire key is done if the job has completed. That should probably similarly be "windowed" as well.

Edit: I'm wondering if we could simply treat the "guest_released" and "guest_started" paths similarly. And we would get the faster lookup from kvs-watch. The main difference is we don't have to watch out for the guest namespace being removed.

    if (check_guest_namespace_status (gw, s) < 0)                                                                                           
        goto error;                                                                                                                         
                                                                                                                                            
    if (gw->guest_released) {                                                                                                               
        /* guest namespace copied to main KVS, just watch it like normal */                                                                 
        if (main_namespace_lookup (gw) < 0)                                                                                                 
            goto error;                                                                                                                     
    }                                                                                                                                       
    else if (gw->guest_started) {                                                                                                           
        /* guest namespace created, watch it and not the main KVS                                                                           
         * namespace */                                                                                                                     
        if (guest_namespace_watch (gw) < 0)                                                                                                 
            goto error;                                                                                                                     
    }                                                                                                                                       
    else {                                                                                                                                  
        /* wait eventlog for start */                                                                                                       
        if (wait_guest_namespace (gw) < 0)                                                                                                  
            goto error;                                                                                                                     
    }    

[chu11]

just documenting prototype failures. The main issue is how can the job-info module "watch" an eventlog after the job has finished and know it is "at the end" of the data. Because nothing indicates this natively.

• first attempt - use KVS treeobj valref array length to indicate number of "events" there are. Ends up not safe, b/c internally multiple "events" may exist within a single content blob.

• possible other solutions

  • kvs: do not allow appends that exceed maximum allowed value size #6264/kvs: ensure large values exceeding maximum rejected consistently #6265/kvs: define maximum size #6266 - generally speaking have way to get known "size" of the the eventlog
  • idea: kvs: support a native "EOF" #6455 - native KVS "EOF"
  • have a new kvs-watch flag which is effectively "stream until current end of valref array and that's it, you're done"
  • there could be a similar flag in job-info.eventlog-watch
    • one of these "watch until end" options appeals to me although I haven't thought about it much