Usage with other DenoKV Wrappers specifically KV-Toolbox

[RickeyWard]

Is there an way to use kvdex with VK-Toolbox's blob storage wrapper? Maybe its obvious and I'm not seeing it?

https://github.com/kitsonk/kv-toolbox/blob/main/blob.ts

[oliver-oloughlin]

I don't think there is any simple or obvious way either. The thing is, I'm not sure there is any need for them to work together either. kitsonk's kv-toolbox works great for what it does and can easily be used alongside kvdex if you so choose to. Just pass the same Deno.Kv instance to both of them. Since kvdex prefixes all keys, you shouldn't be afraid of them colliding.

Moreover, depending on your use case, you might just be fine using serialized collections in kvdex. Have you tried this? It does something similar to kv-toolbox blob utilities by serializing values as a uint8array, compressing it, and then splitting it into multiple KV entries to surpass the 64Kb size limit. This should allow you to store blobs. I can give you an example:

import { kvdex, collection, model } from "kvdex"

const kv = await Deno.openKv()
const db = kvdex(kv, {
  // Note that your model doesn't have to be of type Uint8Array, you can store any value you wish
  blobs: collection(model<Uint8Array>(), { serialize: "json" })
})

const blob = new TextEncoder().encode("foo bar baz")
const result = await db.blobs.add(blob)

The only issue I can see with using serialized collections for this is that it performs an unnecessary serialize step under the hood because it is designed to work with any values, not just blobs. I hadn't actually given much thought to this specific case tbh. I will probably refactor this in the near future to just skip the serialize step for blobs, which should remove the performance hit.

[RickeyWard]

Awesome! I look forward to giving it a try.

I would have expected the batching to happen automatically based on the limitation size. 800kib / 64kib = 12.5 so I'd guess that anytime there's more than 12, 64kib chunks there will have to be multiple batches or it will throw an error. (would have to be calculated based on whatever chunk size you're specifically using) I'm not sure of the benefits of being able to choose an arbitrary batch size. It would either be split because exceeding KV/foundationdb limitations or not split.

I could see wanting to be able to opt in/out since it could possibly cause consistency issues and someone might want it to just throw an error rather than not be atomic, it seems valid to leave that choice up the consumer. Seems like more of a bool than a number, but I can work with a number.

[oliver-oloughlin]

The problem with setting an automatic limit is that the ~800kb limit is the total size of the operation, including keys and values. So if you were to set very large keys (or ids) that 12 batch limit could be broken. I see that kv-toolbox uses a fixed size of 10. That's probably a pretty safe number considering keys cant be larger than 2048 bytes. I might go back and change it to a bool as you said (and other operations that still use batching), but as it stands now it should at least enable a wide range of use cases.

[RickeyWard]

That makes sense, 10 seems like a fair compromise.

Using 0.33.1 I was able to store and retrieve 1-2mb blobs, when I tried to stress it with 5 I got the error:

TypeError: Failed to compress
    at brotliCompressSync (ext:deno_node/_brotli.js:154:15)
    at Object.compress (https://deno.land/x/[email protected]/src/utils.ts:547:18)
    at Collection.setDoc (https://deno.land/x/[email protected]/src/collection.ts:1935:43)
    at Collection.setDocument (https://deno.land/x/[email protected]/src/collection.ts:1891:23)
    at Collection.set (https://deno.land/x/[email protected]/src/collection.ts:570:23)

called from

   const newID = ulid(); //import { ulid } from "https://deno.land/x/ulid/mod.ts"
   const fileData= new Uint8Array(await file.arrayBuffer()); //file from formdata
   await blobs.set(newID, fileData, {atomicBatchSize: 10});

I was able to work around that by modifying my collection:

const db = {
        blobs: collection(model<Uint8Array>(), {
            serialize: {
                serialize: jsonSerialize,
                deserialize: jsonDeserialize,
                compress: (v) => v,
                decompress: (v) => v,
            }
        })
}

using the atomicBatchSize:10 and the manually disabling compression I was able to store a 512mb blob and recover it. I certainly wouldn't do this outside of a test, but it worked.

[oliver-oloughlin]

Overriding the compression would be my recommendation in this case anyway (you dont have to insert the json serialize/deserialize, they are set by default if you dont set them). The merging that happens when doing this is probably not very clear or explained anywhere, so I can definitely do a better job of documenting it and possibly changing the default behaviour when not setting certain options.

I gave your input a lot of thought and through some testing I found out that it is possible to dynamically and automatically estimate the size of operations and do batching according to that. So in the latest release I have removed the atomicBatchSize option and replaced it with a boolean batched option instead. I was able to store and retrieve a 91mb blob with this approach. Haven't tried anything as large as 512mb yet, and this update doesn't affect the compression, so I would guess your issue still persists. I will do some more testing myself with larger files.

I apologize for introducing breaking changes in so quick succession. I just want to make any changes I can for the better now while in 0.x versioning, so that when we eventually hit a stable 1.0.0 release the API and functionality are as good as we can get them.

[RickeyWard]

No problem, thanks for sharing your library! KV and FoundationDB are frustrating to work with directly.