CutSet multiplexing

[pzelasko]

CC @danpovey I think you might find it interesting for dataset combination, the code would look roughly like this:

libri_cuts = load_manifest_lazy(...)
giga_cuts = load_manifest_lazy(...)
tedlium_cuts = load_manifest_lazy(...)

muxed_cuts = CutSet.mux(libri_cuts, giga_cuts, tedlium_cuts, weights=[...], seed=10)

sampler = DynamicBucketingSampler(muxed_cuts, shuffle=True, num_buckets=..., ...)

It completely circumvents the issue of mismatched duration buckets with ZipSampler, as all datasets participate in estimation of duration buckets in this approach (might want to increase num_cuts_for_bucket_estimation param in sampler for a good estimate).

[janvainer]

@pzelasko why not set the weights proportional to cutset sizes by default? This way you would deplete all cutsets at the same time on average. If we keep it uniform, we are risking that a small cutset A is depleted fast and for the rest of the epoch, there will be only larger cutset B.

[pzelasko]

Leaving it up to the user in case the cut sets are very large and opened lazily (I can also imagine len not being available for some types of lazy manifests in the future)

[csukuangfj]

Is it possible to return also the idx?

We may need to select a different network to process the returned item depending on the returned idx.

For instance, when combining gigaspeech and librispeech in transducer training, the encoder network is shared and there are two separate decoder+joiner networks for each dataset. If the returned item is from gigaspeech, we would run the decoder+joiner for gigaspeech; if the returned item is from librispeech, we would run the decoder+joiner for librispeech.

If this function returns only item without idx, it is difficult to tell which dataset the returned item is sampled from. Therefore, it's also difficult to select which decoder+joiner to run.

[pzelasko]

Hmm that would be problematic, it would break the API that CutSet and CutSampler is depending on. But there are two other ways that would be much easier:

1. identify where a cut comes from based on cut/supervision/recording ID
2. extend the manifests by adding assigning a custom field that identifies which domain the cut belongs to (cut.origin = "libri") before saving them to disk

[danpovey]

Nice!
Yes, this might be a good way to go. We can see which is better (might affect both speed and convergence).

[csukuangfj]

Also, can we support to specify that as soon as a specified cutset is exhausted, it breaks the while loop.

[pzelasko]

Sure, see #585