[Discussion] Is the implementation of cycler efficient?

[NivekT]

TL;DR: It seems in most cases users might be better off using .flatmap(lambda x: [x for _ in n_repeat]) rather than .cycle(n_repeat).

Here is the implementation, basically Cycler reads from the source DataPipe for n number of times.

Things to consider:

1. This means repeating certain operations (e.g. reading from disk, complicated transformation) for n number of times, unless you use in_memory_cache.
2. If shuffle is used afterwards, I believe .flatmap(lambda x: [x for _ in n_repeat]) is strictly better than .cycle(n_repeat).
3. For input = [0, 1, 2], the major difference is that .cycle returns [0, 1, 2, 0, 1, 2] compared to .flatmap(...) returning [0, 0, 1, 1, 2, 2].

Questions:

1. Should we change the implementation?
2. Should we add something like .repeat() which basically does .flatmap(lambda x: [x for _ in n_repeat])?
3. Should we advise users to use .flatmap(...) instead unless they specifically want the ordering of [0, 1, 2, 0, 1, 2]?

@VitalyFedyunin @ejguan Let me know what you think.

[NivekT]

I did some quick profiling and cycle is slower for a relatively simple DataPipe.

[ejguan]

• Should we add something like .repeat() which basically does .flatten(lambda x: [x for _ in n_repeat])?

I would support this proposal because I believe those two have different use cases. And, python does support such functionality via https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.repeat_each

BTW, which are you using for profiling? It looks great.

[NivekT]

Adding an operation sounds good. I will probably modify the docstrings as well to mention that the other option exists and may be more suitable for some use cases.

I am using scalene for profiling.

[NivekT]

Closing this as #748 has landed.