Other gpu optimizations

[khosravipasha]

• Init: Initiliazing for gpu init_marginal is slow because it moves to cpu everytime to do the initilization. Right now about 50% of the time of taking marginals is in init

• Move to CuArray{Union{Missing, Float}} for input layer instead of dataframe

• Lazy Batching: Something similar to batchview https://mldatapatternjl.readthedocs.io/en/latest/documentation/dataview.html#as-vector-of-batches

• Last batch size reuse memory: Now if remaining samples in the last batch does not match batch_size, our memory reuse would not work. Also in the beginning of next epoch we have to reallocate memory again for bigger size, since last batch was smaller. Solution is probably to not reallocate again for last batch and just not use the extra memory.

• Different way of Batching: I think our current way of batching is slowing things down. Also right now if we want to reorder data for each epoch cannot do that (useful for minibatching to have random order each time I believe).

• Bottlenecked by CPU?: CPU is always at 100% when paramters learning for me, might suggest we are bottlenecked at cpu. Also trying to find and reduce movements of data between cpu and gpu.

[khosravipasha]

almost all are resolved in cuda-refactor branch.

batchview from LazyBatching might be useful later: https://mldatapatternjl.readthedocs.io/en/latest/documentation/dataview.html#as-vector-of-batches

[khosravipasha]

all are done in v.04