Embarrassingly parallel linear regression over 1MM datasets

[hrishikeshvganu]

Hi,
I have a use case where I need to build a regression model for demand of each product for a retailer. The number of products is > 5million. I plan to use a linear model for each product but the parameters of the model are allowed to be different for each product.

This is a computation where there's a set of {data, model} for each product and there are > 1MM such sets. Since the data at a product level is small ( around 1000 instances) I was thinking of using a miniBatch size of 1000 and train in a loop over the products.

Is there a better approach/built-in functionality that BidMach provides for such embarrassingly parallel tasks?