interested in sgd for complex function not in the available families; sgd for gmm not implemented?

[donboyd5]

Hi. I am not an expert so I may be misinterpreting things.

I am interested in using sgd to minimize a complex function that does not fit into available families. If I understand it properly, I need to provide that function (and, ideally, its gradient) to sgd.

As I understand it, (a) you do not recommend this because it will be slowed down by R code (#103), but (b) if I were to try it, I should base it on the method of moments example.

However, that example appears not to be implemented. When I tried to run it, the output was:

Error in predict.sgd(out, x, type = "response") : 'model' not supported

Looking at the predict.sgd code (lines 24-26) this appears to be because the gmm method is not supported, at least not in the predict.sgd function.

However, looking at the sgd code (lines 411-471), sgd does appear to try to support gmm, with finite differences if no gradient function is provided.

Would you mind confirming whether I might be able to use sgd to minimize an arbitrary function, and if so how?

Many thanks.

[ptoulis]

Hi @donboyd5
I wouldn't recommend using sgd for arbitrary functions.
For one, if you know the function/gradient then writing the sgd code is straightforward.
Second, none of the 'machinery' we have for glms is easily generalizable, especially the implicit updates.

I would be curious if @dustinvtran has any more thoughts.

[dustinvtran]

Thanks for the cc.

I believe the gmm implementation in sgd was experimental and I didn't get it to work very well. Finite differences is not a useful approach these days in any case as autodiff is readily available.

[donboyd5]

@ptoulis, @dustinvtran, thank you both!

Don