FeedForward.scala and NDArrayIter.scala leak memory by not disposing of NDArrays

[milandesai]

Description

FeedForward.scala and NDArrayIter.scala leak memory by not disposing of NDArrays. We see the below leak warnings occur running on MXNet 1.1.0. Additionally, there is no clean way of disposing a FeedForward and its contents.

Version

MXNet 1.1.0

Error Message:

[2018-04-06 01:12:13,895] WARN  - Finalizer - WarnIfNotDisposed - LEAK: An instance of class ml.dmlc.mxnet.NDArray was not disposed. Creation point of this resource was:
	java.lang.Thread.getStackTrace(Thread.java:1552)
	ml.dmlc.mxnet.WarnIfNotDisposed.$init$(WarnIfNotDisposed.scala:52)
	ml.dmlc.mxnet.NDArray.<init>(NDArray.scala:549)
	ml.dmlc.mxnet.NDArray.slice(NDArray.scala:632)
	ml.dmlc.mxnet.FeedForward.$anonfun$predict$5(FeedForward.scala:233)
	scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
	scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
	scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
	scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
	scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
	ml.dmlc.mxnet.FeedForward.predict(FeedForward.scala:232)

[2018-04-06 01:12:13,895] WARN  - Finalizer - WarnIfNotDisposed - LEAK: An instance of class ml.dmlc.mxnet.NDArray was not disposed. Creation point of this resource was:
	java.lang.Thread.getStackTrace(Thread.java:1552)
	ml.dmlc.mxnet.WarnIfNotDisposed.$init$(WarnIfNotDisposed.scala:52)
	ml.dmlc.mxnet.NDArray.<init>(NDArray.scala:549)
	ml.dmlc.mxnet.NDArray.slice(NDArray.scala:632)
	ml.dmlc.mxnet.io.NDArrayIter.$anonfun$_getData$2(NDArrayIter.scala:163)
	scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
	scala.collection.Iterator.foreach(Iterator.scala:929)
	scala.collection.Iterator.foreach$(Iterator.scala:929)
	scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
	scala.collection.IterableLike.foreach(IterableLike.scala:71)
	scala.collection.IterableLike.foreach$(IterableLike.scala:70)
	scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	scala.collection.TraversableLike.map(TraversableLike.scala:234)
	scala.collection.TraversableLike.map$(TraversableLike.scala:227)
	scala.collection.AbstractTraversable.map(Traversable.scala:104)
	ml.dmlc.mxnet.io.NDArrayIter._getData(NDArrayIter.scala:163)
	ml.dmlc.mxnet.io.NDArrayIter.getData(NDArrayIter.scala:176)
	ml.dmlc.mxnet.io.NDArrayIter.next(NDArrayIter.scala:134)
	ml.dmlc.mxnet.FeedForward.predict(FeedForward.scala:226)

The culprit appears to be FeedForward.scala:233 and NDArrayIter:163. Additionally, it would be nice to to provide a method that disposes of a FeedForward; i.e. it disposes the symbol, executor, and aux/arg params.

[wellner]

Any chance of having this issue bumped up? This bug renders training via Scala impossible in many cases and could be problematic for large batch decoding/inference.

[lanking520]

Thanks for your message! We are currently working on a draft proposal to solve this issue and send to the devlist. The disposing method currently brings a great inconvenience to users. In the past, we tried to place that into finalizer stage and it sometimes crash for unknown reasons. Will follow up with this thread with our progress.

[wellner]

Great! Looking forward to it and happy to help test and/or review.

[andrewfayres]

Memory leak was a result of the FeedForward predict method not properly disposing of two NDArrays. This was fixed in the PR above. Issue can be closed. @nswamy

[lanking520]

@wellner Currently @yzhliu brings a solution on Memory leaks here: #11751. Please take a look in here.

[vandanavk]

Can this issue be closed? @milandesai @wellner

[vandanavk]

@sandeep-krishnamurthy Please close this issue as PRs with fixes have been submitted.

@milandesai @wellner Please feel free to reopen the issue if you see it again.