Unit tests for layers

[zaleslaw]

@mkaze wrote in the PR #103

I saw some common patterns in testing the layers which each of us is repeating to some extent in our implementations and my main goal is to refactor them so that we could use them in all of the related tests and avoid duplication. Some of them are (which may currently exist or not):

• Helper functions for creating N-D Arrays.
• Helper functions for casting Arrays to specific dimensionality.
• A single function to run a forward pass on a given instance of a layer and given an input data array of arbitrary dimensionality and return the output of the layer. (either in eager/graph mode)
• Verify the output of layer with the expected result.
• Verify the output of computeOutputShape method of layer with the expected output shape.
• Test config loader/saver functions of the layer.

I think most of these are already implemented; however, I think there are currently some duplication in the tests which could be further refactored and simplified, so that all layers' test use a common API regardless of the type of the layer (i.e. it does not matter if it's a pooling or convolution layer).

I want to add some ideas about layer functionality testing.

• we need to re-invent our own Nd Arrays on Kotlin side (maybe we could use multik for that)
• we need to test in both eager and graph mode everywhere possible (due to different details of implementation in eager/graph modes for the same ops)
• we need to create and share a good collection of different converters from Tensor to multidimensional array (probably via extension functions)
• testing of layers including the same steps for all layers and the same steps for layers from one group like merge layers or convolutional layers, maybe we need to repeat the hierarchy of layer classes in the hierarchy of testing classes for these levels

[zaleslaw]

@avan1235 @therealansh if you have any thoughts to add, please write here (you faced with some problems here, your feedback will be useful)

[therealansh]

Hi @zaleslaw , i do agree with @mkaze about the redundant code in the testing. Also, the abstraction of some functions making a utility/helper class like keras/tf does in testing_utils could also help in writing more generalised code. @avan1235 casting and other helper functions in ShapeFunctions could be brought to use into making it more general for Layer testing or any other Testing as well. We could a raise a ticket for it. While coming onto your question of using multik, i think should we benchmark it with some other like nd4j but since we need pure kotlin i think multik can be used to some extent since i think its still in dev stage. Finally adding some examples as @avan1235 suggested will help.

[avan1235]

Let's see that for now we only consider using multik in tests phase for the simplicity of TCs definitions so the benchmarking wouldn't needed here because in the internal tests the speed doesn't seem to be critical point. Testing utils sounds like a good idea of structuring some helper functions/interfaces and there is definitely a lot of places to reuse hte created helpers (but let's see at my proposal of their structure, what do you think about pure Kotlin funcitons vs structured in interfaces?)

[avan1235]

Reusing multik from my perspective looks like the best idea as for now there are many other fields that need develepers time and reinventing the wheel usually sounds like a bad idea. The good thing about the multik is that we can just depend on its api and in future, if needed, create own implementation (as the api looks pretty in my opinion and makes it simpler to work with arrays especially for the people experienced with numpy). The other point is that, for now, we are talking only about the usage of ndarrays in tests phase, which doesn't have to be the fastest possible and the scale (the sizes of arrays) of test examples don't seem to be large enough to see the actual difference in execution time.

We definitely need the "AbstractExecutor" of test that would be able to use the graph or eager mode of TF to evaluate the test. In my opinion creating the interfaces here with default methods and default properties could be some good approach (that I have recently used with traits in Scala and worked well in combining tests but traits are a little bit different than interfaces). We would have some EagerExecutor and GraphExecutor interfaces from which the tests would inherit getting in this way the capability from default implementations. Of course, it can also be done via standard classes of executors and creating them as the fields in test classes but it brings us again to some blocks of repeated code. With the default methods in interfaces we gain something like multiinheritance which used carefully can simplify the structure of test classes.

I would start with creating the Tensor API that for now would be backed by TF Tensor implementation. Then creating the whole package under the *.api.tensor with the concrete converters that would be functions of Tensor API and the extension functions of multik api ndarrays. That would simplify the future work with some other integrations than TF and looks suitable for this pupose while working only with extension function could force us to future refactors of this code when TF Tensor would change. On the other hand writing such Tensor API seems like creating the copu of multik ndarray API, so maybe the best idea would be to try to implment this multik API with TF Tensor as a backaend extended with a few extra conversion functions?

For the tests writing and repeating hierarchy I am not sure if it flll be suitable in any case (I mean that the implementations hierarchy doesn't have to correspond to tests' needs) but what about wringing some tests functionalities also with interfaces? I mean creating e.g. the interface with methods for creating random tensors as default methods and just derive from them (getting in this way well structured and organized test helper functions that would be groupped in the interfaces according to their capabilities as importing single Kotlin functions may be sometimes problematic but it is another good approach)