experimental research project on capacity theory of neural network
Mimicing bit flipping as a casual model, learning ALU caculation.
Experiment on the capacity reduction and generalization ability.
pipelining the dataset using tf.data API, Reading the data from sets fo files directly
toy models' training and saving result
Standardized models' building process, gonna use dense network with decreasing size
Custom models' building process
resnet?
piNN training
Small task traning 3 bits dataset with ADD, SUB operations
-
Memorization Use all dataset as training set, test curve of memorization(this is more like an explorational experiment)
-
Generalization Split the dataset into 2 halves and measure the generalization metrics.
Spliting the training set will inevitably cause the incompleteness of the definition of some rules? -
Bit hacking Manually calcualte minimal possible bits of different logical and arithematical operations. See if we can analyse the similar capacity of similar research: Nerual ALU/ Nerual GPU.
-
Casual Model 3 bits net with only ADD, SUB, AND, OR, XOR etc.
check if a manual net would work
mess up the weight, check if training helps .