A Neural Network From Scratch

This project was made for educational purposes to practice the skills and knowledge gained during the deep learning course and build my own neural network using minimum number of packages (numpy, pandas, matplotlib).

This neural network is for classifications tasks and it was mostly built for digit dataset from kaggle (train.csv, test.csv files).

Usage

Open main.py file in any notebook or ide.

test = NeuralNetwork(784 , [50, 200, 20] , 10,  'classification', batches = True)

test.prepare(gradient_method = 'gd', activation_func = 'leaky_relu', seed = None, alpha = 0.01, loss_function = 'cross_entropy_loss', val_metric = 'accuracy',  optimizer = 'accelerated_momentum', momentum = 0.9)

test.cosmetic(progress_bar = False, loss_display = True, loss_graphic = False,  iterations = 100)

test.train(train_batches, test_batches, 3)

As you can see, you may choose the number of inputs, outputs, hidden layers and number of neurons for every layer, gradient descent algorithm, activation function, alpha parameter etc.

Note: This implementation of a neural network is highly scalable, unlike other user implementations.

However, be careful when you increase the number of layers and neurons, as due to the high losses, the learning process becomes less controllable.

Neural Net Architecture

 (784 , [50, 200, 50] , 10)

progress_bar and loss_display

loss_graphic

You can also save your model and download it's parameters next time.

test.save(path = 'model_params.npy')
test.load('model_params.npy')

Short brief about .py files

• dataloader.py

  analog of torch.utils.data.DataLoader, but self-made

• functions.py

  contains most popular activation functions: sigmoid, tanh, relu, leaky relu, cross-entropy loss function, softmax function

• gradient_steps.py

  contains 3 most popular gradient descent algorithms: normal, stochastic and stochastic average

• augmentation.py

  provides data augmentation for digit images (rotate, resize, noise etc.)

  

• neural_network.py

  main file with neural network implementation

Predict your own hand-written images

• open draw_digit.py

• train your model

• draw a digit in a Tkinter GUI window

• make prediction

  

To-Do List

• expand neural network for regression puprose (look for new project - PyCandle)
• add L1, L2 regularization (look for new project - PyCandle)
• make it more robust for large number of layers and neurons (look for new project - PyCandle)
• add Batch Normalization (look for new project - PyCandle)
• add optimizers ( RMSProp, Adam etc.) (look for new project - PyCandle)
• make class more pytorch-like (look for new project - PyCandle)
• win Kaggle competition with self-made neural network 😎

Supplementary materials

• very clear explanation of how neural network is made

• nice 3Blue1Brown videos

• very cool network made from scratch by Sebastian Lague

• cool and very clear model from scratch made by Samson Zhang

• chatgpt, to my surprise, can also make network from scratch (he builds it for XOR classification)

• some very usefull videos to understand classification problem ( 1, 2, 3 )