Any thoughts / docs / examples about reinforcement learning

[rikkertkoppes]

I'd like to train the network with reinforcement learning. There have been some talk about that:

• http://stackoverflow.com/questions/10722064/training-a-neural-network-with-reinforcement-learning
• http://stackoverflow.com/questions/9010576/neural-network-learning-without-training-values

Maybe you can provide some examples / wiki about that area? I am clearly lacking some knowledge here. In any case, any pointers, examples or snippets regarding implementation of reinforcement learning ANN using synaptic would be great!

[cazala]

Sorry, reinforcement learning is something I haven't played with yet, but It would be very interesting to see if there can be a solution implemented using synaptic, or if there's a way to expand the library to support such training.

[iamDecode]

I’ve been working on an old evolutionary reinforcement learning project (fish learning to find and eat food) that I wish to refactor to synaptic.js. It basically works by mutating and crossing over genomes (which is the set of connection weights in the network) in a way that the better performing fish’s genome stays more in tact.

My design is a simple perceptron with 4 inputs, 1x6 hidden neurons and 2 outputs. I had to create a getter and setter for the weights in the network itself but other than that my design should work.

Unfortunately I run into some issues. First of all the weights in synaptic are generally being generated between -0.1 to 0.1. This results in the output hardly ever changing and usually always in the 0.47-0.53 range. Why is this the case?
For my project I changed this to -1 to 1 range and I get seemingly similar output as my original project. I'm not sure if this might break something else in synaptic though..

Weirdly enough, even though both projects work almost entirely the same, the synaptic.js version of my project does not actually learn (ie. fishes do not get better at finding food). I debugged the hell out of the code but as far as I see everything looks good.

Perhaps solving this issue for my project could help in making synaptic.js do reinforcement learning. In order to facilitate this ill post both my project’s versions here: original and synaptic. press F to speed up 'time' and you'll be displayed a graph of performance.

I hope this can be of help!

[Matlino]

@iamDecode I am working on similar project where i want to use neural networks together with genetic algorithms. Problem i have is accessing the weights of neurons. If i read the code correctly weigths should be stored in object Neuron.connections, then there is array called projected which should have property weight. But for some reason all elements of array projected are undefined for me. Maybe i am missing something from js syntax, i am not rly used to it yet.

For example i try to print out the weight of first conneciton of third neuron in XOR example.

   var neurons = myNetwork.neurons();
    var n1 = new Neuron(neurons[0]);
    console.log(n1);
    console.log(n1.connections.projected[0]);

This is the output

{ ID: 9,
  label: null,
  connections: { inputs: {}, projected: {}, gated: {} },
  error: { responsibility: 0, projected: 0, gated: 0 },
  trace: { elegibility: {}, extended: {}, influences: {} },
  state: 0,
  old: 0,
  activation: 0,
  selfconnection: 
   { ID: 18,
     from: [Circular],
     to: [Circular],
     weight: 0,
     gain: 1,
     gater: null },
  squash: [Function],
  neighboors: {},
  bias: 0.05332792326807975 }
undefined

How did u access weights values?

[menduz]

Hello, on the first activation synaptic compiles the entire network to
plain js by default, we do no map each reference of the object to the
compiled heap yet. You can use a flag to avoid this compilation, but the
network will be MUCH slower.

Try this before the first activation:

'''javascript
network.optimized = false;
'''
On Mon, Jan 11, 2016 at 10:05 Matlino [email protected] wrote:

@iamDecode https://github.com/iamDecode I am working on similar project
where i want to use neural networks together with genetic algorithms.
Problem i have is accessing the weights of neurons. If i read the code
correctly weigths should be stored in object Neuron.connections, then there
is array called projected which should have property weight. But for some
reason all elements of array projected are undefined for me. Maybe i am
missing something from js syntax, i am not rly used to it yet.

For example i try to print out the weight of first conneciton of third
neuron in XOR example.

var neurons = myNetwork.neurons();
var n1 = new Neuron(neurons[0]);
console.log(n1);
console.log(n1.connections.projected[0]);

This is the output

{ ID: 9,
label: null,
connections: { inputs: {}, projected: {}, gated: {} },
error: { responsibility: 0, projected: 0, gated: 0 },
trace: { elegibility: {}, extended: {}, influences: {} },
state: 0,
old: 0,
activation: 0,
selfconnection:
{ ID: 18,
from: [Circular],
to: [Circular],
weight: 0,
gain: 1,
gater: null },
squash: [Function],
neighboors: {},
bias: 0.05332792326807975 }undefined

How did u access weights values?

—
Reply to this email directly or view it on GitHub
#43 (comment).

[Matlino]

@menduz
Thank u for advice, when i set optimized = false network runs slower as u said but i still cannot access the weights. I tried access weights both before and after training.

[cazala]

@Matlino To set a network as unoptimized you have to use the setOptimize method.

var myNetwork = new Architect.Perceptron(2,4,2)
myNetwork.setOptimize(false)
myNetwork.activate([ ... ]) // <- unoptimized activation

But that's not recomended, since the network becomes waaaay slower.
What you want is to populate back all the network parameters from the values of the heap, to do that just call the method restore()

var myNetwork = new Architect.Perceptron(2,4,1)
console.log(myNetwork.layers.input.list[0].connections.projected[11480].weight) // -0.09417646038345993
myNetwork.trainer.XOR() // <- train the network
console.log(myNetwork.layers.input.list[0].connections.projected[11480].weight) // -0.09417646038345993
myNetwork.restore() // <- popullate the network
console.log(myNetwork.layers.input.list[0].connections.projected[11480].weight) // 0.9376776976219627

[Matlino]

@cazala thanks, that worked

[yonatanmn]

hi @cazala , Iv'e just started playing with nn, and your lib looks amazing.
I'm currently working on some project that uses reinforcement learning, and I'm not sure how to implement it with Synaptic.
The problem is that I wand output of "actions" (arrows to press, in order to move the creature in screen), but I don't have "desired-actions" to compare with, I have some other "reward" that I can give, and the network should change weights to get better and better reward.

I've seen that Convnet.JS has deepqlearn out of the box, but I prefer this lib so far, and would love to see it supports reinforcement learning.
Is there any progress, or any way to implement it with Synaptic?

Thanks a lot!

[menduz]

@yonatanmn maybe you could dive into genetic algorithms. That works better with the "reward" training. Take a look at this: https://github.com/ivanseidel/IAMDinosaur

Saludos

[robtarr]

I've been working on a similar problem, and I setup IAMDinosaur locally, but it never seemed to learn how to jump.

I am running N number of random genomes, taking the best one, and then creating mutations from that one. Doing this by exporting the Network JSON from the best, mutating some of the neuron bias values, and then creating a new network from that data. The problem I have is that once I begin the mutations, one of my output values never changes - it seems to just be stuck at one value or another. Any tips or resources would be GREATLY appreciated.

function _mutateData(data) {
  const MUTATION_PROBABILITY = 0.3;

  for (let i in data.neurons) {
    if (Math.random() < MUTATION_PROBABILITY &&
        data.neurons[i].layer !== 'input' &&
        data.neurons[i].layer !== 'output') {
      data.neurons[i].bias = Math.random() * 2 - 1;
    }
  }

  return data;
}

const exportData = this.baseGenome.export();
      let newData = _mutateData(exportData);

      this.currentGenome = new Genome(newData);
      this.genomes.push(this.currentGenome);

[wagenaartje]

@robtarr. I had this problem as well. What you should try and do is clear() the network/layer/neuron you're mutating. For some reason, this made mutation to actually work. I did exactly the same project as you, used synaptic to create an AI for the google dinosaur game.

If you want to have some other ideas on how to mutate and breed synaptic neural networks, feel free to take a look at my library Gynaptic , synaptic with genetic algorithm implementations.

I also made an issue about your problem #197.