which layer's probabilities are getting changed?

[ra-MANUJ-an]

Hi @the-crypt-keeper !
It might be a little dumb question for you, but which layer's probabilities are you changing? And if it's just last layer's then is there any way we can change probabilities of inner layers (maybe 24th or 26th)?

[the-crypt-keeper]

Hi @ra-MANUJ-an this is actually a very good question!

Logically, it is the "output layer" that we are are tweaking here. This is the one doing projection from embedding space (where all the other, feed forward layers operate) into the logit probability space where we can sample tokens sample from. The Muse is applied after this projection.

Tweaking an intermediate feed forward layers outputs is an interesting thought - but the key is that these layers operate on embedding vectors and not logits. In what way would you need to tweak the embedding vector to make the final output logit 'interesting' in some way is a much tougher challenge then what we are solving here.. embedding dimensions certainly carry meaning but in what way is unclear.

[ra-MANUJ-an]

@the-crypt-keeper
I understand what you want to say. So, do you think it'll have much less impact on the output if the intermediate layer is being tweaked? I was reading the following paper :
https://arxiv.org/abs/2304.13734

which said ``However, if a statement is classified as false, the LLM can delete it and generate a different statement instead. To avoid regenerating the same statement again, the probability of sampling the words that appear in the current statement should be adjusted downward.'' and they're using the intermediate layers as inputs to their classifier. I'm not sure if they're saying to dampen the intermediate layer's output or the last layer's output where logits operate.

And one more question: is there any way we can involve another model (let's call it Model B) to manipulate the logits of Model A? This way, we could incorporate information from Model B when calculating the final outputs of Model A. In this scenario, the logits of Model A would only be altered if signaled by Model B, instead of simply damping the first 100 (I hope this makes sense).

[the-crypt-keeper]

@ra-MANUJ-an That paper sounds like an variation of Reinforcement Learning (https://github.com/opendilab/awesome-RLHF#detailed-explanation) which is a fine-tuning technique, to my understanding you would need to use gradient descent under the hood to actually modifying the weights. I am not sure if its possible to steer the network AWAY from something? Usually you'd descend towards a more-desirable output instead.

As to your second question, that's an interesting thought. Speculative Decoding is the closest relative to this idea that I've seen implemented: this runs a small model and a large model at the same time with a sampler in between, but in this instance the sampler's job is to NOT skew the probability distributions while doing so. There's a fairly simple python implementation of this idea here. I see no reason you couldn't combine these ideas to do something different with the smaller model (like say, use it as a classifier) to decide how to mess with the big one.

[ra-MANUJ-an]

Hi @the-crypt-keeper thanks for the suggestions!
I might not use the first method, but the second method you suggested about Speculative Decoding sounds good. For it to work, how do you think the draft model should interfere with the target model?

As how in code of speculative decoding changes can be made? It'll be helpful if you could suggest.