Any way to try 1-(0.1 N) scoring?

[wiseodd]

Hi,

I noticed in the new version of the paper, you mentioned 1-(0.1 N) scoring and I'd like to try that.
I didn't see any parameter for this in the code though.
Do you have instructions how to do that somewhere?

[TimDettmers]

For 1-(0.1N) scoring, I hacked my code just to get some evaluation metrics. I will look into this and will see if it makes sense to integrate it into the code.

[SawanKumar28]

Hi @TimDettmers , would it be possible for you to share the changes necessary for 1-(0.1N) or else, 1-1 scoring. I believe Table 7 in the paper has ablation results with 1-1 scoring.

[TimDettmers]

One reason why I did not provide my code for this is that it is very messy and cannot be combine with my framework.

The simplest solution might be to sample from the label matrix e2_multi and select N% of the labels and use this as the label for the model. When you calculate the loss, you need to index both the output of the model and the e2_multi labels with the same index. In pseudo code:

p= 0.1 //10% sampling
N = e2_multi.sum(1)
idx = sample_index(e2_multi, N)
new_labels = e2_mullti[:, idx]
pred = model.forward(e1, rel)
new_pred = pred[:, new_idx]
loss = model.loss(new_pred, new_labels)