there is no performance report on the pure generation, is it really bad?

[xiaoyunwu]

as title.

[sh0416]

If you mean the exact match score, I exclude it as I am not confident on that time.
LLaMA2 would give great performance, but I didn't test it yet.

[xiaoyunwu]

https://paperswithcode.com/sota/text-classification-on-ag-news
according to this, the performance should have been around 95%.
Is your much accurancy number too good to be true so you excluded?

[sh0416]

I don't think so, but you could try.

[xiaoyunwu]

Have you do fine-tuning on it, and if so, how did you do it?

[sh0416]

This repository does not finetune the models.
I measure the accuracy with only prompting.
Therefore, we do not need any training dataset and we consider it as zero-shot classification.
The link mentioned in your comment is finetuned model specialized on that task.
Current LLMs could not exceed highly specialized task such as this one.
I attached the paper related to this repository.

https://aclanthology.org/2022.acl-long.365/

[xiaoyunwu]

this is a bit puzzling, given the amount of the data that goes into the pre-training of these things. By the way, are you still in UW?

[sh0416]

Could you please clarify your question?

[xiaoyunwu]

. We use channel models
for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context
demonstration or prompt tuning
I find this in your abstract, my question is how did prompt tuning work on this data set?

[sh0416]

I said that the paper is "related" to this repository, not "reproducing" the paper.

[sh0416]

FYI, I am not the author of that paper.