[Bug/Question] Write With Transformers Implementation vs. Custom Implementation

[krrishdholakia]

Hi,

Not sure if this is a bug or perhaps a misimplementation, but i'm comparing the results of using gpt2-large on the 'Write With Transformers' Text Generation example - https://transformer.huggingface.co/doc/gpt2-large, vs. my own implementation of the text generation tool. My use-case is to generate distractor options in an MCQ environment, given a few (15-20) prior examples for the style of distractor questions to generate.

The general format I am implementing is:
Question: _______ . Answer 1: ___<the correct answer>___ . Answer 2: _____<distractor 1>_____ . Answer 3: ____<distractor 2>_____

Write With Transformers

full doc available here - https://transformer.huggingface.co/share/CZqVXdngic

Model Config

Model size - gpt2/large
Top-p - 0.9
Temperature - 1
Max time - 2.3

Output

On the 'Write With Transformers' page, I write in the examples:
eg. Question: How is calorie related to the S.I unit of that quantity?Answer 1: 1 cal = 4.2 J.Answer 2: 1 cal = 3.2 J.Answer 3: 1 cal = 10 J.

and when I try to generate predictions for the following Question-Answer pair:
Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W.

am able to generate a few solid distractors:
Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: 1 H. P. = 1. 3 W. Answer 3 : 1 H . P. = 1. 5 W.

Custom Case

Output

Here's how I create a dataset from the set of questions I've initially written myself.

def make_dataset(dataset, epochs):
    total_text = '<|endoftext|>'
    qa = [t for t in dataset]
    for _ in range(epochs):
        random.shuffle(qa)
        total_text += '<|endoftext|>'.join(qa) + '<|endoftext|>'
    return total_text

This is the training model params:

!python run_language_modeling.py \
    --output_dir=output/$handle \
    --overwrite_output_dir \
    --overwrite_cache \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train --train_data_file=$handle\_train.txt \
    --logging_steps 20 \
    --per_gpu_train_batch_size 1 \
    --num_train_epochs 15

num_return_sequences = 5

for start in SENTENCES:
    val = !python run_generation.py \
        --model_type gpt2 \
        --model_name_or_path output/$handle \
        --length 40 \
        --num_return_sequences $num_return_sequences \
        --temperature 0.23 \
        --p 0.95 \
        --seed $seed \
        --prompt {'"<|endoftext|>' + start + '"'}
    generated = [val[-1-2*k] for k in range(num_return_sequences)[::-1]]
    print(f'\nStart of sentence: {start}')
    for i, g in enumerate(generated):
        g = g.replace('<|endoftext|>', '')
        print(f'* Generated #{i+1}: {g}')

These are my generated mcq pairs:

Generated #1: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.
Generated #2: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.
Generated #3: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: ______________ = 905 J.Answer 3: 50 J.
Generated #4: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.
Generated #5: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.