Could consider using standard gitignore file for python in full rewrite, e.g. https://github.com/github/gitignore/blob/main/Python.gitignore

Not yet functional or tested.

Also ensures empty context tokens work

Note that the API does not accept empty context, thus adding a newline to get generations
with empty context, ideally this doesn't affect distribution much

This replaces old transformer tokenizer with API calls.
Also removes tokenisation were not strictly necessary.

Removes one of two newlines at end of file

This also removes old logic where newline was added
to context. Disabling is greedy computation helps when dealing with
empty context data.

With and without disabling greedy computation.

unlike the gpt3 model we need a custom loglikelihood_rolling
 implementation
 because we don't know the eot token, thus we use a separate prefix token.
 Preferred setting separate prefix token to avoid confusion.

 This also adds the capability to
 _loglikelihood_tokens to use tokens instead of str requests

also prevent tokenisation of strings that too long