LM Format Enforcer Guided Decoding Support

[noamgat]

FIX #3713
This PR implements the issue raised by simon-mo to add LMFE decoding support to the OpenAI server via a command line argument.

It introduces a new command line arg
--guided-decoding-backend which defaults to outlines (current implementation) but also support lm-format-enforcer.

Example curl command to test with:

If you start the server with

python -m vllm.entrypoints.openai.api_server --guided-decoding-backend lm-format-enforcer --model meta-llama/Llama-2-7b-chat-hf

You can then run

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-2-7b-chat-hf",
"prompt": "Please create a random person's name.",
"max_tokens": 100,
"temperature": 0,
"guided_json": {  "$schema": "http://json-schema.org/draft-04/schema#",  "description": "",  "type": "object",  "properties": {    "name": {      "type": "object",      "properties": {        "first": {          "type": "string",          "minLength": 1        },        "middle": {          "type": "string",          "minLength": 1        },        "last": {          "type": "string",          "minLength": 1        }      },      "required": [        "first",        "middle",        "last"      ]    }  },  "required": [    "name"  ]}
}'

And get a proper json response.

[noamgat]

The checks caused me to realize some gaps in my original PR. They have all been addressed, this PR is 100% ready to be reviewed.

[simon-mo]

This is pretty good! However, it would be even better if you can help separate out the tests in tests/test_entrypoints/test_openai_server that tests guided decoding into a separate file and we can create a fixture over the server (perhaps different ports) to run end to end test cases. In that case we would have a lot of confidence in recommending both solutions.

[noamgat]

In order to make the tests simpler, I ended up adding a per-request option as well (same idea guided_decoding_backend alongside guided_json etc), and run the tests for both outlines and LMFE.
It required a small change to the outlines implementation, the current one modified the input tokenizer in a breaking fashion, I modified it to work on a copy of it (and added lru_cache to avoid a performance hit)

[jamestwhedbee]

This is great, it is working exactly as expected for me

[jamestwhedbee]

Anything else you think is blocking a merge @simon-mo?

[noamgat]

The failing test is due to a temporary networking issue during the run.

E       socket.gaierror: [Errno -3] Temporary failure in name resolution
 ```

[noamgat]

@simon-mo any update on the review?