How to deploy MPT with llama.cpp?

[streetycat]

I found that llama.cpp already supports mpt, I downloaded gguf from here, and it did load it with llama.cpp, but its return result looks bad.

I start the server as follow:

git clone https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python
docker build -t llama-cpp-python-cuda docker/cuda_simple/
docker run --gpus all --rm -it -p 8000:8000 -v ${/path/to/models}:/models -e MODEL=/models/${model-filename} llama-cpp-python-cuda python3 -m llama_cpp.server --n_gpu_layers ${X} --n_ctx ${L}

And I post the request as follow:

URL: http://localhost:8000/v1/chat/completions
BODY:

{
"messages": [
{"role": "user", "content": "What is 5+7?"}],
"max_tokens": 8000
}

RESPONCE:

{
    "id": "chatcmpl-bfe9eaf4-2ab4-419d-ab8b-313add0706f9",
    "object": "chat.completion",
    "created": 1700798859,
    "model": "/models/mosaicml-mpt-30b-chat-Q8_0.gguf",
    "choices": [
        {
            "index": 0,
            "message": {
                "content": "></s>\n<s>&nbsp;</s><br/> </body> </html> \n\nNotice the use of JavaScript to handle user input and submit it via AJAX. ...\n\n",
                "role": "assistant"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 16,
        "completion_tokens": 2258,
        "total_tokens": 2274
    }
}

But, it works well on the demo page:

I worked well with llama in the same method, I don't know what's wrong happened.

[gardner]

How are you sending the POST? Can you share your curl command or fetch() request?

[streetycat]

Thank you, I have found a parameter(chat_format) in source code for startup to fix it.

and update the command:

docker run --gpus all --rm -it -p 8000:8000 -v ${/path/to/models}:/models -e MODEL=/models/${model-filename} llama-cpp-python-cuda python3 -m llama_cpp.server --n_gpu_layers ${X} --n_ctx ${L} --chat_format mistrallite

the responce:

By the way, maybe the </s> is a bug for the chat_format?

[gardner]

I can't find anywhere that mosaic describes the prompt template. There is one note on the model card here: https://huggingface.co/TheBloke/mpt-30B-chat-GGML#prompt-template

[streetycat]

Thank you for your help, I am a beginner and don't know where to find this information. I guess gguf and ggml are just an information organization format, and the specific behavior of the model should be consistent with the official mpt.

[gardner]

Correct. .gguf is just a file format to store the model weights. Much like .zip is a file format to store other files. The contents of a .gguf can contain different models that have been trained to use different prompt formats. You can usually find the prompt format on the model card on HuggingFace but sometimes you need to do some searching to figure out what to use.

Good luck in your learnings!

[tk-master]

Definitely looks like prompt format issue