README: Add support for llama.cpp

* README.org: The llama.cpp server supports OpenAI's API, so we
can reuse it.  Closes #121.
This commit is contained in:
Karthik Chikmagalur 2023-12-28 22:23:31 -08:00
parent f571323174
commit 85bd47cb4c

View file

@ -5,14 +5,14 @@
GPTel is a simple Large Language Model chat client for Emacs, with support for multiple models/backends. GPTel is a simple Large Language Model chat client for Emacs, with support for multiple models/backends.
| LLM Backend | Supports | Requires | | LLM Backend | Supports | Requires |
|-------------+----------+-------------------------| |-------------+----------+---------------------------|
| ChatGPT | ✓ | [[https://platform.openai.com/account/api-keys][API key]] | | ChatGPT | ✓ | [[https://platform.openai.com/account/api-keys][API key]] |
| Azure | ✓ | Deployment and API key | | Azure | ✓ | Deployment and API key |
| Ollama | ✓ | [[https://ollama.ai/][Ollama running locally]] | | Ollama | ✓ | [[https://ollama.ai/][Ollama running locally]] |
| GPT4All | ✓ | [[https://gpt4all.io/index.html][GPT4All running locally]] | | GPT4All | ✓ | [[https://gpt4all.io/index.html][GPT4All running locally]] |
| Gemini | ✓ | [[https://makersuite.google.com/app/apikey][API key]] | | Gemini | ✓ | [[https://makersuite.google.com/app/apikey][API key]] |
| Llama.cpp | ✓ | [[https://github.com/ggerganov/llama.cpp/tree/master/examples/server#quick-start][Llama.cpp running locally]] |
| PrivateGPT | Planned | - | | PrivateGPT | Planned | - |
| Llama.cpp | Planned | - |
*General usage*: ([[https://www.youtube.com/watch?v=bsRnh_brggM][YouTube Demo]]) *General usage*: ([[https://www.youtube.com/watch?v=bsRnh_brggM][YouTube Demo]])
@ -46,6 +46,7 @@ GPTel uses Curl if available, but falls back to url-retrieve to work without ext
- [[#gpt4all][GPT4All]] - [[#gpt4all][GPT4All]]
- [[#ollama][Ollama]] - [[#ollama][Ollama]]
- [[#gemini][Gemini]] - [[#gemini][Gemini]]
- [[#llamacpp][Llama.cpp]]
- [[#usage][Usage]] - [[#usage][Usage]]
- [[#in-any-buffer][In any buffer:]] - [[#in-any-buffer][In any buffer:]]
- [[#in-a-dedicated-chat-buffer][In a dedicated chat buffer:]] - [[#in-a-dedicated-chat-buffer][In a dedicated chat buffer:]]
@ -221,6 +222,30 @@ You can pick this backend from the transient menu when using gptel (see Usage),
#+html: </details> #+html: </details>
#+html: <details>
#+html: <summary>
**** Llama.cpp
#+html: </summary>
Register a backend with
#+begin_src emacs-lisp
(gptel-make-openai ;Not a typo, same API as OpenAI
"llama-cpp" ;Any name
:stream t ;Stream responses
:protocol "http"
:host "localhost:8000" ;Llama.cpp server location
:models '("test")) ;List of available models
#+end_src
These are the required parameters, refer to the documentation of =gptel-make-openai= for more.
You can pick this backend from the transient menu when using gptel (see Usage), or set this as the default value of =gptel-backend=:
#+begin_src emacs-lisp
(setq-default gptel-backend (gptel-make-openai "llama-cpp" ...)
gptel-model "test")
#+end_src
#+html: </details>
** Usage ** Usage
(This is also a [[https://www.youtube.com/watch?v=bsRnh_brggM][video demo]] showing various uses of gptel.) (This is also a [[https://www.youtube.com/watch?v=bsRnh_brggM][video demo]] showing various uses of gptel.)