tristan/gptel
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

README: Add instructions for Llamafile

Browse source 
* README.org (* Llama.cpp): As it turns out, text-generation
Llamafile models (currently Mistral Instruct and Llava) offer an
OpenAI-compatible API, so we can use them easily from gptel.  Add
instructions for Llamafiles to the Llama section of the README.
This commit is contained in:
Karthik Chikmagalur 2023-12-31 14:37:26 -08:00
parent 48047c0600
commit 3ac5963080
1 changed files with 18 additions and 16 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

22
README.org
View file

@ -12,6 +12,7 @@ GPTel is a simple Large Language Model chat client for Emacs, with support for m
| GPT4All | ✓ | [[https://gpt4all.io/index.html][GPT4All running locally]] | | GPT4All | ✓ | [[https://gpt4all.io/index.html][GPT4All running locally]] |
| Gemini | ✓ | [[https://makersuite.google.com/app/apikey][API key]] | | Gemini | ✓ | [[https://makersuite.google.com/app/apikey][API key]] |
| Llama.cpp | ✓ | [[https://github.com/ggerganov/llama.cpp/tree/master/examples/server#quick-start][Llama.cpp running locally]] | | Llama.cpp | ✓ | [[https://github.com/ggerganov/llama.cpp/tree/master/examples/server#quick-start][Llama.cpp running locally]] |
| Llamafile | ✓ | [[https://github.com/Mozilla-Ocho/llamafile#quickstart][Local Llamafile server]] |
| PrivateGPT | Planned | - | | PrivateGPT | Planned | - |
*General usage*: ([[https://www.youtube.com/watch?v=bsRnh_brggM][YouTube Demo]]) *General usage*: ([[https://www.youtube.com/watch?v=bsRnh_brggM][YouTube Demo]])
@ -46,7 +47,7 @@ GPTel uses Curl if available, but falls back to url-retrieve to work without ext
- [[#gpt4all][GPT4All]] - [[#gpt4all][GPT4All]]
- [[#ollama][Ollama]] - [[#ollama][Ollama]]
- [[#gemini][Gemini]] - [[#gemini][Gemini]]
- [[#llamacpp][Llama.cpp]] - [[#llamacpp-or-llamafile][Llama.cpp or Llamafile]]
- [[#usage][Usage]] - [[#usage][Usage]]
- [[#in-any-buffer][In any buffer:]] - [[#in-any-buffer][In any buffer:]]
- [[#in-a-dedicated-chat-buffer][In a dedicated chat buffer:]] - [[#in-a-dedicated-chat-buffer][In a dedicated chat buffer:]]
@ -138,7 +139,7 @@ Register a backend with
#+end_src #+end_src
Refer to the documentation of =gptel-make-azure= to set more parameters. Refer to the documentation of =gptel-make-azure= to set more parameters.
You can pick this backend from the transient menu when using gptel. (See usage) You can pick this backend from the menu when using gptel. (see [[#usage][Usage]])
If you want it to be the default, set it as the default value of =gptel-backend=: If you want it to be the default, set it as the default value of =gptel-backend=:
#+begin_src emacs-lisp #+begin_src emacs-lisp
@ -163,7 +164,7 @@ Register a backend with
#+end_src #+end_src
These are the required parameters, refer to the documentation of =gptel-make-gpt4all= for more. These are the required parameters, refer to the documentation of =gptel-make-gpt4all= for more.
You can pick this backend from the transient menu when using gptel (see usage), or set this as the default value of =gptel-backend=. Additionally you may want to increase the response token size since GPT4All uses very short (often truncated) responses by default: You can pick this backend from the menu when using gptel (see [[#usage][Usage]]), or set this as the default value of =gptel-backend=. Additionally you may want to increase the response token size since GPT4All uses very short (often truncated) responses by default:
#+begin_src emacs-lisp #+begin_src emacs-lisp
;; OPTIONAL configuration ;; OPTIONAL configuration
@ -188,7 +189,7 @@ Register a backend with
#+end_src #+end_src
These are the required parameters, refer to the documentation of =gptel-make-ollama= for more. These are the required parameters, refer to the documentation of =gptel-make-ollama= for more.
You can pick this backend from the transient menu when using gptel (see Usage), or set this as the default value of =gptel-backend=: You can pick this backend from the menu when using gptel (see [[#usage][Usage]]), or set this as the default value of =gptel-backend=:
#+begin_src emacs-lisp #+begin_src emacs-lisp
;; OPTIONAL configuration ;; OPTIONAL configuration
@ -212,7 +213,7 @@ Register a backend with
#+end_src #+end_src
These are the required parameters, refer to the documentation of =gptel-make-gemini= for more. These are the required parameters, refer to the documentation of =gptel-make-gemini= for more.
You can pick this backend from the transient menu when using gptel (see Usage), or set this as the default value of =gptel-backend=: You can pick this backend from the menu when using gptel (see [[#usage][Usage]]), or set this as the default value of =gptel-backend=:
#+begin_src emacs-lisp #+begin_src emacs-lisp
;; OPTIONAL configuration ;; OPTIONAL configuration
@ -224,28 +225,29 @@ You can pick this backend from the transient menu when using gptel (see Usage),
#+html: <details> #+html: <details>
#+html: <summary> #+html: <summary>
**** Llama.cpp **** Llama.cpp or Llamafile
#+html: </summary> #+html: </summary>
(If using a llamafile, run a [[https://github.com/Mozilla-Ocho/llamafile#other-example-llamafiles][server llamafile]] instead of a "command-line llamafile", and a model that supports text generation.)
Register a backend with Register a backend with
#+begin_src emacs-lisp #+begin_src emacs-lisp
(gptel-make-openai ;Not a typo, same API as OpenAI (gptel-make-openai ;Not a typo, same API as OpenAI
"llama-cpp" ;Any name "llama-cpp" ;Any name
:stream t ;Stream responses :stream t ;Stream responses
:protocol "http" :protocol "http"
:host "localhost:8000" ;Llama.cpp server location :host "localhost:8000" ;Llama.cpp server location, typically localhost:8080 for Llamafile
:models '("test")) ;List of available models :models '("test")) ;Any names, doesn't matter for Llama
#+end_src #+end_src
These are the required parameters, refer to the documentation of =gptel-make-openai= for more. These are the required parameters, refer to the documentation of =gptel-make-openai= for more.
You can pick this backend from the transient menu when using gptel (see Usage), or set this as the default value of =gptel-backend=: You can pick this backend from the menu when using gptel (see [[#usage][Usage]]), or set this as the default value of =gptel-backend=:
#+begin_src emacs-lisp #+begin_src emacs-lisp
(setq-default gptel-backend (gptel-make-openai "llama-cpp" ...) (setq-default gptel-backend (gptel-make-openai "llama-cpp" ...)
gptel-model "test") gptel-model "test")
#+end_src #+end_src
#+html: </details> #+html: </details>
** Usage ** Usage
(This is also a [[https://www.youtube.com/watch?v=bsRnh_brggM][video demo]] showing various uses of gptel.) (This is also a [[https://www.youtube.com/watch?v=bsRnh_brggM][video demo]] showing various uses of gptel.)