vault81/redvault-ai
4
0
Fork 0
Code Issues 1 Pull requests 1 Projects 1 Releases Packages Wiki Activity

Update llama_proxy_man/README.md

Browse source
This commit is contained in:
Tristan D. 2024-10-08 15:37:58 +00:00
parent 66cf52e2ce
commit 734a6300a1
1 changed files with 5 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
llama_proxy_man/README.md
View file

@ -3,3 +3,8 @@
- manages multiple llama.cpp instances in the background
- keeps track of used & available video & cpu memory
- starts/stops llama.cpp instances as needed, to ensure memory limit is never reached
## Ideas
- smarter logic to decide what to stop
- unified api, with proxying by model_name param for stamdartized `/v1/chat/completions` and `/completion` like endpoints