Refactor llama_server role for Windows support and EL-specific setup
- Added Windows-specific tasks for llama_server role, including installation of dependencies, setup of services using NSSM, and configuration of firewall rules. - Introduced a new task file for Windows Exporter setup. - Consolidated EL-specific setup tasks into a separate include file. - Updated inventory to include additional host for llama server. - Adjusted error handling in Prometheus service removal task to prevent failures.
This commit is contained in:
@@ -168,6 +168,37 @@ llama_server_models:
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
|
||||
llama_server_args_extra:
|
||||
[
|
||||
"-t",
|
||||
10,
|
||||
"-np",
|
||||
1,
|
||||
"-b",
|
||||
1024,
|
||||
"-ub",
|
||||
512,
|
||||
"-fa",
|
||||
"on",
|
||||
"-ctk",
|
||||
"q4_0",
|
||||
"-ctv",
|
||||
"q8_0",
|
||||
"-cram",
|
||||
-1,
|
||||
"-sm",
|
||||
"layer",
|
||||
"-dev",
|
||||
"Vulkan1,Vulkan2",
|
||||
"-ts",
|
||||
"8,12",
|
||||
"-fit",
|
||||
"off",
|
||||
"-mg",
|
||||
1,
|
||||
"--mmap",
|
||||
]
|
||||
|
||||
# # Optional global model preset options applied under [*] in models.ini.
|
||||
# # Keys map to llama.cpp CLI args without leading dashes, short args, or LLAMA_ARG_* env names.
|
||||
# llama_preset_global:
|
||||
|
||||
Reference in New Issue
Block a user