Llama Java parameters

Llama Java parameters

The following is a list of all the parameters involved in this project.

Note

Other reference documents: Transformers docs.

Model parameters

• Basic parameters

Parameter	Default	Description
model_path	/	Llama model path.
lora_base	/	Optional model to use as a base for the layers modified by the LoRA adapter.
lora_path	/	Apply a LoRA (Low-Rank Adaptation) adapter to the model (implies --no-mmap).
lora_scale	0.0	apply LoRA adapter with user defined scaling S (implies --no-mmap).
verbose	false	Print verbose output to stderr.
numa_strategy	0	Attempt one of the below optimization strategies that may help on some NUMA systems (default: disabled). enum LlamaNumaStrategy

• Context parameters

Parameter	Default	Description
seed	-1	Set the random number generator seed.
context_size	512	Option allows you to set the size of the prompt context used by the LLaMA models during text generation.
batch_size	2048	Set the batch size for prompt processing.
ubatch	512	Physical maximum batch size (default: 512).
seq_max	1	Max number of sequences (default: 1).
threads	4	Set the number of threads used for generation (single token).
threads_batch	4	Set the number of threads used for prompt and batch processing (multiple tokens).
rope_scaling_type	-1	RoPE scaling type. enum LlamaRoPEScalingType
pooling_type	-1	Pooling type for embeddings. enum LlamaPoolingType
rope_freq_base	0.0	Base frequency for RoPE sampling.
rope_freq_scale	0.0	Scale factor for RoPE sampling.
yarn_ext_factor	-1.0	YaRN extrapolation mix factor, NaN = from model.
yarn_attn_factor	1.0	YaRN magnitude scaling factor.
yarn_beta_fast	32.0	YaRN low correction dim.
yarn_beta_slow	1.0	YaRN high correction dim.
yarn_orig_ctx	0	YaRN original context size.
defrag_thold	-1.0	KV cache defragmentation threshold (default: -1.0, < 0 = disabled).
logits_all	false	Return logits for all tokens, not just the last token.
embedding	false	Embedding mode only.
offload_kqv	true	Whether to offload the KQV ops (including the KV cache) to GPU.
flash_attn	false	Enable flash attention (default: disabled).

• Model parameters

Parameter	Default	Description
gpu_layers	0	Number of layers to offload to GPU (-ngl). If -1, all layers are offloaded.
split_mode	1	How to split the model across multiple GPUs. enum LlamaSplitMode
main_gpu	/	When using multiple GPUs this option controls which GPU is used for small tensors for which the overhead of splitting the computation across all GPUs is not worthwhile.
tensor_split	/	When using multiple GPUs this option controls how large tensors should be split across all GPUs.
vocab_only	false	Only load the vocabulary no weights.
mmap	true	Use mmap if possible (slower load but may reduce pageouts if not using mlock).
mlock	false	Lock the model in memory, preventing it from being swapped out when memory-mapped.
check_tensors	false	Validate model tensor data (default: disabled).

JSON template

{
  "model_path": "",
  "lora_base": "",
  "lora_path": "",
  "lora_scale": 0.0,
  "verbose": false,
  "numa_strategy": 0,
  "seed": -1,
  "context_size": 512,
  "batch_size": 2048,
  "ubatch": 512,
  "seq_max": 1,
  "threads": 4,
  "threads_batch": 4,
  "rope_scaling_type": -1,
  "pooling_type": -1,
  "rope_freq_base": 0.0,
  "rope_freq_scale": 0.0,
  "yarn_ext_factor": -1.0,
  "yarn_attn_factor": 1.0,
  "yarn_beta_fast": 32.0,
  "yarn_beta_slow": 1.0,
  "yarn_orig_ctx": 0,
  "defrag_thold": -1.0,
  "logits_all": false,
  "embedding": false,
  "offload_kqv": true,
  "flash_attn": false,
  "gpu_layers": 0,
  "split_mode": 1,
  "main_gpu": 0,
  "tensor_split": [],
  "vocab_only": false,
  "mmap": true,
  "mlock": false,
  "check_tensors": false
}

Generate parameters

Parameter	Default	Description
temperature	0.8	Adjust the randomness of the generated text.
repeat_penalty	1.1	Control the repetition of token sequences in the generated text.
penalize_nl	true	Disable penalization for newline tokens when applying the repeat penalty.
frequency_penalty	0.0	Repeat alpha frequency penalty.
presence_penalty	0.0	Repeat alpha presence penalty.
top_k	40	TOP-K Sampling Limit the next token selection to the K most probable tokens.
top_p	0.9	TOP-P Sampling Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P.
tsf	1.0	Tail Free Sampling (TFS) Enable tail free sampling with parameter z.
typical	1.0	Typical Sampling Enable typical sampling sampling with parameter p.
min_p	0.05	Min P Sampling Sets a minimum base probability threshold for token selection.
mirostat_mode	DISABLED	Mirostat Sampling Enable Mirostat sampling, controlling perplexity during text generation. DISABLED = disabled, V1 = Mirostat, V2 = Mirostat 2.0
mirostat_eta	0.1	Mirostat Sampling Set the Mirostat learning rate, parameter eta.
mirostat_tau	5.0	Mirostat Sampling Set the Mirostat target entropy, parameter tau.
dynatemp_range	0.0	Dynamic Temperature Sampling Dynamic temperature range. The final temperature will be in the range of (temperature - dynatemp_range) and (temperature + dynatemp_range).
dynatemp_exponent	1.0	Dynamic Temperature Sampling Dynamic temperature exponent.
grammar_rules	/	Specify a grammar (defined inline or in a file) to constrain model output to a specific format.
max_new_token_size	512	Maximum new token generation size.
verbose_prompt	false	Print the prompt before generating text.
last_tokens_size	64	Maximum number of tokens to keep in the last_n_tokens deque.
special	false	If true, special tokens are rendered in the output.
logit_bias	/	Adjust the probability distribution of words.
stopping_word	/	Control the stop word list for generating stops, with values that can be text or token IDs.
infill	false	Enable infill mode for the model.
spm_fill	false	Use Suffix/Prefix/Middle pattern for infill (instead of Prefix/Suffix/Middle) as some models prefer this.
prefix_token	/	Specify a prefix token in fill mode. (If not specified, read from the model by default)
suffix_token	/	Specify a suffix token in fill mode.
middle_token	/	Specify a middle token in fill mode.
session_cache	false	If enabled, each chat conversation will be stored in the session cache.
prompt_cache	false	Cache the system prompt in the session and does not update them again.
user	User	Specify user nickname.
assistant	Assistant	Specify bot nickname.

JSON template

{
  "temperature": 0.8,
  "repeat_penalty": 1.1,
  "penalize_nl": true,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "top_k": 40,
  "top_p": 0.9,
  "tsf": 1.0,
  "typical": 1.0,
  "min_p": 0.05,
  "mirostat_mode": "DISABLED",
  "mirostat_eta": 0.1,
  "mirostat_tau": 5.0,
  "dynatemp_range": 0.0,
  "dynatemp_exponent": 1.0,
  "grammar_rules": null,
  "max_new_token_size": 512,
  "verbose_prompt": false,
  "last_tokens_size": 64,
  "special": false,
  "logit_bias": null,
  "stopping_word": null,
  "infill": false,
  "spm_fill": false,
  "prefix_token": "",
  "suffix_token": "",
  "middle_token": "",
  "session_cache": false,
  "prompt_cache": false,
  "user": "User",
  "assistant": "Assistant"
}