Compiling LLama Models

Here is a list of arguments

1. --model_dir: Specifies the directory where the pre-trained model is stored.

2. --meta_ckpt_dir: Specifies the directory where the meta checkpoint is stored.

3. --tp_size: Sets the N-way tensor parallelism size.

4. --pp_size: Sets the N-way pipeline parallelism size.

5. --dtype: Determines the data type for the model weights, with options including float32, bfloat16, and float16.

6. --vocab_size: Specifies the vocabulary size of the model.

7. --n_positions: Sets the number of positions for embeddings.

8. --n_layer: Specifies the number of layers in the model.

9. --n_head: Sets the number of attention heads in the model.

10. --n_kv_head: Specifies the number of key-value heads, if different from n_head.

11. --n_embd: Sets the dimensionality of embeddings.

12. --inter_size: Specifies the size of the intermediate layer in the transformer.

13. --rms_norm_eps: Sets the epsilon value for RMS normalization.

14. --use_weight_only: Enables quantization of weights only, without affecting activations.

15. --disable_weight_only_quant_plugin: Disables the plugin implementation for weight-only quantization, using the out-of-the-box implementation instead.

16. --weight_only_precision: Defines the precision for weight-only quantization, with options including int8, int4, int4_awq, and int4_gptq.

17. --smoothquant: Activates SmoothQuant quantization with a specified alpha parameter.

18. --per_channel: Uses a different static scaling factor for each channel in GEMM's result for quantization.

19. --per_token: Chooses a custom scaling factor for each token at runtime during quantization.

20. --int8_kv_cache: Enables INT8 quantization for the key-value cache.

21. --ammo_quant_ckpt_path: Path to a quantized model checkpoint in .npz format.

22. --per_group: Chooses a custom scaling factor for each group at runtime, specifically for GPTQ/AWQ quantization.

23. --quantize_lm_head: Quantizes the language model head weights as well when using INT4_AWQ.

24. --enable_fp8: Uses FP8 Linear layer for Attention QKV/Dense and MLP.

25. --fp8_kv_cache: Chooses FP8 quantization for the key-value cache.

26. --load_by_shard: Enables loading a pre-trained model shard-by-shard.

27. --hidden_act: Specifies the hidden activation function.

28. --rotary_base: Sets the base value for rotary embeddings.

29. --rotary_scaling: Specifies the type and factor for rotary scaling.

30. --group_size: Sets the group size used in GPTQ/AWQ quantization.

31. --storage-type: Specifies the storage type, with options including fp32 and fp16.

32. --dataset-cache-dir: Sets the cache directory to load the Hugging Face dataset.

33. --load-model-on-cpu: Forces the model to load on the CPU.

34. --convert-model-on-cpu: Forces the model conversion to occur on the CPU.

35. --use_parallel_embedding: Enables embedding parallelism.

36. --embedding_sharding_dim: Specifies the dimension along which to shard the embedding lookup table.

37. --use_embedding_sharing: Attempts to reduce the engine size by sharing the embedding lookup table between layers.

38. --use_prompt_tuning: Enables prompt tuning.

39. --output_dir: Specifies the directory to save the converted model checkpoint.

40. --workers: Sets the number of workers for converting the checkpoint in parallel.

41. --moe_num_experts: Specifies the number of experts for MOE layers.

42. --moe_top_k: Sets the top_k value for MOE layers.

43. --moe_tp_mode: Determines how to distribute experts in tensor parallelism.

44. --moe_renorm_mode: Controls renormalization after gate logits for MOE.

45. --use_fused_mlp: Enables horizontal fusion in GatedMLP.

46. --enable_pos_shift: Enables position shift for the streaming LLM method.

47. --dense_context_fmha: Enables dense FMHA in context phase, as opposed to sliding window attention.

48. --hf_lora_dir: Specifies the directory for a LoRA model.