# Build arguments



In this script:



<table><thead><tr><th width="227" align="center">Configuration</th><th width="253" align="center">Explanation</th><th align="center">Suggested Values</th></tr></thead><tbody><tr><td align="center">max_input_len</td><td align="center">The maximum length of the input sequence.</td><td align="center">512 - 1024 (depending on the model architecture and available GPU memory)</td></tr><tr><td align="center">max_output_len</td><td align="center">The maximum length of the output sequence.</td><td align="center">256 - 512 (depending on the desired output length and available GPU memory)</td></tr><tr><td align="center">max_batch_size</td><td align="center">The maximum batch size for the engine.</td><td align="center">1 - 32 (depending on the available GPU memory and desired throughput)</td></tr><tr><td align="center">max_beam_width</td><td align="center">The maximum beam width for beam search during generation.</td><td align="center">1 - 8 (higher values can improve output quality but increase computational cost)</td></tr><tr><td align="center">max_num_tokens</td><td align="center">The maximum number of tokens to generate.</td><td align="center">100 - 500 (depending on the desired output length)</td></tr><tr><td align="center">opt_num_tokens</td><td align="center">The optimal number of tokens to generate.</td><td align="center">50 - 200 (depending on the desired output length and trade-off between quality and efficiency)</td></tr><tr><td align="center">max_prompt_embedding_table_size</td><td align="center">The maximum size of the prompt embedding table (for prompt tuning).</td><td align="center">0 - 10000 (depending on the number of prompt templates used for prompt tuning)</td></tr><tr><td align="center">gather_context_logits</td><td align="center">Whether to gather context logits during generation.</td><td align="center">False (set to True for debugging or analysis purposes)</td></tr><tr><td align="center">gather_generation_logits</td><td align="center">Whether to gather generation logits during generation.</td><td align="center">False (set to True for debugging or analysis purposes)</td></tr><tr><td align="center">strongly_typed</td><td align="center">Whether to use strongly typed TensorRT networks.</td><td align="center">True (enables additional optimizations and error checking)</td></tr><tr><td align="center">builder_opt</td><td align="center">The optimization level for the TensorRT builder.</td><td align="center">3 (default value, higher values may result in longer build times but potentially better performance)</td></tr><tr><td align="center">profiling_verbosity</td><td align="center">The verbosity level for TensorRT profiling.</td><td align="center">"layer_names_only" (provides a good balance between profiling information and readability)</td></tr><tr><td align="center">enable_debug_output</td><td align="center">Whether to enable debug output for the TensorRT network.</td><td align="center">False (set to True for debugging purposes)</td></tr><tr><td align="center">max_draft_len</td><td align="center">The maximum length of the draft sequence (for Medusa models).</td><td align="center">0 - 200 (depending on the desired draft length for Medusa models)</td></tr><tr><td align="center">use_refit</td><td align="center">Whether to use the refit feature for multi-GPU building.</td><td align="center">False (set to True for multi-GPU builds to reduce build time)</td></tr><tr><td align="center">input_timing_cache</td><td align="center">The path to the input timing cache file.</td><td align="center">"timing_cache.bin" (provide a path to a previously generated timing cache file to speed up the build process)</td></tr><tr><td align="center">output_timing_cache</td><td align="center">The path to the output timing cache file.</td><td align="center">"output_cache.bin" (provide a path to store the generated timing cache for future builds)</td></tr><tr><td align="center">lora_config</td><td align="center">A <code>LoraBuildConfig</code> object specifying LoRA configuration for the model.</td><td align="center">Depends on the specific LoRA adaptation requirements and available pre-trained LoRA weights</td></tr><tr><td align="center">auto_parallel_config</td><td align="center">An <code>AutoParallelConfig</code> object specifying auto-parallel configuration for the model.</td><td align="center">Depends on the number of available GPUs and the desired trade-off between build time and inference performance</td></tr><tr><td align="center">weight_sparsity</td><td align="center">Whether to enable weight sparsity for the engine.</td><td align="center">False (set to True if the model weights are sparse and you want to optimize for storage and computation)</td></tr><tr><td align="center">plugin_config</td><td align="center">A <code>PluginConfig</code> object specifying plugin configuration for the engine.</td><td align="center">Depends on the specific requirements and available custom TensorRT plugins</td></tr><tr><td align="center">max_encoder_input_len</td><td align="center">The maximum length of the encoder input sequence (for encoder-decoder models).</td><td align="center">512 - 1024 (depending on the encoder architecture and available GPU memory)</td></tr><tr><td align="center">use_fused_mlp</td><td align="center">Whether to use fused MLP layers for optimization.</td><td align="center">True (can improve performance by reducing memory accesses and kernel launches)</td></tr><tr><td align="center">dry_run</td><td align="center">Whether to perform a dry run without actually building the engine.</td><td align="center">False (set to True for testing purposes or to validate the build configuration without spending time on the actual build)</td></tr><tr><td align="center">visualize_network</td><td align="center">Whether to visualize the TensorRT network as a DOT graph.</td><td align="center">False (set to True for debugging or understanding the network structure)</td></tr></tbody></table>
