# Building from Source

### <mark style="color:blue;">Fetching the Sources</mark>

#### <mark style="color:green;">1. Install</mark> <mark style="color:yellow;">`git-lfs`</mark>

```bash
git lfs install
```

This command prepares your local Git environment to handle large files by not storing them directly in the repository but as references.&#x20;

This initial setup is crucial for working with large files efficiently and is required only once per repository to ensure that your Git configuration is optimized for LFS operations.

It replaces these large files with text pointers inside Git, while storing the file contents on a remote server like GitHub LFS.

#### <mark style="color:green;">2. Clone the TensorRT-LLM repository</mark>

To start working with the TensorRT-LLM, you first need to clone the repository to your local machine.&#x20;

This can be done by executing the following command in your terminal:

```bash
git clone https://github.com/NVIDIA/TensorRT-LLM.git
```

This command clones the entire repository from GitHub to your local directory, allowing you to work with the files, including large files that are handled efficiently through Git Large File Storage (LFS).

#### <mark style="color:green;">3. Move into the directory</mark>

```bash
cd TensorRT-LLM
```

#### <mark style="color:green;">4. Initialise and  update the submodules</mark>

```bash
git submodule update --init --recursive
```

This command performs several actions:

* <mark style="color:yellow;">**`--init`**</mark> initialises your local configuration file to include the submodules defined in the `.gitmodules` file of the repository.
* <mark style="color:yellow;">**`--update`**</mark> fetches all the data from the project and checks out the appropriate commit as specified in your project.
* <mark style="color:yellow;">**`--recursive`**</mark> ensures that this command is run not only in the current module but also in any nested submodules, effectively updating all the submodules within the project.

#### <mark style="color:green;">5. Pulling Large Files with Git LFS</mark>

After initialising and updating your repository's submodules, you'll need to handle large files managed with Git Large File Storage (LFS).   This is where <mark style="color:yellow;">**`git lfs pull`**</mark> comes into play.&#x20;

Running this command will download the large files associated with the current branch from the remote repository, based on the tracking configurations established by Git LFS.

```bash
git lfs pull
```

This step ensures all the necessary assets, which are too large to be efficiently managed by standard Git operations, are properly downloaded and available for use.&#x20;

It's a step before proceeding with operations that depend on these large files, such as building Docker images or executing large-scale data processing tasks.

### <mark style="color:blue;">Building TensorRT-LLM in One Step</mark>

Once Git LFS is set up and the necessary files are pulled, you can proceed to <mark style="color:blue;">**build**</mark> the TensorRT-LLM Docker image.&#x20;

This can be done with a single command:

```bash
make -C docker release_build
```

This command builds a Docker image that contains everything you need to run TensorRT-LLM, simplifying the setup process and ensuring consistency across environments.

* Optionally specify GPU architectures with <mark style="color:yellow;">**`CUDA_ARCHS`**</mark>

<details>

<summary><mark style="color:green;"><strong>Dockerfile Analysis</strong></mark></summary>

#### <mark style="color:blue;">Base Image</mark>

```docker
ARG BASE_IMAGE=nvcr.io/nvidia/pytorch
ARG BASE_TAG=23.08-py3
FROM ${BASE_IMAGE}:${BASE_TAG} as base
```

* Defines two arguments <mark style="color:yellow;">**`BASE_IMAGE`**</mark> and <mark style="color:yellow;">**`BASE_TAG`**</mark>, which are used to specify the base image. In this case, it's using NVIDIA's PyTorch image.
* The <mark style="color:yellow;">**`FROM`**</mark> instruction initialises a new build stage and sets the base image for subsequent instructions. The <mark style="color:yellow;">**`as base`**</mark> names this stage as <mark style="color:yellow;">**`base`**</mark>.

#### <mark style="color:blue;">Setting Up Bash Environment</mark>

```docker
ENV BASH_ENV="/tmp/bash_env"
SHELL ["/bin/bash", "-c"]
```

* Sets an environment variable <mark style="color:yellow;">**`BASH_ENV`**</mark> pointing to a file. This file <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`/tmp/bash_env`**</mark><mark style="color:yellow;">**)**</mark> will be sourced whenever a bash shell is started non-interactively.
* Changes the default shell to bash.

#### &#x20;<mark style="color:blue;">Development Stage</mark>

```docker
FROM base as devel
```

* Starts a new stage named `devel` based on the `base` stage.

#### <mark style="color:blue;">Copying and Running Scripts</mark>

The following set of instructions:

* Copies various shell scripts from the host to the container.
* Executes these scripts to install different software components.
* Each script is removed after its execution.

<mark style="color:blue;">**These installations include**</mark>

* Base dependencies (<mark style="color:yellow;">**`install_base.sh`**</mark>).
* CMake (<mark style="color:yellow;">**`install_cmake.sh`**</mark>).
* TensorRT (<mark style="color:yellow;">**`install_tensorrt.sh`**</mark>) - an NVIDIA library for high-performance deep learning inference.
* Polygraphy (<mark style="color:yellow;">**`install_polygraphy.sh`**</mark>) - a toolkit for working with TensorRT.
* mpi4py (<mark style="color:yellow;">**`install_mpi4py.sh`**</mark>) - MPI for Python.
* PyTorch (<mark style="color:yellow;">**`install_pytorch.sh`**</mark>) - the torch installation can be controlled by the <mark style="color:yellow;">**`TORCH_INSTALL_TYPE`**</mark> argument.

Environment variables like <mark style="color:yellow;">**`RELEASE_URL_TRT`**</mark>, <mark style="color:yellow;">**`TARGETARCH`**</mark>, and <mark style="color:yellow;">**`TORCH_INSTALL_TYPE`**</mark> are used to control the behavior of the scripts.

#### <mark style="color:blue;">Wheel Stage</mark>

```docker
FROM devel as wheel
```

* Initiates a new stage named <mark style="color:yellow;">**`wheel`**</mark> based on the <mark style="color:yellow;">`devel`</mark> stage.

```docker
WORKDIR /src/tensorrt_llm
...
RUN python3 scripts/build_wheel.py ${BUILD_WHEEL_ARGS}
```

* Sets the working directory.
* <mark style="color:yellow;">Copies various directories and files from the host into the container</mark>.
* Runs a Python script to build a wheel file of the application, controlled by <mark style="color:yellow;">**`BUILD_WHEEL_ARGS`**</mark><mark style="color:yellow;">.</mark>

&#x20;<mark style="color:blue;">Release Stage</mark>

```docker
FROM devel as release
```

* Initiates the final stage named `release`, again based on the `devel` stage.

```docker
WORKDIR /app/tensorrt_llm
COPY --from=wheel /src/tensorrt_llm/build/tensorrt_llm*.whl .
...
RUN pip install tensorrt_llm*.whl && \
    rm tensorrt_llm*.whl
```

* Sets another working directory.
* Copies the wheel file built in the `wheel` stage and installs it using `pip`.
* Removes the wheel file after installation.

```docker
COPY README.md ./
COPY examples examples
```

* Copies `README.md` and the `examples` directory to the container.

</details>

{% hint style="warning" %}
The build process will take some time
{% endhint %}

### <mark style="color:blue;">Fire up the Docker Container</mark>

Once built, <mark style="color:yellow;">**execute the Docker container**</mark> using <mark style="color:yellow;">**`make -C docker release_run`**</mark>.

```bash
make -C docker release_run
```

* To run as a local user instead of root, use <mark style="color:yellow;">**`LOCAL_USER=1`**</mark>.

<details>

<summary><mark style="color:green;">Analysis of Dockerfile process</mark></summary>

The provided output is from running a Docker container for NVIDIA's TensorRT-LLM

#### Docker Run Command Breakdown

The <mark style="color:yellow;">**`docker run`**</mark> command is used to create and start a container. The command and its options are as follows:

<mark style="color:yellow;">**`--rm`**</mark>: Automatically remove the container when it exits.

<mark style="color:yellow;">**`-it`**</mark><mark style="color:yellow;">**:**</mark> Run the container in interactive mode (i.e., attached to the terminal) and allocate a pseudo-TTY.

<mark style="color:yellow;">**`--ipc=host`**</mark>: Use the host's IPC namespace, which allows the container to share memory with the host.

<mark style="color:yellow;">**`--ulimit memlock=-1 --ulimit stack=67108864`**</mark>: Set certain limits on system resources. Here, <mark style="color:yellow;">**`memlock=-1`**</mark> removes the memory lock limit, and <mark style="color:yellow;">**`stack=67108864`**</mark> sets the stack size.

<mark style="color:yellow;">**`--gpus=all`**</mark>: Allocate all available GPUs to the container. This is important for machine learning tasks that require GPU acceleration.

<mark style="color:yellow;">**`--volume /home/jack/TensorRT-LLM:/code/tensorrt_llm`**</mark>: Mount the host directory `/home/jack/TensorRT-LLM` to the container directory `/code/tensorrt_llm`.

<mark style="color:yellow;">**`--workdir /code/tensorrt_llm`**</mark>: Set the working directory inside the container to <mark style="color:yellow;">**`/code/tensorrt_llm`**</mark>.

<mark style="color:yellow;">**`--hostname-laptop-release`**</mark>: Set the hostname of the container.

<mark style="color:yellow;">**`--name tensorrt_llm-release-jack`**</mark><mark style="color:yellow;">**:**</mark> Assign a name to the container for easy reference.

<mark style="color:yellow;">**`--tmpfs /tmp:exec`**</mark><mark style="color:yellow;">**:**</mark> Mount a temporary file system (<mark style="color:yellow;">**`tmpfs`**</mark>) at <mark style="color:yellow;">**`/tmp`**</mark> with execution permissions. This can be used for temporary storage that's faster than writing to disk.

<mark style="color:yellow;">**`tensorrt_llm/release:latest`**</mark><mark style="color:yellow;">**:**</mark> The Docker image to use, where `tensorrt_llm/release` is the image name and <mark style="color:yellow;">**`latest`**</mark> is the tag.

</details>

<details>

<summary><mark style="color:green;">What is GNU?  What does the MAKE command do?</mark></summary>

<mark style="color:yellow;">`make`</mark> and <mark style="color:yellow;">GNU</mark> are fundamental concepts in software development, particularly in the context of building and compiling code.

#### <mark style="color:green;">What is</mark> <mark style="color:green;"></mark><mark style="color:green;">`make`</mark><mark style="color:green;">?</mark>

<mark style="color:purple;">**Function**</mark><mark style="color:purple;">:</mark> <mark style="color:yellow;">**`make`**</mark> is a utility that automatically builds executable programs and libraries from source code by reading files called <mark style="color:yellow;">**`Makefiles`**</mark> which specify how to derive the target program.

<mark style="color:purple;">**Automation**</mark><mark style="color:purple;">:</mark> It helps in automating the compilation process, reducing the complexity and potential errors in building software, especially large projects with multiple components and dependencies.

<mark style="color:purple;">**Efficiency**</mark><mark style="color:purple;">:</mark> <mark style="color:yellow;">**`make`**</mark> determines which portions of a program need to be recompiled and issues commands to recompile them. This is efficient because only those parts of a program that have been modified are recompiled, saving time.

<mark style="color:purple;">**Platform**</mark><mark style="color:purple;">:</mark> It is widely used in Unix and Unix-like systems but is available for many other operating systems.

#### <mark style="color:green;">What is GNU?</mark>

<mark style="color:purple;">**GNU Project**</mark><mark style="color:purple;">:</mark> GNU stands for "GNU's Not Unix!" It's a recursive acronym and is part of the GNU Project, which was launched in 1983 by Richard Stallman to create a complete, free operating system.

<mark style="color:purple;">**Free Software**</mark><mark style="color:purple;">:</mark> The GNU Project has developed a comprehensive collection of free software. When people refer to “GNU software”, they are usually referring to software released under the GNU General Public License (GPL), which is known for its commitment to free software principles.

<mark style="color:purple;">**GNU Tools**</mark><mark style="color:purple;">:</mark> The project has produced a number of tools widely used in software development, including the GNU Compiler Collection (GCC), GNU Debugger (GDB), and GNU Make (a version of the `make` utility).

<mark style="color:purple;">**GNU/Linux**</mark><mark style="color:purple;">:</mark> The combination of GNU tools and the Linux kernel resulted in the GNU/Linux operating system, commonly referred to as just “Linux”, which is used in systems around the world.

#### <mark style="color:green;">GNU Make</mark>

<mark style="color:purple;">**GNU Make**</mark><mark style="color:purple;">:</mark> In the context of your query, GNU Make is a version of <mark style="color:yellow;">**`make`**</mark> utility developed by the GNU Project. It is an enhanced version of the original <mark style="color:yellow;">**`make`**</mark> utility and is more feature-rich and portable.

<mark style="color:purple;">**Usage in TensorRT-LLM**</mark><mark style="color:purple;">:</mark> The <mark style="color:yellow;">**`make`**</mark> commands you mentioned in the TensorRT-LLM build process are using GNU Make. This tool simplifies the building process by reading the specified <mark style="color:yellow;">**`Makefile`**</mark> to automate the compilation and linking of the TensorRT-LLM software.

In summary, <mark style="color:yellow;">**`make`**</mark> is a tool for automating the build process in software development, and GNU is an organization that provides a variety of free software tools, including GNU Make.

</details>