Persistence

To ensure that your Docker container retains the TensorRT-LLM library and any installed models, you need to create a persistent environment within the Docker container.

This involves a few key steps:

This involves a few key steps:

Using Docker Volumes for Persistence

When you run a Docker container and install software or make changes within it, these changes are lost once the container is stopped or removed unless you've set up a persistent storage solution.

Docker volumes are the preferred way to persist data in Docker containers.

• Mount a Docker Volume: When you run your container, mount a volume to a specific path inside the container.

• This volume will store all the data you want to persist, such as the installed TensorRT-LLM library and models.

Example command:

docker run --rm -it \
           --gpus=all \
           --volume my_tensorrt_llm_data:/path/in/container \
           tensorrt_llm/devel:latest

Replace my_tensorrt_llm_data with your volume name and /path/in/container with the path where you want to persist data inside the container.

Building and Installing TensorRT-LLM Inside the Container

Once you have your development container running with the mounted volume, proceed to build and install TensorRT-LLM.

• Build the TensorRT-LLM: Use the provided build_wheel.py script to compile the TensorRT-LLM from source.

• Ensure you are doing this within the directory that is mounted to your Docker volume.

python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt

• Install the TensorRT-LLM: After building, install the TensorRT-LLM library using pip. This should also be done within the mounted volume directory.

pip install ./build/tensorrt_llm*.whl

Persisting the Environment

Every time you want to work with TensorRT-LLM, make sure to run the container with the volume attached. This ensures that the installed library and any models you've added will be available in subsequent sessions.

Committing Changes to a New Docker Image (Optional)

Alternatively, you can commit the changes made in your container to a new Docker image. This way, the environment with the installed TensorRT-LLM is saved in a new image, and you can use this image directly in the future.

• Commit the Container to a New Image: After installing TensorRT-LLM in the container, open a new terminal and use the docker commit command to create a new image from your container's current state.

docker commit [CONTAINER_ID] my_tensorrt_llm_image:latest

Using the New Image or Mounted Volume

For future use, you can either run a container from the new image you created or keep using the original image with the volume mounted.

Both methods will retain the installed TensorRT-LLM library and models.

By following these steps, you will create a Docker environment where the TensorRT-LLM library and its models persist between container runs, ensuring that your setup is saved and reusable.