> For the complete documentation index, see [llms.txt](https://tensorrt-llm.continuumlabs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://tensorrt-llm.continuumlabs.ai/cuda-introduction/installing-cuda.md).

# Installing Cuda

CUDA 12.3 should have already been installed as part of the virtual machine creation.

### <mark style="color:blue;">Check the installation of the</mark> <mark style="color:yellow;">NVIDIA CUDA Toolkit</mark>

<mark style="color:green;">**What is the CUDA Toolkit?**</mark>

The NVIDIA CUDA Toolkit provides a <mark style="color:yellow;">development environment</mark> for creating high performance GPU-accelerated applications.&#x20;

With the CUDA Toolkit, you can develop, optimise, and deploy your applications on GPU-accelerated systems.

The toolkit includes GPU-accelerated libraries, debugging and optimisation tools, a C/C++ compiler, and a runtime library to deploy your application.

If not installed, follow the instructions below.

### <mark style="color:blue;">Installing the NVIDA CUDA Toolkit</mark>

First, check to see if the CUDA Toolkit is installed, we can check to see whether the core compiler is installed, the NVIDIA CUDA Compiler (NVCC)

<mark style="color:blue;">**Nvidia CUDA Compiler (NVCC)**</mark> is a part of the CUDA Toolkit.  It is the <mark style="color:yellow;">compiler for CUDA</mark>, responsible for *<mark style="color:yellow;">**translating CUDA code into executable programs.**</mark>*  &#x20;

NVCC takes high-level CUDA code and turns it into a form that can be understood and executed by the GPU. &#x20;

It handles the partitioning of code into segments that can be run on either the CPU or GPU, and manages the compilation of the GPU parts of the code.

First, to check if <mark style="color:yellow;">NVCC is installed</mark> and its version, run

```bash
nvcc --version
```

If the NVIDIA CUDA Toolkit has been installed, the output will look something like this:

<pre class="language-bash"><code class="lang-bash">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 12.1, V11.8.89
<a data-footnote-ref href="#user-content-fn-1">Build cuda_</a>12.1.r11.8/compiler.31833905_0
</code></pre>

You should see <mark style="color:yellow;">release 12.3</mark> - which indicates the <mark style="color:yellow;">CUDA Toolkit Version 12.3</mark> has been successfully installed.

### <mark style="color:blue;">Install Cuda 12.3</mark>

{% embed url="<https://developer.nvidia.com/cuda-12-3-0-download-archive?Distribution=Ubuntu&target_arch=x86_64&target_os=Linux&target_type=deb_local&target_version=22.04>" %}
CUDA 12.3
{% endembed %}

<figure><img src="/files/DVf1lAcR7BMPxJ4X7iy8" alt=""><figcaption><p>Enter in the requirements and the instructions will appear</p></figcaption></figure>

#### <mark style="color:green;">Download the CUDA keyring package</mark>

{% code overflow="wrap" %}

```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
```

{% endcode %}

* This command uses `wget` to <mark style="color:yellow;">download the CUDA keyring package</mark> from the specified URL.
* The CUDA keyring package is a <mark style="color:yellow;">**digital signature package that ensures the authenticity**</mark> of the CUDA repositories.

#### <mark style="color:green;">This command  installs the downloaded CUDA keyring package:</mark>

```bash
sudo dpkg -i cuda-keyring_1.1-1_all.deb
```

The `dpkg -i` flag is used to <mark style="color:yellow;">install the downloaded CUDA keyring package</mark> (`cuda-keyring_1.0-1_all.deb`).

Installing the keyring package adds the CUDA repository's GPG key to the system's keyring, allowing the package manager to verify the integrity of the CUDA packages.

#### <mark style="color:green;">Update package lists from repositories on system including the newly added CUDA repository</mark>

```bash
sudo apt-get update
```

#### <mark style="color:green;">This command will then install the CUDA package and its dependencies on your system</mark>

```bash
sudo apt-get -y install cuda-toolkit-12-3
```

* The `-y` flag automatically answers "yes" to any prompts during the installation process, allowing the installation to proceed without user intervention.
* The `cuda` package is a meta-package that installs the CUDA Toolkit, including the CUDA libraries, runtime, and development tools.

<details>

<summary><mark style="color:yellow;">Make sure CUDA is on PATH</mark></summary>

### <mark style="color:blue;">**Verify the Installation**</mark><mark style="color:blue;">:</mark>

Ensure that CUDA was installed correctly without errors. You can check the installation logs for any errors or warnings that might have occurred during the installation process.

Use the command to <mark style="color:yellow;">list all CUDA-related packages</mark> installed and their versions.  Enter into the terminal:

```bash
dpkg -l | grep cuda
```

This will provide a list of all CUDA related packages:

Below are some example files that this command highlights, for example:

<mark style="color:blue;">**cuda**</mark> These are meta-packages for CUDA. Installing a meta-package <mark style="color:yellow;">installs all the components of CUDA</mark>

<mark style="color:blue;">**cuda-cccl**</mark><mark style="color:blue;">:</mark> CUDA CCCL (CUDA C++ Core Library) is part of the CUDA Toolkit and provides <mark style="color:yellow;">essential libraries for CUDA C++ development.</mark>

<mark style="color:blue;">**cuda-command-line-tools**</mark><mark style="color:blue;">:</mark> This package includes <mark style="color:yellow;">command-line tools for CUDA,</mark> such as `nvcc` (NVIDIA CUDA Compiler), which is crucial for compiling CUDA code.

<mark style="color:blue;">**cuda-compiler**</mark><mark style="color:blue;">:</mark> This package includes the <mark style="color:yellow;">CUDA compiler,</mark> which is essential for converting CUDA code into code that can run on Nvidia GPUs.

<mark style="color:blue;">**cuda-demo-suite**</mark><mark style="color:blue;">:</mark> This package <mark style="color:yellow;">contains demos showcasing the capabilities</mark> and features of CUDA.

<mark style="color:blue;">**cuda-documentation**</mark><mark style="color:blue;">:</mark> Provides the <mark style="color:yellow;">documentation for CUDA,</mark> useful for developers to understand and use CUDA APIs.

<mark style="color:blue;">**cuda-driver-dev**</mark><mark style="color:blue;">:</mark> Includes <mark style="color:yellow;">development resources for the CUDA driver,</mark> such as headers and stub libraries.

<mark style="color:blue;">**cuda-libraries, cuda-libraries-dev**</mark><mark style="color:blue;">:</mark> These meta-packages include <mark style="color:yellow;">libraries necessary for CUDA development and their development counterparts.</mark>

<mark style="color:blue;">**cuda-nvcc**</mark><mark style="color:blue;">:</mark> NVIDIA CUDA Compiler (NVCC) is a tool for compiling CUDA code.

### <mark style="color:blue;">**Check Environment Variables**</mark><mark style="color:blue;">:</mark>

* Ensure that your environment variables are pointing to the new CUDA installation. Specifically, check the <mark style="color:yellow;">`PATH`</mark> and <mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> environment variables via the following commands:

```bash
echo $PATH
```

```bash
echo $LD_LIBRARY_PATH
```

Update these variables if they are pointing to an older CUDA version.&#x20;

If CUDA is not on PATH:

### <mark style="color:blue;">**Export PATH**</mark>

```bash
export PATH=/usr/local/cuda-12.3/bin${PATH:+:${PATH}}
```

This command adds <mark style="color:yellow;">`/usr/local/cuda-12.3/bin`</mark> to the `PATH` environment variable.

<mark style="color:yellow;">`PATH`</mark> is a list of directories the shell searches for executable files. Adding CUDA's bin directory makes it possible to run CUDA tools and compilers directly from the command line without specifying their full path.

`${PATH:+:${PATH}}` is a <mark style="color:yellow;">shell parameter expansion pattern</mark> that appends the existing `PATH` variable to the new path. If <mark style="color:yellow;">`PATH`</mark> is unset, nothing is appended.

### <mark style="color:blue;">LD\_LIBRARY\_PATH Variable Setup</mark>

**Export LD\_LIBRARY\_PATH for 64-bit OS**:

{% code overflow="wrap" %}

```bash
export LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

{% endcode %}

<mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> is an environment variable specifying directories where libraries are searched for first, before the standard set of directories.

This command <mark style="color:yellow;">adds the</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`lib64`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">directory of the CUDA installation to</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`LD_LIBRARY_PATH`</mark>, which is necessary for 64-bit systems. It ensures that the system can find and use the CUDA libraries.

### <mark style="color:blue;">**Verify NVCC Version**</mark><mark style="color:blue;">:</mark>

After updating the environment variables, check the NVCC version again using:

```bash
nvcc --version
```

This should reflect the new version if the environment variables are set correctly.

**Update Alternatives**:

* Sometimes, multiple versions of CUDA can coexist, <mark style="color:yellow;">and the system may still use the old version.</mark> Use `update-alternatives` to configure the default CUDA version.
* Run `sudo update-alternatives --config nvcc` to choose the correct version of NVCC.

```bash
sudo update-alternatives --config
```

</details>

After the installation is complete, you will have the CUDA Toolkit and runtime installed on your Ubuntu environment, enabling you to develop and run CUDA-accelerated applications using NVIDIA GPUs.

[^1]: This is the version of CUDA


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://tensorrt-llm.continuumlabs.ai/cuda-introduction/installing-cuda.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.