FasterTransfomer Library

https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/

NVIDIA Triton Inference Server's FasterTransformer (FT) library is designed for accelerated inference of large transformer models. Here are the five key points:

Introduction to FasterTransformer and its benefits: FasterTransformer is a library for distributed inference of transformers with many parameters, even reaching trillions. It is considered one of the fastest libraries available for this purpose. The text provides an overview of the library and highlights the advantages of using it.

Importance and versatility of transformers: Transformers have become influential AI model architectures, used in various domains such as natural language processing, computer vision, speech recognition, and financial data processing. The attention mechanism, a key component of transformers, enhances computational efficiency, quality, and accuracy of models.

Challenges of training large transformer models: Large transformer-based models with hundreds of billions of parameters contain extensive knowledge and offer opportunities for one-shot or few-shot learning techniques. However, training such models can be challenging due to memory limitations. Open-source tools like the NeMo framework help optimize the training process.

NVIDIA Triton Inference Server for accelerated inference: The NVIDIA Triton Inference Server is an open-source software that standardizes model deployment and execution, enabling fast and scalable AI in production. Triton supports various model backends, including PyTorch, Tensorflow, ONNX Runtime, and OpenVINO.

Features and compatibility of FasterTransformer: The FT library includes a highly optimized version of the transformer block, encompassing both encoder and decoder parts. It supports full encoder-decoder architectures like T5, encoder-only models like BERT, and decoder-only models like GPT.

FT is built using C++/CUDA and leverages optimized libraries such as cuBLAS, cuBLASLt, and cuSPARSELt. It offers distributed inference support for large transformer models through techniques like tensor parallelism and pipeline parallelism. Integration options include TensorFlow, PyTorch, and Triton, with multi-GPU and multi-node support. FT is compatible with GPUs with compute capability >= 7.0.

FasterTransformer (FT) enables faster inference pipeline with lower latency and higher throughput compared to common deep learning training frameworks. It is optimized for transformer-based neural networks like GPT-3 and other large models.

Optimization techniques in FT include layer fusion, which combines multiple layers of neural networks into a single one to reduce data transfer and increase computational efficiency. Caching mechanisms are employed to prevent recomputing previous keys and values for autoregressive models. Memory optimization techniques are used to reduce memory usage for large transformer models.

FT utilizes MPI and NCCL for inter and intra-node communication, enabling support for model parallelism. Tensor parallelism and pipeline parallelism are utilized in GPT models, splitting weights and batching requests to optimize computation and hide communication overhead.

MatMul kernel autotuning is employed to benchmark and select the best low-level algorithms for matrix multiplication operations. FT supports inference with lower precisions (fp16 and int8) to accelerate computation and leverage specialized hardware.

The FasterTransformer library provides additional features such as a fast C++ BeamSearch implementation and optimized all-reduce for TensorParallelism mode. Currently, Triton with FT backend supports models like GPT-J, GPT-Megatron, and T5.

FasterTransfomer Library

The statement refers to two optimization techniques employed in the FasterTransformer (FT) library: MatMul kernel autotuning and support for lower precisions in inference.

MatMul kernel autotuning: Matrix multiplication is a fundamental operation in transformer-based neural networks. The FT library uses the MatMul kernel autotuning technique to benchmark and select the best low-level algorithms for matrix multiplication operations. MatMul operation can be executed in various ways using different low-level algorithms at the hardware level.

The library performs a real-time benchmark of these algorithms based on the model's parameters (e.g., attention layers, number of heads, hidden layer size) and input data. It then chooses the most efficient algorithm for the given configuration, optimizing the performance of matrix multiplication operations.

Support for lower precisions: FT supports inference using lower precisions, specifically fp16 (half-precision) and int8 (8-bit integer). Inference with lower precisions can accelerate computation and leverage specialized hardware.

For example, tensor cores in GPUs starting from Volta architecture are specifically designed to handle fp16 computations efficiently. By using lower-precision input data, FT reduces the amount of data transfer and required memory, leading to faster inference. This optimization allows for increased throughput and improved performance on specialized hardware such as tensor cores or upcoming GPUs like the transformers engine in Hopper GPUs.