NVIDIA offers TensorRT
Product
High-performance deep learning inference SDK for optimizing trained models on NVIDIA GPUs.
- Currently Offered
High-performance deep learning inference SDK for optimizing trained models on NVIDIA GPUs.
TensorRT is NVIDIA's software development kit for high-throughput, low-latency deep learning inference. The runtime compiles trained models from PyTorch, TensorFlow, ONNX, and JAX into optimized engines targeting NVIDIA Hopper, Blackwell, RTX, and Jetson GPUs through quantization, kernel auto-tuning, and graph fusion.
TensorRT-LLM extends the runtime to large language model serving with in-flight batching, paged attention, and speculative decoding, while TensorRT for RTX targets Windows PCs. The SDK is consumed by Triton Inference Server, NVIDIA NIM microservices, and direct integrations across enterprise AI deployments.
InferenceX continuously benchmarks LLM inference throughput, latency, and cost across GPU vendors.
Open-source hybrid edge-cloud AI inference engine for mobile devices and wearables.
Open-source deep learning training platform with GPU orchestration, hyperparameter search, and distributed training.
An integrated compute, training, inference, and sandbox stack for enterprises to train and deploy their own AI models.
Baseten Dedicated Inference is a high-performance inference platform for deploying, serving, and scaling custom and open-source AI models in production.
Unified production inference platform with gateway, router, cache, observability, evaluation, fine-tuning, and custom GPU hosting.
Edge AI Studio helps developers train, optimize, and deploy AI models on Texas Instruments edge devices.
Edge software that runs AI models locally for real-time decisions, including offline.
Intel oneAPI is a unified open multiarchitecture software stack for AI and accelerated computing across Intel CPUs, GPUs, and accelerators.
InferenceX continuously benchmarks LLM inference throughput, latency, and cost across GPU vendors.
Open-source hybrid edge-cloud AI inference engine for mobile devices and wearables.
Open-source deep learning training platform with GPU orchestration, hyperparameter search, and distributed training.
KV-cache transfer engine that prefills small models and decodes large ones.
An integrated compute, training, inference, and sandbox stack for enterprises to train and deploy their own AI models.
Baseten Dedicated Inference is a high-performance inference platform for deploying, serving, and scaling custom and open-source AI models in production.
Unified production inference platform with gateway, router, cache, observability, evaluation, fine-tuning, and custom GPU hosting.
Meta's on-device machine-learning runtime for mobile and edge devices.
MLPerf is an industry-standard suite that measures AI training and inference system performance.
Hermes is PrismML's agentic inference stack for running on-device models.
Edge AI Studio helps developers train, optimize, and deploy AI models on Texas Instruments edge devices.
Edge software that runs AI models locally for real-time decisions, including offline.
Intel oneAPI is a unified open multiarchitecture software stack for AI and accelerated computing across Intel CPUs, GPUs, and accelerators.
