NVIDIA offers Triton Inference Server
Product
Open-source inference serving platform for deploying AI models from any framework on CPUs and GPUs.
- Currently Offered
Open-source inference serving platform for deploying AI models from any framework on CPUs and GPUs.
Triton Inference Server is NVIDIA's open-source platform for serving AI models at production scale. The runtime hosts models from TensorRT, TensorRT-LLM, PyTorch, TensorFlow, ONNX, OpenVINO, vLLM, and custom Python backends on a single endpoint with dynamic batching, model ensembles, and concurrent model execution.
Kubernetes operators and integrations with NVIDIA AI Enterprise, NIM microservices, and managed services on AWS, Azure, Google Cloud, and Oracle Cloud allow Triton to scale across GPU fleets. The server is the default inference layer in many enterprise AI factories.
ONNX Runtime is a cross-platform inference and training accelerator for machine-learning models.
Baseten Dedicated Inference is a high-performance inference platform for deploying, serving, and scaling custom and open-source AI models in production.
Open-source deep learning training platform with GPU orchestration, hyperparameter search, and distributed training.
An integrated compute, training, inference, and sandbox stack for enterprises to train and deploy their own AI models.
AI inference platform delivering serverless and dedicated GPU inference with top-ranked speed and price-performance.
Unified production inference platform with gateway, router, cache, observability, evaluation, fine-tuning, and custom GPU hosting.
Pipeshift's platform provides infrastructure for fine-tuning, inference, and orchestration of open-source AI models.
Open-source hybrid edge-cloud AI inference engine for mobile devices and wearables.
Baseten Inference Stack for high performance with custom kernels, decoding, caching on optimized infra.
IonRouter is a high-throughput, low-cost inference API and drop-in OpenRouter alternative powered by Cumulus Labs' IonAttention engine.
InferenceX continuously benchmarks LLM inference throughput, latency, and cost across GPU vendors.
Kneron's Edge AI Platform delivers on-device NPUs and full-stack software for edge inference.
Crusoe Managed Inference deploys and scales production LLM workloads with breakthrough speed and throughput.
Train models on B200s, T4s and inference-optimized infra with one-click deploy to production.
Hugging Face Hub is the leading platform for sharing, discovering, and deploying machine learning models and datasets.
ONNX Runtime is a cross-platform inference and training accelerator for machine-learning models.
Baseten Dedicated Inference is a high-performance inference platform for deploying, serving, and scaling custom and open-source AI models in production.
Open-source deep learning training platform with GPU orchestration, hyperparameter search, and distributed training.
An integrated compute, training, inference, and sandbox stack for enterprises to train and deploy their own AI models.
AI inference platform delivering serverless and dedicated GPU inference with top-ranked speed and price-performance.
Unified production inference platform with gateway, router, cache, observability, evaluation, fine-tuning, and custom GPU hosting.
Pipeshift's platform provides infrastructure for fine-tuning, inference, and orchestration of open-source AI models.
Managed GPU cloud for inference, fine-tuning, and prompt workflows.
Open-source hybrid edge-cloud AI inference engine for mobile devices and wearables.
Baseten Inference Stack for high performance with custom kernels, decoding, caching on optimized infra.
IonRouter is a high-throughput, low-cost inference API and drop-in OpenRouter alternative powered by Cumulus Labs' IonAttention engine.
InferenceX continuously benchmarks LLM inference throughput, latency, and cost across GPU vendors.
Kneron's Edge AI Platform delivers on-device NPUs and full-stack software for edge inference.
KV-cache transfer engine that prefills small models and decodes large ones.
Crusoe Managed Inference deploys and scales production LLM workloads with breakthrough speed and throughput.
Train models on B200s, T4s and inference-optimized infra with one-click deploy to production.
Hugging Face Hub is the leading platform for sharing, discovering, and deploying machine learning models and datasets.
Custom AI model builder for enterprise infrastructure.
