NVIDIA GPU Infrastructure Sizing
Required VRAM / Node
140 GB
KV Cache + Model Weights
NVIDIA HGX H100 Nodes
4 Nodes
32x SXM5 80GB GPUs
Max Sustainable QPS:
42.5 req/s
Token Budget per Request:
4,850 toks
Deployment Blueprint (YAML)
apiVersion: nim.nvidia.com/v1alpha1
kind: AgentDeployment
metadata:
name: semantic-supervisor-prod
spec:
orchestrator: semantic-kernel-azure
models:
- name: meta/llama-3.3-70b-instruct
engine: tensorrt-llm
tensor_parallel_size: 4
guardrails:
engine: nemo-guardrails
config:
jailbreak_detection: true
hallucination_rail: true
concurrency_target_qps: 35