LLM Performance Tuning: Track Joules per Token On Prem

Technical sketch framing the article title

Benchmark first, then tune in order: scheduler and batching, parallelism, quantization and KV cache, each validated against a latency or energy budget. Measure Time to First Token (TTFT), Inter-Token Latency (ITL/TPOT), throughput in tokens per second, and Joules per token before changing anything. The config that wins is the one sitting on the time-energy Pareto frontier for your actual SLO, not the one with the best headline throughput number.

TL;DR:

Pin hardware and software settings, warm up caches, test three prompt shapes, repeat runs across seeds, and report medians plus tail percentiles.

Choose metrics by workload: TTFT for chat, ITL/TPOT for streaming, throughput for offline jobs; select the lowest energy configuration that meets its latency SLO.

For bursty traffic, chunked prefills interleaved with decode steps reduce blocking; use smaller chunks under tight TPOT limits and inspect tail latency, not averages.

Tensor parallelism often lowers latency and energy; pipeline parallelism can raise throughput for oversized models but adds idle stage time and may require more activation memory.

For memory pressure from long contexts or concurrency, quantize the KV cache first, then validate task relevant accuracy and rerun latency and energy benchmarks.

Benchmarking workflow you can run today
Core performance metrics and how to prioritize them
Batching and scheduler tactics that balance latency and throughput
Parallelism and distribution: picking TP/PP/EP/DP and managing NUMA
Quantization and KV cache strategies that free HBM and reduce latency
Observability and instrumentation: measuring power and constructing Pareto frontiers
Forge’s practitioner playbook for enterprise on-prem tuning and operations
Where LLM inference tuning is headed
Forge AI Deployment services for secure, high-performance LLM inference
FAQ
Sources

Benchmarking workflow you can run today

A tuning decision is only as good as the benchmark behind it. Before touching batch sizes or parallelism settings, lock down an environment and a measurement routine you can repeat weeks later and trust.

Start with the infrastructure layer:

Pin GPU clock and power states, driver version, and container image so a rerun six weeks from now uses the same stack.
Warm up the engine and let kernel compilation or graph caches settle before recording anything; the first few requests are not representative.
Reuse compiled artifacts (CUDA graphs, kernel autotuning caches) across runs so compile time does not bleed into your latency numbers.
Run each configuration with multiple seeds and multiple request batches, then report medians and tail percentiles (p50, p95, p99) rather than a single average.
Save raw logs, configs, and environment metadata for every run so a result is reproducible without guesswork months later.

Request shape matters as much as hardware settings. A benchmark built on short chat turns will tune your system toward fast decode and will mislead you if your real traffic is long-document summarization with heavy prefill. Build at least two or three representative prompt sets:

A short-prompt, short-response set that mimics interactive chat.
A long-prompt, short-response set that stresses prefill (document QA, RAG context stuffing).
A long-generation set that stresses decode (code generation, report writing).

For tooling, engine-native benchmarking utilities such as trtllm-bench or the serving benchmarks bundled with vLLM give you request-level latency and throughput numbers under controlled concurrency. Layer in energy-aware tooling where available: the ML.ENERGY Benchmark runs configuration sweeps across GPU, parallelism, and batch size, measures time and energy using Zeus, then builds a Pareto frontier that recommends the minimal-energy configuration meeting a latency target. TokenPowerBench takes a similar phase-aware approach, attributing energy to prefill and decode separately and supporting sweeps across vLLM, TensorRT-LLM, DeepSpeed, and Transformers.

A repeatable workflow looks like this: fix the environment, run the baseline config against all three prompt sets, record TTFT, ITL, throughput, and Joules per token for each, then change exactly one variable (batch size, parallelism degree, or quantization format) before the next run. Changing two variables at once is the single most common reason teams can’t explain a regression three weeks later.

Controlled benchmark runs compare energy per token

Pro Tip: Keep a plain-text changelog next to your benchmark artifacts: one line per run naming the variable you changed and the result. It turns a folder of JSON files into a decision trail.

Automate the sweep itself. A script that iterates batch size, tensor parallel degree, and quantization mode in a loop, writing each result to a structured log, replaces days of manual trial and error with an unattended run. Benchmark contamination is also worth guarding against: validation sets that leak into training data, or prompt sets reused so often the engine’s cache behavior stops reflecting production traffic, both produce numbers that look better than reality. Our practitioner notes on reliable benchmarking cover common pitfalls in more depth.

Core performance metrics and how to prioritize them

Four metrics drive almost every tuning decision, and each means something different depending on whether your workload is prefill-heavy or decode-heavy.

Time to First Token (TTFT) measures the delay between a request arriving and the first output token appearing. It is dominated by the prefill phase, where the model processes the full input prompt in parallel. Long prompts, large batch queues, and KV cache allocation overhead all push TTFT up.

Inter-Token Latency (ITL), also called Time Per Output Token (TPOT), measures the delay between consecutive output tokens during decode. Decode is typically memory-bandwidth-bound rather than compute-bound, since each step generates one token per sequence but still has to read the full KV cache.

Throughput (TPS) is tokens generated per second across the system, and Requests Per Second (RPS) is completed requests per second. Both climb with larger batch sizes, but usually at the cost of ITL, since more concurrent sequences compete for the same memory bandwidth.

End-to-end latency is the full request duration, the sum of TTFT and the decode time across all output tokens. The NVIDIA NIM benchmarking documentation formalizes these definitions and flags a subtle but important decision: whether TTFT is counted inside or outside the ITL calculation, which changes comparisons across tools if left inconsistent.

TTFT matters most for interactive chat, where users notice the pause before the response starts.
ITL/TPOT matters most for streaming use cases, where a choppy token-by-token cadence is more noticeable than the initial delay.
Throughput and RPS matter most for batch jobs like offline summarization, where no human is watching a single request in real time.
End-to-end latency matters most for any SLA written in terms of total response time regardless of streaming behavior.

Reported findings show that inference running in production can account for the large majority of operational energy use in deployed LLM services, far outweighing training energy over the system’s lifetime. Inference energy studies back this framing, which is the main reason Joules per token deserves a permanent place next to latency on any performance dashboard rather than being treated as a one-time audit metric.

Workload type	Priority metric	Secondary metric
Interactive chat	TTFT	ITL/TPOT
Streaming code generation	ITL/TPOT	TTFT
Offline batch summarization	Throughput (TPS)	Joules per token
High-concurrency API serving	RPS	End-to-end latency

Joules per token, and its aggregate Joules per response, convert raw energy draw into a unit you can compare across configurations the same way you compare latency. A config that cuts TTFT by 15% but doubles energy per token is not a win if your deployment runs on a fixed power budget or a sustainability mandate. Treat energy as a metric to optimize, not a byproduct to measure after the fact.

Batching and scheduler tactics that balance latency and throughput

Static batching, where the scheduler waits to fill a batch before running any requests, wastes GPU cycles and inflates TTFT under bursty traffic. The fix adopted by modern serving engines is chunked-prefills combined with stall-free batching, described in detail in Sarathi-Serve. Instead of running an entire long prompt’s prefill in one blocking pass, the engine splits it into fixed-size chunks and interleaves those chunks with ongoing decode steps from other requests. No request waits for another’s prefill to finish, and the GPU never sits idle between decode steps.

A concrete example: a 4,000-token prompt split into chunks of 512 tokens runs across roughly eight scheduling iterations, with decode steps from other active requests slotted in between chunks rather than paused. Sarathi-Serve’s authors report multi-fold capacity improvements over naive batching under strict latency service-level objectives, because the scheduler keeps the GPU doing useful work instead of stalling on one large prefill.

Picking the right chunk size is a direct trade-off against your TPOT target:

Start from your TPOT SLO and work backward: a tighter decode-latency budget calls for smaller prefill chunks so decode steps are never blocked for long.
For short-prompt interactive traffic, small token budgets (in the low hundreds) preserve latency while still letting the scheduler batch multiple requests.
For long-document workloads with heavy prefill, larger chunks reduce scheduling overhead but should be capped so they don’t create visible decode stalls for concurrent users.
Re-run your benchmark sweep at each candidate chunk size and plot ITL distribution, not just the average, since stalls show up as tail latency spikes rather than a shifted mean.

Pro Tip: Plot ITL as a time series across a single long request rather than as a single aggregate number. Pipeline bubbles and chunking stalls show up as visible spikes in the trace that an averaged metric hides completely.

The experiment matrix worth running has three axes: chunk size, maximum batch size, and request arrival pattern (steady versus bursty). For each cell, capture TTFT, ITL distribution, throughput, and Joules per token, then plot throughput against ITL for each chunk size to see where the curve bends. Expected trade-off: smaller chunks protect decode latency but add scheduling overhead at high concurrency; larger chunks raise throughput ceilings but risk decode stalls if a long prefill chunk still takes too long relative to your TPOT target. The right answer depends on which SLO you are protecting, not on which configuration looks fastest in isolation.

Parallelism and distribution: picking TP/PP/EP/DP and managing NUMA

Parallelism strategy changes latency, throughput, and energy simultaneously, and the three rarely move in the same direction. Tensor parallelism (TP) splits individual layers across GPUs, which reduces per-token latency for large models because each GPU does less work per layer, but it adds communication overhead on every forward pass. Pipeline parallelism (PP) splits the model by layer groups across GPUs, which raises throughput for large batches but introduces bubbles: idle GPU time while later pipeline stages wait for earlier ones to finish a microbatch.

Expert parallelism (EP), used for mixture-of-experts models, routes tokens to different expert shards on different devices, which scales well with model size but depends heavily on routing balance; a skewed router load leaves some GPUs idle while others queue up. Data parallelism (DP) simply replicates the full model across devices to serve more concurrent requests, trading memory for throughput headroom.

TP tends to minimize per-token latency and often wins on energy efficiency for large batched jobs, since pipeline stages that sit idle waste power without producing tokens.
PP raises aggregate throughput on very large models that don’t fit on a single device’s memory, but every stage boundary is a potential bubble.
TokenPowerBench experiments show TP-heavy splits often yield lower Joules per token than long PP pipelines across common inference workloads, precisely because PP idle time doesn’t stop the GPU from drawing power.
Bubbles in PP are mitigated by increasing microbatch count so more work is in flight at once, filling the gaps between stages at the cost of higher memory use for activations.

NUMA and thread affinity matter more than most tuning guides admit, especially on multi-socket servers. When a GPU is attached to one CPU socket but its driver thread gets scheduled on the other socket, every host-to-device transfer crosses the socket interconnect and adds latency that has nothing to do with the model itself. Pin inference processes to the CPU cores local to their GPU’s PCIe root complex, and verify the binding with numactl or your orchestration layer’s NUMA-aware scheduling rather than assuming the container runtime got it right by default.

To test parallelism choices properly, build an experiment matrix across TP degree (1, 2, 4, 8), PP stage count where the model requires it, and batch size, then record TTFT, ITL, throughput, and Joules per token for each cell. The winning configuration is rarely the one with the highest raw throughput: it’s the one that meets your latency SLO at the lowest energy cost, which is exactly the trade-off a Pareto frontier analysis is built to surface.

Quantization and KV cache strategies that free HBM and reduce latency

KV cache size grows linearly with sequence length and batch size, and on long-context workloads it can consume more High Bandwidth Memory (HBM) than the model weights themselves. A larger cache footprint means fewer concurrent sequences fit in memory, which caps your effective batch size and throughput ceiling, while a cache that spills or fragments adds latency to both TTFT and TPOT.

Quantizing the KV cache, alongside weight and activation quantization through post-training quantization (PTQ), shrinks that footprint directly. NVIDIA’s work on NVFP4 KV cache formats reports that lower-precision cache formats reduce memory footprint meaningfully while keeping accuracy close to the full-precision baseline, which in practice means more concurrent long-context sequences fit on the same GPU.

Practical steps for rolling this out:

Quantize the KV cache before touching weights if your bottleneck is context length or concurrent sequence count rather than raw compute.
Apply weight and activation PTQ separately, since the two affect different parts of the latency profile: weight precision mostly affects decode throughput, while activation precision interacts with prefill compute.
Validate accuracy on a held-out task-relevant evaluation set after each quantization change, not just a generic perplexity check.
Re-run the full benchmark sweep (TTFT, ITL, throughput, Joules per token) after quantization, since memory savings don’t always translate into latency gains if compute becomes the new bottleneck.

Our detailed quantization guidance walks through calibration data selection and format trade-offs in more depth for teams building out a formal pipeline.

Pro Tip: Track cache-hit rate alongside latency when your serving stack supports prefix caching. A dropping hit rate under steady traffic is often the earliest warning sign of a memory-pressure problem, well before TTFT visibly degrades.

Guardrails matter as much as the technique. Set a maximum acceptable accuracy drop on your task-specific evaluation before you start, and treat it as a hard stop, not a target to approach. Watch for cache-induced stalls, which show up as periodic latency spikes correlated with cache eviction events rather than steady degradation, and are best caught by correlating your latency trace against cache occupancy metrics over the same time window.

Observability and instrumentation: measuring power and constructing Pareto frontiers

Power measurement has three practical levels, each with a different accuracy-effort trade-off.

GPU telemetry (NVML, nvidia-smi, or vendor equivalents) gives per-device power draw with minimal setup, but it misses host CPU, memory, and networking power, so it understates total system energy, particularly in multi-socket servers.
Zeus or IPMI-based measurement reads board-level or node-level power sensors, closing part of that gap and forming the backbone of energy benchmarks like the ML.ENERGY Benchmark, which measures time and energy together across configuration sweeps before building a Pareto frontier.
Rack-level PDU measurement captures the full power draw of everything plugged into that circuit, the most accurate total but the hardest to attribute to a single inference job running alongside other workloads.

Aligning power samples to inference phases is what turns raw wattage into something actionable. Timestamp the start and end of prefill and decode separately in your serving code, then join those timestamps against your power sampling stream, as TokenPowerBench’s methodology does, to attribute energy to each phase rather than reporting one blended average for the whole request.

Building a time-energy Pareto frontier follows a direct recipe: run your configuration sweep (GPU type, parallelism degree, batch size, quantization mode), record total time and total energy for each configuration under identical traffic, then plot energy on one axis against latency on the other. The points that no other configuration beats on both axes form the frontier. Pick the point on that frontier that satisfies your latency SLO with the lowest energy, which is exactly the recommendation process the ML.ENERGY Benchmark automates for teams running these sweeps regularly.

Instrumentation level	Granularity	Setup effort
GPU telemetry (NVML)	Per-device, GPU only	Low
Zeus / IPMI	Node-level, includes some host power	Medium
Rack PDU	Full circuit, highest accuracy	High

Automate the full loop: sweep script, power sampling, phase alignment, and Pareto plot generation, so a configuration change produces an updated frontier without a day of manual data wrangling. Our observability techniques for production LLM serving cover low-overhead tracing approaches that keep this instrumentation from becoming a latency tax of its own.

Forge’s practitioner playbook for enterprise on-prem tuning and operations

Air-gapped and sovereign deployments add constraints that standard tuning guides skip entirely. Package management has to work offline, which means pre-staging container images, Python wheels, and compiled kernel caches before a system ever reaches its production network. Compile caches built on one air-gapped node should be portable across identical hardware in the same environment, since rebuilding CUDA graphs or kernel autotuning results from scratch on every node wastes hours that regulated deployment windows rarely allow.

Operational discipline matters more than any single tuning trick in these environments:

Maintain reproducible runbooks for every benchmark and tuning change, versioned alongside the model and engine configuration they describe.
Keep tracing and telemetry overhead minimal, since observability tooling itself consumes compute and power budget that constrained edge or air-gapped hardware may not have to spare.
Size KV cache allocations against the HBM actually available on the deployed hardware, not a reference GPU spec sheet, since regulated environments often standardize on specific, sometimes older, hardware generations.
Validate every tuning change against the same accuracy and latency guardrails used in initial deployment approval, since re-certification requirements in regulated sectors often apply to performance changes, not just model changes.

This is where a systems integrator earns its place in the process. When compliance requirements demand data never leaves a defined perimeter, when reproducible benchmark evidence has to be documented for audit, or when an SLA-driven operations team needs to own tuning changes on an ongoing basis rather than a one-time project, the work shifts from a tuning exercise to an operational commitment. We design and operate sovereign AI systems inside infrastructure our customers control, including air-gapped networks, and we apply this same benchmark-first discipline as a standing operational practice rather than a one-time project. For teams building their own governance framework around AI adoption, data privacy considerations for enterprise AI outline the kind of readiness questions worth asking before any tuning engagement begins.

Where LLM inference tuning is headed

Predictive power and latency models are the next real shift in this space. Instead of running a full benchmark sweep on every candidate GPU and model pairing, tools like WattGPU estimate mean power draw and inter-token latency for unseen GPU and model combinations using only public metadata, which narrows the candidate list before a single watt gets measured on real hardware.

The honest prediction is that energy-aware service-level agreements become normal, not optional, over the next product cycle. Teams currently writing SLAs purely in latency terms will start writing them in Joules per token or Joules per response, especially wherever power budgets or sustainability reporting already constrain data center operations. Hardware trends, from new memory formats to denser HBM tiers, will keep shifting where the bottleneck sits between compute and bandwidth, which means today’s parallelism rule of thumb won’t hold forever.

The sequencing advice doesn’t change even as the tools get smarter: measurement first, automation second. A predictive model narrows your search space, it doesn’t replace the benchmark run that confirms the winning configuration actually meets your SLO on your hardware.

— John Ezzell, Founder

Forge AI Deployment services for secure, high-performance LLM inference

We built our engagements around the same discipline laid out across this guide: benchmark first, tune against a measured baseline, and validate every change before it reaches production traffic. The difference for enterprise teams is that we run this work inside infrastructure you control, including fully air-gapped networks, so performance tuning never requires sending model weights, prompts, or output data outside your perimeter.

Forge AI Deployment

As an official webAI systems integrator, we bring webAI’s local-first platform into production environments, covering architecture, secure deployment, and ongoing operation. Our work centers on:

Private and air-gapped deployment of the full inference stack, including the performance tuning practices covered in this guide.
webAI Personas, domain-specific AI configurations tuned for your operational context.
The Intelligence Delivery Network for distributing models and updates across sovereign environments without external dependencies.
webAI Frontline, which gives frontline teams fast, source-backed answers from large technical document libraries, fully offline on a single device.

If your organization needs reproducible benchmark evidence for an audit, strict data-residency compliance, or an operations team that owns tuning and performance as an ongoing SLA rather than a one-time project, visit our solutions page to see the full engagement scope, or learn more about the webAI partnership behind our deployments.

FAQ

Can an LLM be fine-tuned?

Yes, open-weight and many commercial LLMs support fine-tuning through techniques like LoRA, which adapts a model to specific tasks or domains without retraining all its parameters. Fine-tuning is a separate process from inference performance tuning covered in this guide, though both affect the final latency and quality of a deployed model.

How do I improve LLM model performance?

Start by benchmarking baseline metrics (TTFT, ITL/TPOT, throughput, and Joules per token), then tune in order: scheduler and batching strategy, parallelism configuration, and quantization or KV cache format. Each change should be validated against the same benchmark suite before moving to the next, since gains from one layer can mask or offset losses in another.

Is LLM fine-tuning dead?

No, fine-tuning remains a standard technique for adapting models to specialized domains or proprietary data, particularly in on-premise deployments where data cannot leave a controlled environment. It addresses model quality and task fit, not inference speed, so it works alongside performance tuning rather than replacing it.

How much does LLM fine-tuning cost?

Cost depends heavily on model size, dataset size, infrastructure, and whether the work runs on-premise or in the cloud, so there is no single published figure that applies across deployments. For on-premise fine-tuning specifically, our breakdown of on-prem LoRA economics walks through the cost factors enterprise teams typically weigh.