Objective and research direction
Serving converts a trained model into a system that handles inference requests. For language models, deployment efficiency depends on the model, the hardware, and the way requests share compute and memory. The project will investigate these interactions with an emphasis on lower-cost inference.
The research question is how to improve useful serving throughput within a fixed resource budget while maintaining acceptable response times and output quality. The aim is to identify an optimization that remains effective as request concurrency and context length change.
Candidate optimization methods
The literature review examines three groups of methods. The aim is to identify an approach that can improve serving efficiency under the available hardware and workload constraints.
- Request scheduling and batching. Examine how requests enter and leave a running batch, and how scheduling affects queueing delay and accelerator utilization.
- Attention memory and caching. Study key-value cache management, reusable prompt prefixes, and the different resource demands of prompt processing and token generation.
- Model compression and execution. Evaluate whether reduced numerical precision or other model-level changes provide a useful performance gain after accounting for output quality and hardware support.
Alex Smola’s Efficiency in LLMs tutorial provides the initial technical reading framework. Google Cloud’s inference optimization overview distinguishes changes to serving infrastructure from changes to the model itself.
Proposed evaluation protocol
The proposed study starts with a reproducible serving baseline, then changes one selected mechanism at a time. Candidate workload variables include concurrent requests, prompt length, generated sequence length, and shared-prefix frequency. Model weights, hardware, software versions, and workload settings should remain fixed within each comparison.
| Measure | Evaluation purpose |
|---|---|
| Time to first token | Separate initial request responsiveness from the subsequent generation rate. |
| Inter-token and end-to-end latency | Report typical and tail response times as the offered workload increases. |
| Request and token throughput | Count completed work under explicitly stated latency constraints. |
| Peak memory and resource use | Measure the capacity needed to support the same workload. |
| Output quality and cost | Check quality against the baseline and report cost only with a stated hardware or pricing model. |
The final protocol should include repeated measurements, warm-up behavior, failed requests, and the conditions under which an optimization stops helping. These controls are intended to distinguish a reproducible improvement from a favorable isolated run.
Current status and next steps
The project is in progress. Current work centers on method selection and evaluation design. Next steps are to select the model and hardware configuration, establish a serving baseline, and define the comparison before running experiments. No experimental results are available yet.
The project is part of Columbia University’s High Performance Machine Learning course (COMS 6998) in Fall 2026. Implementation details, measurements, and conclusions will be added as the work develops.
Initial references
- Alex Smola — Efficiency in LLMs. Tutorial and slides on inference hardware, serving, and compression, with links to research papers.
- Alex Smola — Efficiency in LLMs lecture recordings. Overview and hardware sessions from the 2026 Machine Learning Summer School.
- Google Cloud — What is inference optimization?. Introduction to infrastructure-level and model-level optimization.
- Databricks — Model Serving. Production context for managed deployment, scaling, and governance; a product reference rather than experimental evidence.