Making language models think faster.

Traditional language model inference computes probability scores for every token in the vocabulary—typically 50,000 to 100,000 possibilities—at every generation step. For constrained decoding, where only a subset of tokens are valid at any position, this creates massive computational overhead. When generating JSON, maybe 20 tokens are valid after an opening brace. When following a grammar, the valid set shrinks even further. Yet standard inference engines score them all, then mask out the invalid ones. It's like calculating the price of every item in a grocery store when you only need the cost of items in your cart. The waste compounds: constrained generation often requires longer sequences, more complex reasoning, and multiple attempts to satisfy all constraints.

LLM-QP flips this model by computing scores only for tokens that could possibly be valid. Instead of scoring 50,000 tokens and masking 49,980 of them, it identifies the 20 valid candidates and scores only those. This requires solving two hard problems: efficiently determining which tokens are valid at each step, and maintaining numerical precision when working with sparse score distributions. The system uses contextual bandits to learn which execution strategy works best for different types of constraints. Sometimes it's faster to compute a full forward pass with masking. Sometimes sparse scoring wins. The system learns to route between approaches based on the specific constraint pattern and model architecture.

The most interesting part isn't the sparse scoring—it's how LLM-QP integrates constraint checking directly into the model's computation graph through MLIR and StableHLO. Instead of treating constraints as a post-processing step, they become part of the compilation process. The constraint logic gets fused with the model's forward pass, eliminating redundant memory transfers and enabling optimizations that wouldn't be possible with separate constraint checking. This isn't just faster inference; it's a different computational model where constraints shape the computation itself rather than filtering its outputs.

Faster constrained decoding doesn't just save compute—it enables applications that weren't feasible before. Multi-step reasoning becomes practical when each step doesn't waste cycles on impossible paths. Interactive applications can maintain complex state constraints in real-time. Research teams can explore more sophisticated constraint patterns without waiting hours for results. The efficiency gains compound when you consider that most practical AI applications need some form of constrained generation: structured data extraction, code generation, formal reasoning, API interactions. LLM-QP suggests that the future of language model inference isn't just bigger models, but smarter computation that respects the actual structure of the problems we're trying to solve.