Kog looks to squeeze more AI inference performance out of existing GPUs

French startup Kog has a bet: standard data center GPUs still have plenty of untapped potential for AI inference. In May, the company made it to the front page of Hacker News with a technical preview showing extremely fast single-request decoding on AMD MI300X and NVIDIA H200 GPUs, the kind of hardware enterprises already own. While some were disappointed that the approach did not work on laptop-class GPUs, others saw a meaningful opportunity.

With inference speed and cost becoming a critical bottleneck, Kog’s pitch of unlocking new capabilities through software optimization attracted real interest. CEO Gaël Delalleau said the preview generated 200 tangible business leads. Based on early feedback, he expects software engineering to be the first major use case. Many Claude Code users know the frustration of waiting hours for results, and Anthropic charges a premium for its Claude Fast Mode. Kog is going after customers who cannot afford those delays, especially those relying on AI workflows for professional work. The startup also has design partners building products that generate games and apps from a prompt — faster inference would mean more revenue for them.

Kog acknowledges that the market is still immature. It has learned that prospective customers are not willing to fine-tune small models. That is why, since launching, the company has shifted its focus to accelerating larger models. Its public demo delivered an impressive 3,000 tokens per second per request, but using a purpose-built small model of about 2 billion parameters, now open-sourced as Laneformer 2B. The overarching goal is a 30x speedup for large language model inference.

Delalleau insists GPUs have a bright future. He argues that newer GPUs have growing memory bandwidth that software can unlock. Kog is not alone in this line of thinking — French company ZML has released hardware-agnostic software that bypasses NVIDIA’s CUDA to enable fast inference across different chips. But Delalleau compares Kog to Stanford’s Hazy Research lab, with an even deeper focus on GPU acceleration.

Delalleau’s background is unusual: he studied solid-state physics at École Polytechnique and later worked in offensive cybersecurity, including white hat hacking. He says this taught him to reverse-engineer systems down to assembly language and binary code, and to apply that mindset to GPU engineering. The downside is that it is highly manual and time-consuming — the team of 11 spends weeks or months on each new GPU. That limits how many chips Kog can support for now; eventually it hopes to use agent-based pipelines to widen coverage.

As Europe pushes to build its own AI capabilities, Kog has tailwinds from sovereignty-related support: Scaleway, Bpifrance and French Tech 2030 are backing it. Still, the immediate task is to prove the approach on large language models. Delalleau expects to implement its first major model at 10x speed in September, which would enable customer traction and a Series A round.