How Together AI is rethinking the economics of running open models

We took a close look at Together AI, the AI Native Cloud for running open-source models, from its serverless inference API and GPU clusters to the customer stories and pricing pages behind the pitch. Inside: why a company that builds no frontier model of its own keeps winning production traffic, which buyers defect from closed APIs first, and the number most of the inference category still measures wrong.

Here is the awkward truth of AI products: the demo was cheap, production is not. Teams that built on closed frontier APIs discovered that usage growth and margin growth point in opposite directions. Voice agents need to answer in under a second or the caller hangs up. Coding assistants need tokens streaming faster than a developer types. Meanwhile someone in finance keeps asking why cost per turn slopes up while revenue per user slopes down. The Washington Post gave the feeling a name, "AI Independence," when it explained its move toward open models. The open models themselves became good enough. What never showed up was a way to run them at production scale without hiring a datacenter team. That gap is the entire business Together AI is in.

The wedge: own the kernels, not the models

Most inference providers rent GPUs and resell margin on top of open weights. Together AI starts lower, at the kernels. Co-founder and Chief Scientist Tri Dao is the FlashAttention researcher, and the company ships its own systems work into the platform: FlashAttention-3, the Together Kernel Collection, ThunderKittens, ATLAS speculative decoding. The homepage attributes some loud numbers to that research, namely 2x faster inference, 60% lower cost with workload-specific optimization, and 90% faster pre-training.

The hard part to copy is talent concentration. There are only so many people alive who can write an attention kernel that beats the hardware vendor's own library, and one of them co-founded the company. You cannot acquire that in a single hiring quarter.

Then the stack does the rest. Serverless inference at published per-million-token prices. Provisioned throughput with a 99% uptime SLA. Dedicated endpoints, containers, instant GPU clusters from B200s up to GB200 NVL72 nodes, plus fine-tuning, custom training, sandboxes, and evaluations. A team can prototype in a sandbox, shape a model, then move to dedicated capacity without rewriting its integration. The marketing leans into the comparison on purpose: benchmarks against GPT-5 mini and GPT-4 position open models as a drop-in swap, not a compromise.

The ICP: teams whose product dies above 500 milliseconds

The customer stories page reads like a list of companies where latency is the product. Decagon builds voice agents and its story cites 11x faster inference and a 6x cost reduction per turn versus gpt-5 mini, with time-to-first-token under 500ms. Voice forgives nothing. Cursor, the AI code editor, runs real-time, low-latency inference at scale on GB200 NVL72 clusters. Runware scales generative image and video APIs at 5-10x versus competitors. Latent Health runs clinical AI with two-second response times on Instant Clusters, and hesitation there is not a UX problem, it is a patient problem.

The second group is enterprise AI teams. Caiming Xiong, VP of Salesforce AI Research, put his name to "a 2x reduction in latency (time to first token)" and costs cut "by approximately a third." Notice who signs these deals: research VPs, CTOs, heads of ML and platform engineering. Not marketers. The common thread across both groups is that they already shipped, already carry real traffic, and hit a wall the closed APIs would not move. Experimenters stay wherever is convenient. Teams with a latency budget and a unit-economics problem are the ones who switch, and they arrive knowing their numbers.

What the category still gets wrong: it benchmarks the model, not the bill

The inference market behaves like a leaderboard. Vendors compete on benchmark points and sell quality per token, with cost relegated to a footnote. Read the Together case studies again and notice which numbers buyers volunteer: cost per turn, time to first token, throughput at a latency floor. Nobody cites benchmark scores. A model that scores two points higher but triples your cost per completed task loses the contract, and the category's default metric cannot even see that.

The second error is the framing. The industry narrates this as open models versus closed models, a war between model providers. The more useful read is that weights are commoditizing on a quarterly cadence, which means the systems layer now decides unit economics. Together sells performance per dollar per millisecond, and that advantage compounds across every model release for free. Last, the cheap-serverless pitch misleads buyers. At production scale the headline per-token price matters less than the shape of the cost: cached-input discounts, batch pricing, reserved throughput backed by a 99% SLA, dedicated capacity when a traffic spike costs you customers. Together's pricing page is organized around exactly those distinctions, which tells you what its buyers actually ask about.

If you run an AI product, the takeaway is not "switch to Together AI." It is to price your own workload honestly, end to end: latency floor, cache-hit ratio, batch share, cost per completed task. Model loyalty is depreciating fast. The winners of the next phase are the vendors making the underlying math better one kernel at a time, and the operators who benefit most are the ones who know their own numbers cold.