The AI router that sends easy prompts to cheap models, and hard ones to your best.
Change one line of code. Halfy reads every request, picks the cheapest model that can answer it well, and shows you what you saved.
~50% cheaper on 10,000 held-out prompts · beats RouteLLM's router · see your savings free
Animation: Halfy, a round half-pink half-white mascot, catches incoming requests and throws easy ones into the cheap model's bucket and hard ones into the strong model's bucket.
Works in front of the providers you already use
Halfly speaks the OpenAI API, so your SDK, streaming and tool calls keep working. It routes to OpenAI, Anthropic Claude, Azure, Groq, OpenRouter, Ollama, or anything OpenAI-compatible.
Free to try: no card, no sign-up
Pick your cheap and strong models
Proxy on :8000 · dashboard on /dashboard
1from openai import OpenAI2 3client = OpenAI(base_url="http://localhost:8000/v1")4 5reply = client.chat.completions.create(6 model="halfly-auto",7 messages=[{"role": "user", "content": question}],8)Three habits, one goal: never pay top price for an answer a cheaper model could give just as well.
Step 1
A small model we trained predicts whether a cheap model will answer well. Easy questions go cheap. Hard ones go to your best model.
Step 2
Asked the exact same thing before? Halfy answers from memory, instantly, at no cost. Each customer's cache is private to them.
Step 3
Halfly keeps the sentences your question needs and drops the filler. Code, numbers, links and the question itself are never touched.
Safety first. If a model is down or busy, Halfy tries the other one. If anything about the router fails, requests go to your strong model, so quality never drops silently.
No thresholds to tune. Say how good answers must be, and Halfly works out the routing. Its router's scores are honest, so “95% sure” really means about 95%.
Target 95%: Halfly delivered 96.1% and cut cost by 58%.
You ask for
95%
of answers
You get
96.1%
+1.1 pts
You save
58%
off the bill
# halfly.yaml
router:
quality_target: 0.95How it works. For each prompt, the router estimates the chance a cheap answer is good enough, and those estimates are honest. Halfly picks the cut-off that meets your target, then keeps checking itself with audits.
5,000 held-out prompts, cheap model priced at 6% of the strong one, before any audits. Higher quality, smaller saving.
Holds on every kind of prompt
On code, maths, short and long prompts it stayed within about 1 point of the target. Lowest: 94.0% on 166 code prompts at a 95% target.
Gets bolder as it learns
It starts careful. Once audits confirm it on your traffic, it routes more boldly: after 1,000 audits, a 95% target saves 61.7% instead of 58% (median quality 95.5%).
Change it per request
One call that must be perfect? Send "halfly": {"quality": 0.99} with it.
Seven more new things, so saving money never means losing sleep.
Try it risk-free
Turn on observe mode. Every request still goes to your strong model, so nothing changes for your users. Halfly reports what routing would have saved, and at what checked quality. Like the numbers? Flip one setting.
mode: observeChecks its own work
For a random 2% of requests (at most 30 an hour), Halfly quietly asks the other model too. A judge model compares the two answers. Your dashboard shows the measured quality, with a confidence range. Audit costs are shown and taken off your savings. No prompt text is stored.
sample_rate: 0.02Never a surprise bill
Set a daily or monthly dollar cap for each API key. From 80% of the cap, responses carry a warning header. At the cap, you choose: keep serving with the cheap model, or refuse with a 429.
on_exceed: weakLearns your traffic
Retrain the router on your own checked requests. Halfly keeps the new router only if it beats the current one on held-out audits. If it doesn't, nothing changes.
$ halfly learnBonus for Claude
If your app sets no cache markers, Halfly marks the system prompt and earlier turns for Anthropic's prompt caching. Repeated input then costs about 1/10 of the normal price. It only counts in your savings when Halfly added it.
auto_cache: trueCost per feature
Tag each request with the part of your product that sent it. The dashboard shows what every feature costs and saves, and each one can have its own quality level: relaxed for small talk, strict for anything legal.
"halfly": {"tag": "support-chat"}Savings in your team chat
Every week Halfly posts what it saved you, and at what checked quality, to Slack or any chat that takes a webhook. If a budget gets close to its cap, you hear about it straight away.
HALFLY_WEBHOOK_URL=…Every result below comes from test data the models never saw while training. We share the raw metrics and the test scripts with any team evaluating Halfly.
Average quality gap recovered (APGR) across every budget, on 10,000 held-out prompts. Higher is better.
99%99%
of re-tests won vs RouteLLM
2,000 bootstrap resamples · lead +0.0015 to +0.0168 (95% CI)
19×19×
more honest scores
Calibration error 0.016 vs 0.303: “90% sure” means ~90%
8×8×
smaller model
33M vs 278M parameters
47ms47ms
to decide
Median, 2 CPU threads, no GPU
In plain words
With a cheap model at 6% of the strong model's price, the bill drops about 50% while 97.3% of prompts still get strong-model quality. Playing it safer: about 25% off at 99.3%.
Share of long prompts (≈2,900 tokens) that still contain the answer after cutting them to half their tokens. 200 fresh SQuAD v2 items.
Trade-off: about 1 second per 1,000 tokens on a CPU, so it is off by default and meant for long documents.
Wrong answers served on 54 hand-written trap pairs, like “sort ascending” vs “sort descending”.
Honest note: it is cautious and catches only about a quarter of true rewordings, so “same meaning” caching ships switched off. Exact-match caching is always on.
Router labels: a GPT-4 judge scored the cheap model's answer (Mixtral) against GPT-4's; a prompt counts as “fine on the cheap model” at 4/5 or above (86.5% of prompts). Data: RouteLLM gpt4_dataset (Apache-2.0), SQuAD v2 (CC BY-SA 4.0). Your own model pair will differ, so Halfly logs your real savings per request.
Install Halfly free and it watches your real traffic: your users see no change, and your dashboard counts what routing would save you, at what checked quality. Like the number? Switch it on for £5 a month.
£0forever
See what you would save, on your own traffic.
Switch the savings on
£5a month
Turn the savings you saw into real ones.
halfly report£10a month
Savings that keep growing as Halfly learns.
halfly learn tunes routing on your own checked trafficCancel any time. If you stop paying, your app keeps working: Halfly simply goes back to free observe mode.
Halfly runs on your servers, so your prompts never reach us. Prices in pounds, per month. Bigger team? Talk to us.
Example month using our Balanced benchmark (97.3% of prompts still at strong-model quality; cheap model at 6% of the strong one's price). AI bills are in dollars and our prices in pounds, so we convert at about $1.30 per £1. Your own number shows up in the free dashboard before you pay anything.
See your number freeStraight answers, no small print.
You decide. Set a quality target, for example 95% of answers as good as your strong model's, and Halfly only uses the cheap model while it can keep that promise. On our benchmark, asking for 95% delivered 96.1% at 58% lower cost. Halfly double-checks a sample of real answers so you can see the quality it actually delivers, and if anything goes wrong with the router, requests go to your strong model.
Yes, and it is free: no card, no sign-up. Without a paid plan Halfly runs in observe mode: every request still goes to your strong model, so your users see no change, and the dashboard shows what routing would have saved, at what checked quality. When you like the numbers, buy Starter (£5 a month), activate your key, and the savings switch on.
Yes, a little. Halfly re-asks the other model for about 2% of requests, at most 30 an hour, and a judge model scores the answers. That spend is shown on the dashboard and taken off the savings Halfly reports, so the savings you see are already net of it. Audits also pause when a budget cap is reached.
A small machine-learning model we trained on 104,000 real prompts, each labelled by whether a cheaper model answered it well. It reads a prompt in about 47 ms on a normal CPU and predicts the chance the cheap model is good enough. On 10,000 prompts it never saw, it beats RouteLLM's public router.
Anything that speaks the OpenAI Chat Completions API (OpenAI, Azure OpenAI, Groq, Together, OpenRouter, Ollama, vLLM) plus Anthropic Claude through its official SDK. You pick one cheap and one strong model and set their prices.
No. Halfly runs on your own servers, so prompts never come to us. By default it does not even write prompt text to its own log; it records only costs and routing decisions. Each API key gets its own cache, so customers never see each other's answers. A paid licence checks in with us about once a day, and it sends only the licence key: no prompts, no usage data.
RouteLLM is research code for routers; Halfly is a ready-to-run proxy with a router that scores higher on the same test, plus caching, prompt compression, fallback and a savings dashboard. OpenRouter is a hosted marketplace for many models; Halfly runs next to your app, on your own servers, in front of whichever providers you already use.
Free to install and try: observe mode, the savings dashboard and budget caps cost nothing. Starter is £5 a month and switches the savings on (routing with the quality dial, exact cache, fallback, dashboard and reports). Pro is £10 a month and adds self-learning, cost and quality per feature, a weekly savings report with budget alerts, and prompt compression. Cancel any time: if you stop paying, your app keeps working and Halfly goes back to free observe mode.
From held-out test sets the models never saw while training: the RouteLLM gpt4_dataset for routing and SQuAD v2 for compression. We share the raw metrics and the test scripts with any team evaluating Halfly. Your own savings depend on your models and traffic, which is why Halfly logs real costs per request and starts in observe mode.
See your savings free. Switch them on for £5 a month. Cancel any time.