What is Learning-to-Defer?
Learning-to-defer is a machine learning setting in which a model can either make a prediction itself or hand the input to a human expert, a stronger model, or a tool, and learns which of the two to do on each input. It generalises learning with abstention and selective prediction, where a model may decline to answer but has nobody to pass the case to, and it is the learning problem underneath what is often called model routing.
The motivation is that most models are built to answer every question they are asked, including the ones they have no business answering. Deferral gives the model a way out. The interesting part is that using it well is itself something that has to be learned: defer too rarely and you keep the bad predictions, defer too often and you have built an expensive way of forwarding email.
That turns prediction into a routing problem. For every query you are choosing who should handle it, and the cheap option is not always the right one. Accuracy alone stops being a useful target here, because deferring everything to a human would score perfectly and be completely useless. What you actually want to control is the trade-off between how often you are wrong and how much you spend asking for help.
Try it yourself
The plot below has a fixed linear decision boundary and a messy pocket of mixed points near the top, where the boundary does a poor job. Draw a region over the part you would rather not classify at all.
Draw with your mouse, finger, or pen. Anything inside your region gets deferred and drops out of the accuracy count. Covering the whole plot gets you 100% accuracy on nothing at all, so the real game is to give up as little coverage as possible for the accuracy you gain.
Deferred 0 of 0 points.
What I work on
I work on both the guarantees and the systems that use them. The objective you actually care about here counts discrete mistakes and discrete costs, which makes it impossible to optimise directly, so you train on a surrogate instead. A good part of my work is about which surrogates you are allowed to use: whether minimising one really does drive you toward the best possible deferral rule, and how much you lose if it does not. The rest is building the thing and running it — PyTorch pipelines, distributed experiments, and LLM routing benchmarks, because a bound that no one can implement is not much of a result.
The other half is what happens once the clean assumptions break. Experts disagree with each other. They charge different amounts, and the cheap one is sometimes the good one. They come and go between rounds. Someone may be actively trying to make your router send queries to the wrong place. You rarely get to see what the expert you did not ask would have said. Each of these breaks a different piece of the standard analysis, which is why they end up as separate papers.
Routing, deferral, and abstention
Model routing, query routing, and learning-to-defer are the same problem under different names. Routing is what it is called on the systems side, usually in LLM serving, where a router decides which model handles each query; learning-to-defer is the same decision written as a learning problem, where the thing you are fitting is a policy over who acts and the cost of each choice sits inside the objective. The vocabulary differs because the communities differ, not because the problems do. A model cascade is the special case where candidates are tried in a fixed order until one is confident enough.
The one that genuinely is a different problem is selective prediction, or learning with abstention. There the model may decline to answer, but there is nobody to pass the case to, so the only choice is answer or stay silent. Adding a second party is what turns abstention into routing, and it is what makes cost part of the objective rather than an afterthought.
Most deployed routers today are a confidence threshold picked by hand. That works until it does not, and there is usually no way to say in advance which regime you are in. Treating routing as a learning problem with a well-founded surrogate loss is what makes the question answerable: does minimising your training objective actually produce the best achievable router, and if not, how far off is it?
Research directions
- Theory: consistency and excess-risk bounds for surrogate losses over defer-or-act decisions.
- Robustness: how deferral breaks under adversarial perturbation, and what it takes to keep the guarantees.
- Multi-expert routing: choosing among many experts, including how many to ask, not just whether to ask.
- Online and constrained settings: bandit feedback, non-stationary data, and hard budget or latency limits.
Where this shows up
The setting is older than the name. Radiology already works this way: routine scans get read quickly, ambiguous ones get escalated, and the cost of escalating wrongly is measured in someone's time. Fraud and claims systems do the same with a review queue. What is new is that the second reader is now often another model, which is where LLM cascades and tool-routing come in: a small model answers, or retrieves, or hands off to something bigger and slower. The economics only work if the handoff decision is good, and that decision is usually the least principled part of the pipeline.
That is the gap I care about. These systems get deployed with a confidence threshold picked by hand, and nobody can say what it costs when it is wrong. Getting a guarantee on the routing rule tells you when you are allowed to trust the thing.
Papers and results are on the learning-to-defer publications page.