New board topics in Databricks Community
Databricks Community
Databricks Release Notes
Databricks Release Notes updates
All Technical Blog posts
Technical Blog articles
article Databricks AI Model Serving in production: scaling, cost, and latency lessons in Technical Blog
<p><span>Getting a model behind a REST endpoint is the easy part; however keeping it fast, affordable, and stable once production traffic shows up is where teams spend their real time.</span></p> <p><span>Databricks </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/" rel="noopener" target="_blank"><span>Model Serving</span></a><span> (formerly called Mosaic AI Model Serving) is a serverless service that turns any model into an autoscaling REST endpoint and manages the infrastructure for you, but “managed” does not mean automatically optimize for your specific workload, rather it will manage your infrastructure setup, maintenance etc. while you still can make the choices during setup time and decide whether you end up paying for idle GPUs, watching p99 latency spike during a traffic burst, or quietly doing both. With </span><a href="https://www.databricks.com/product/artificial-intelligence/unity-ai-gateway" rel="noopener" target="_blank"><span>Unity AI Gateway</span></a><span>, you govern models from any source, Databricks-hosted and third-party or external, behind one layer for guardrails, rate limiting, cost tracking, and observability, without locking into a single provider, Model Serving is the serving layer in that bigger picture, a single platform that runs classic ML, deep learning, and today's GenAI and LLMs, including the models behind your agents and MCP tools, all under one governance layer with Unity Catalog and Unity AI Gateway.</span></p> <p><span>This post walks through how to set Model Serving up for production around three levers: </span><strong>how you scale, what you pay, and how fast you respond</strong><span>. When you reduce the latency, the cost tends to climb and, when you reduce cost, the tail latency tends to suffer. None of the three is free but the goal is to know which one matters most for the workload in front of you, and tune deliberately.</span></p> <h2><strong>01 </strong><strong>Start with the right endpoint type</strong></h2> <p><span>Your initial choice of endpoint dictates all downstream cost and latency outcomes. Model Serving provides four distinct endpoint types, each with unique performance characteristics and cost profiles.</span></p> <ul> <li style="font-weight: 400;"><strong>Custom model endpoint</strong><span> for your own MLflow models promoted from Unity Catalog, on CPU or GPU. These are your classifiers, recommenders, embeddings, and anything you trained or fine-tuned yourself.</span></li> <li style="font-weight: 400;"><strong>Foundation Model API, provisioned throughput</strong><span> for open or fine-tuned LLMs that need dedicated, guaranteed capacity. Databricks recommends this mode for production GenAI applications/workloads. This includes support for Partner models (OpenAI, Anthropic, Gemini), Open models (Llama, GPT OSS, Kimi, Qwen), and External models (third party models hosted outside of Databricks).</span></li> <li style="font-weight: 400;"><strong>Foundation Model API, pay-per-token</strong><span> for the same models on a shared, multi-tenant endpoint. Cheap to start, but latency varies under load, so it suits experimentation and low volume.</span></li> <li style="font-weight: 400;"><strong>External model endpoint</strong><span> for third-party providers such as OpenAI or Anthropic, fronted by Databricks so you keep one governance layer across everything.</span></li> </ul> <p><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 999px;"><img alt="figure1-endpoint-decision.png" src="https://community.databricks.com/t5/image/serverpage/image-id/28512i25DD568104D12878/image-size/large?v=v2&px=999" title="figure1-endpoint-decision.png" /></span></p> <p><strong>Figure 1. </strong><span>Pick the endpoint shape from what you are serving and where it runs. Everything downstream → autoscaling, billing, and latency tuning → follows from this branch.</span></p> <p><i><span>Two scaling terms appear throughout this section. </span></i><strong><i>Scale to zero</i></strong><i><span> means an endpoint spins its compute all the way down to zero when idle, so you pay nothing while it is idle, at the cost of a cold-start wake-up on the next request. </span></i><strong><i>Provisioned concurrency</i></strong><i><span> keeps a warm floor of replicas always running, so there is no cold start.</span></i></p> <table> <thead> <tr> <th> <p><strong>Endpoint type</strong></p> </th> <th> <p><strong>Best for</strong></p> </th> <th> <p><strong>How it scales</strong></p> </th> <th> <p><strong>How you pay</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p><strong>Custom · CPU</strong></p> </td> <td> <p><span>Classic ML, embeddings, high-QPS real-time</span></p> </td> <td> <p><span>Provisioned concurrency; scale-to-zero in dev</span></p> </td> <td> <p><span>DBU per second of active compute</span></p> </td> </tr> <tr> <td> <p><strong>Custom · GPU</strong></p> </td> <td> <p><span>Deep learning, vision, self-hosted models, fine tuned OSS models</span></p> </td> <td> <p><span>Provisioned concurrency; GPU autoscaling is slower</span></p> </td> <td> <p><span>DBU per second, by GPU tier</span></p> </td> </tr> <tr> <td> <p><strong>Provisioned throughput</strong></p> </td> <td> <p><span>Production LLM and GenAI apps</span></p> </td> <td> <p><span>Autoscales within a tokens-per-second range</span></p> </td> <td> <p><span>Per hour, by the throughput range</span></p> </td> </tr> <tr> <td> <p><strong>Pay-per-token</strong></p> </td> <td> <p><span>Experiments, low or bursty volume</span></p> </td> <td> <p><span>Shared multi-tenant; latency varies under load</span></p> </td> <td> <p><span>Per million input and output tokens</span></p> </td> </tr> <tr> <td> <p><strong>External model</strong></p> </td> <td> <p><span>Third-party APIs under your governance</span></p> </td> <td> <p><span>Delegated to the provider</span></p> </td> <td> <p><span>Provider token fees, billed directly</span></p> </td> </tr> </tbody> </table> <p> </p> <h2><strong>02 </strong><strong>The production reference architecture</strong></h2> <p><span>A production endpoint is rarely just an endpoint. In practice it is wrapped as a governance layer on the request path into the model, and an observability layer on the telemetry path coming out of it, all sitting on Unity Catalog. The shape below is what we recommend teams converge on.</span></p> <p> </p> <p><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 999px;"><img alt="figure2-reference-architecture.png" src="https://community.databricks.com/t5/image/serverpage/image-id/28513iB6E1CAAD532C37C3/image-size/large?v=v2&px=999" title="figure2-reference-architecture.png" /></span></p> <p><strong>Figure 2. </strong><span>Requests flow left to right through the Gateway into the endpoint; telemetry flows down into Delta-backed inference tables and monitoring. Unity Catalog is the substrate for the model registry, permissions, and usage system tables.</span></p> <p> </p> <h2><strong>03 </strong><strong>Scaling: the autoscaling decisions that matter</strong></h2> <h3><strong>Scale to zero is suitable for Development environments</strong></h3> <p><span>Scale to zero spins compute down to nothing when an endpoint is idle, which is excellent for keeping dev and test inexpensive. </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><span>It is not a production setting</span></a><span>: a scaled-to-zero endpoint has no guaranteed capacity, and the first request after an idle period is slow because it triggers a cold start; the wake-up can add seconds while compute spins back up. On GPUs, this is worse as GPU autoscaling takes even longer. For anything user-facing or latency-sensitive, turn off scale to zero and keep a warm floor of capacity.</span></p> <h3><strong>Provisioned concurrency: size it from data</strong></h3> <p><span>Provisioned concurrency is the maximum number of parallel requests an endpoint will handle, and it is the single most useful knob for real-time models. There is a clean </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><span>formula</span></a><span> to start from:</span></p> <p><strong>SIZING RULE </strong><span>provisioned concurrency = peak QPS (Query Per Second) × model execution time (seconds)</span></p> <p><span>Set the </span><strong>minimum</strong><span> high enough to absorb baseline traffic without queuing, and the </span><strong>maximum</strong><span> high enough to ride out spikes while still capping spend. The honest way to find both numbers is a </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/configure-load-test" rel="noopener" target="_blank"><span>load test</span></a><span> with a representative payload, because execution time is always hard to guess. If you start seeing </span><span>HTTP 429</span><span> responses under load, that is your signal that requests are queuing and concurrency is too low. A standard endpoint has a limit of </span><a href="https://docs.databricks.com/gcp/en/machine-learning/model-serving/model-serving-limits" rel="noopener" target="_blank"><strong>200 requests per second</strong></a><span>, and route optimization takes that up to 300,000. Requests at once. Separately, provisioned concurrency (how many requests run at once) is 4,096 per workspace by default, and a single model can reach 1,024 with a custom endpoint and route optimization. All of these can be raised on request.</span></p> <h3><strong>Provisioned throughput for LLMs</strong></h3> <p><span>For foundation models in production, </span><a href="https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/deploy-prov-throughput-foundation-model-apis" rel="noopener" target="_blank"><span>provisioned throughput</span></a><span> reserves dedicated capacity and is configured as a range of tokens per second (newer model families use model units instead). The endpoint autoscales within that range, and you are billed per hour for the range it is scaled to.Therefore, you pay for extra capacity only while you are using it. Because output tokens are more expensive to generate than input tokens, the right range depends on your real prompt and response shapes. Benchmark with a representative dataset rather than reading throughput off a spec sheet.</span></p> <h3><strong>Route optimization for high throughput</strong></h3> <p><span>A standard endpoint tops out at </span><a href="https://docs.databricks.com/gcp/machine-learning/model-serving/model-serving-limits" rel="noopener" target="_blank"><span>200 QPS per workspace</span></a><span>. Route optimization improves the network path between client and model and lifts that ceiling: published limits move to </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>50,000+ QPS</span></a><span>, and Databricks reports real-time workloads scaling to </span><a href="https://www.databricks.com/blog/best-practices-high-qps-model-serving-databricks" rel="noopener" target="_blank"><span>300K+ QPS on CPU</span></a><span>. Opt for it when you need more than 200 QPS or want to push the per-request overhead toward </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>sub-50ms</span></a><span>. Two caveats: route optimization is available for </span><strong>custom model endpoints only</strong><span> (not Foundation Model APIs or external models), and it requires OAuth tokens rather than personal access tokens.To enable it, select Enable route optimization when creating the endpoint (it can't be added later) and query the route-optimized URL with an OAuth token, see Route optimization on serving endpoints.</span></p> <h2><strong>04 </strong><strong>Cost</strong></h2> <p><span>Model Serving is serverless and billed in DBUs per second of active compute, and because it is serverless the DBU rate already includes the underlying cloud instance, so you see a single charge rather than a separate VM bill. What you pay comes down to how many DBUs each instance uses per hour, as published on the </span><a href="https://www.databricks.com/product/pricing/serverless-realtime-inference" rel="noopener" target="_blank"><span>Databricks pricing page</span></a><span>:</span></p> <table> <thead> <tr> <th> <p><strong>Instance size</strong></p> </th> <th> <p><strong>GPU configuration</strong></p> </th> <th> <p><strong>DBUs / hour</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p><strong>Small</strong></p> </td> <td> <p><span>T4 or equivalent</span></p> </td> <td> <p><span>10.48</span></p> </td> </tr> <tr> <td> <p><strong>Medium</strong></p> </td> <td> <p><span>A10G × 1</span></p> </td> <td> <p><span>20.00</span></p> </td> </tr> <tr> <td> <p><strong>Medium 4X</strong></p> </td> <td> <p><span>A10G × 4</span></p> </td> <td> <p><span>112.00</span></p> </td> </tr> <tr> <td> <p><strong>Medium 8X</strong></p> </td> <td> <p><span>A10G × 8</span></p> </td> <td> <p><span>290.80</span></p> </td> </tr> <tr> <td> <p><strong>Large 8X 40GB</strong></p> </td> <td> <p><span>A100 40GB × 8</span></p> </td> <td> <p><span>538.40</span></p> </td> </tr> <tr> <td> <p><strong>Large 8X 80GB</strong></p> </td> <td> <p><span>A100 80GB × 8</span></p> </td> <td> <p><span>628.00</span></p> </td> </tr> </tbody> </table> <p><span>DBU rates from the </span><a href="https://www.databricks.com/product/pricing/serverless-realtime-inference" rel="noopener" target="_blank"><span>Model Serving pricing page</span></a><i><span>. Rates vary by cloud, region, and tier and change over time, so confirm current figures there before building a cost model.</span></i></p> <p><span>The takeaway is simple: a Small T4 endpoint costs little to keep warm, but a large multi-GPU endpoint left running around the clock can use hundreds of DBUs an hour (up to 628 on a Large 8X 80GB, per the table above). For LLMs the pricing model differs again, pay-per-token is billed per million input and output tokens, while </span><a href="https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/deploy-prov-throughput-foundation-model-apis" rel="noopener" target="_blank"><span>provisioned throughput</span></a><span> is billed per hour by its configured range. Practical levers for keeping the bill honest:</span></p> <ul> <li style="font-weight: 400;"><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><strong>Start on CPU</strong></a><span>. Move a custom model to GPU only when you measure high inference latency or the model genuinely needs specialized compute. Do not default to GPU.</span></li> <li style="font-weight: 400;"><strong>Right-size the GPU tier </strong><span>after a load test. The jump from a single A10G (20 DBUs/hour) to a Large 8X 80GB (628) is more than a 30x difference in spend.</span></li> <li style="font-weight: 400;"><strong>Shrink the model. </strong><span>Quantization or pruning cuts both latency and the compute you pay for, and Databricks applies its own LLM serving optimizations on top.</span></li> <li style="font-weight: 400;"><strong>Use scale to zero where cold starts are acceptable </strong><span>(dev, internal tools, spiky low-volume endpoints) and keep production warm.</span></li> <li style="font-weight: 400;"><strong>Find the waste. </strong><span>Usage system tables log every request with requester and cost detail, so you can attribute spend by team and shut down idle endpoints.</span></li> </ul> <h2><strong>05 </strong><strong>Latency</strong></h2> <p><span>When a response feels slow, the time is hiding in one of a few places. Knowing which segment is costing you points straight at the lever that fixes it.</span></p> <p><span class="lia-inline-image-display-wrapper lia-image-align-inline" style="width: 999px;"><img alt="figure3-latency-budget.png" src="https://community.databricks.com/t5/image/serverpage/image-id/28514i9A22C4654A1B6B3B/image-size/large?v=v2&px=999" title="figure3-latency-budget.png" /></span></p> <p><strong>Figure 3. </strong><span>Segment widths are illustrative. Inference is the real work; queue and cold start are the avoidable tax. Each segment maps to a specific lever.</span></p> <h3><strong>The cost versus latency tradeoff</strong></h3> <p><span>Most latency levers cost more money, and most cost levers increase latency. </span></p> <table> <thead> <tr> <th> <p><strong>Setup</strong></p> </th> <th> <p><strong>Idle cost</strong></p> </th> <th> <p><strong>Cold start</strong></p> </th> <th> <p><strong>Tail latency</strong></p> </th> <th> <p><strong>Use when</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p><strong>Scale to zero</strong></p> </td> <td> <p><span>none</span></p> </td> <td> <p><span>yes</span></p> </td> <td> <p><span>spiky</span></p> </td> <td> <p><span>Dev, internal tools, low volume</span></p> </td> </tr> <tr> <td> <p><strong>Provisioned concurrency, min > 0</strong></p> </td> <td> <p><span>always on</span></p> </td> <td> <p><span>none</span></p> </td> <td> <p><span>low</span></p> </td> <td> <p><span>Production real-time models</span></p> </td> </tr> <tr> <td> <p><strong>Provisioned + route optimization</strong></p> </td> <td> <p><span>highest</span></p> </td> <td> <p><span>none</span></p> </td> <td> <p><span>lowest</span></p> </td> <td> <p><span>High QPS, </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>sub-50ms overhead</span></a></p> </td> </tr> <tr> <td> <p><strong>Provisioned throughput (LLM)</strong></p> </td> <td> <p><span>always on</span></p> </td> <td> <p><span>none</span></p> </td> <td> <p><span>consistent</span></p> </td> <td> <p><span>Customer-facing GenAI</span></p> </td> </tr> </tbody> </table> <p><span>Other levers are close to free latency wins: enable client-side </span><strong>batching</strong><span> when your app can group requests, reuse </span><strong>connection pooling</strong><span> (the Databricks SDK does this for you), and </span><strong>offload heavy pre- and post-processing</strong><span> off the serving path so the endpoint only does inference.</span></p> <h2><strong>06 </strong><strong>Govern and observe with Unity AI Gateway</strong></h2> <p><span>Unity AI Gateway (the governance layer that was Mosaic AI Gateway) is what keeps a production endpoint controllable. Configure it before launch, not after the first incident.</span></p> <ul> <li style="font-weight: 400;"><strong>Rate limiting</strong><span> in queries per minute and tokens per minute, set on the endpoint or per user, group, or service principal. This is your first line of defense against a runaway client.</span></li> <li style="font-weight: 400;"><strong>AI Guardrails</strong><span> to block or mask unsafe content and PII on the way in and out, set at endpoint or request level.</span></li> <li style="font-weight: 400;"><strong>Payload logging</strong><span> into </span><a href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/inference-tables" rel="noopener" target="_blank"><span>inference tables</span></a><span>, which continuously capture raw inputs, outputs, HTTP status codes, and latency as a Delta table in Unity Catalog. That table feeds Lakehouse Monitoring for quality and drift, and doubles as an evaluation or fine-tuning corpus.</span></li> <li style="font-weight: 400;"><strong>Usage tracking</strong><span> into system tables for cost attribution across every endpoint in the account.</span></li> <li style="font-weight: 400;"><strong>Fallbacks and traffic splitting</strong><span> for resilience and safe rollouts: fallbacks (on external endpoints) keep you serving during a provider outage, and traffic splitting lets you carry a new model version before shifting all traffic.</span></li> </ul> <p><strong>WATCH THIS LIMIT </strong><span>Turning on inference tables </span><a href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/inference-tables" rel="noopener" target="_blank"><span>caps total max concurrency on a single endpoint at 128</span></a><span>. If you are pushing very high concurrency and need full payload logging, plan for it or ask your account team to raise the limit.</span></p> <h2><strong>07 </strong><strong>The usual failure pattern, and the fix</strong></h2> <p><span>To make the levers concrete, here is the pattern we see most often when a custom model moves from notebook to production. It is an illustrative walkthrough rather than one customer’s story, so it carries no invented latency or traffic figures. Every limit and lever below is documented and linked.</span></p> <h3><strong>What goes wrong</strong></h3> <p><span>A team ships a custom CPU endpoint on the defaults: scale to zero left on from dev, modest concurrency, a standard rather than route-optimized endpoint, and clients on personal access tokens. A smooth, uniform load test looks fine, and then real traffic bites in three places at once. A standard endpoint is capped at </span><a href="https://docs.databricks.com/gcp/machine-learning/model-serving/model-serving-limits" rel="noopener" target="_blank"><span>200 QPS per workspace</span></a><span>, so a spike beyond it is throttled with </span><span>HTTP 429</span><span> errors. Once requests queue, tail latency climbs. And 429s under load are the documented sign that </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>concurrency is set too low</span></a><span>. And the first calls after an idle window pay a wake-up, because </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><span>scale to zero is not a production setting</span></a><span>.</span></p> <h3><strong>The fix is configuration, not hardware</strong></h3> <p><span>Throwing a GPU at a routing problem is the classic wrong turn. A single A10G lists at </span><a href="https://www.databricks.com/product/pricing/serverless-realtime-inference" rel="noopener" target="_blank"><span>20 DBUs/hour and an eight-GPU box at 290.8</span></a><span>, and neither touches a QPS ceiling. The fixes that do work are all configuration:</span></p> <ul> <li style="font-weight: 400;"><strong>Size concurrency from the </strong><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><strong>documented formula</strong></a><span> (provisioned concurrency = peak QPS × execution time), measured under load rather than guessed.</span></li> <li style="font-weight: 400;"><strong>Disable scale to zero and keep a warm minimum </strong><span>so no request pays the cold-start tax.</span></li> <li style="font-weight: 400;"><strong>Turn on route optimization </strong><span>to lift the ceiling </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>from 200 to 50,000+ QPS</span></a><span> (Databricks reports </span><a href="https://www.databricks.com/blog/best-practices-high-qps-model-serving-databricks" rel="noopener" target="_blank"><span>300K+ QPS on CPU</span></a><span>). It is custom-endpoint-only and needs OAuth, not PATs.</span></li> <li style="font-weight: 400;"><strong>Reuse </strong><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><strong>connection pooling and client-side batching</strong></a><span> to cut per-request overhead.</span></li> </ul> <p><strong>THE TRADEOFF TO KNOW UP FRONT </strong><span>Route optimization is not free of tradeoffs. On a custom endpoint, </span><a href="https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/overview-serving-endpoints" rel="noopener" target="_blank"><span>route-optimized endpoints do not currently support rate limiting or usage tracking</span></a><span>, and inference-table logging for them is still in preview. If you need both maximum throughput and full gateway governance, decide where each one lives before launch.</span></p> <table> <thead> <tr> <th> <p><strong>Symptom</strong></p> </th> <th> <p><strong>What it usually means</strong></p> </th> <th> <p><strong>Lever</strong></p> </th> </tr> </thead> <tbody> <tr> <td> <p><strong>HTTP 429 under load</strong></p> </td> <td> <p><span>Past the standard-endpoint QPS ceiling</span></p> </td> <td> <p><span>Route optimization, </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>200 → 50,000+ QPS</span></a></p> </td> </tr> <tr> <td> <p><strong>Tail latency climbs under load</strong></p> </td> <td> <p><span>Concurrency too low; requests queue</span></p> </td> <td> <p><span>Size from the </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><span>QPS × execution-time formula</span></a></p> </td> </tr> <tr> <td> <p><strong>Slow first request after idle</strong></p> </td> <td> <p><span>Scale-to-zero cold start</span></p> </td> <td> <p><span>Disable </span><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><span>scale to zero</span></a><span>, keep a warm minimum</span></p> </td> </tr> </tbody> </table> <p><span>The lesson is, measure execution time, size concurrency from it, keep a warm floor, and clear the routing ceiling. Also consider what route optimization costs you on the governance side.</span></p> <h2><strong>08 </strong><strong>Two more shapes, in brief</strong></h2> <p><span>Not every workload has a high-QPS requirement but there are also two more patterns we often reach for:</span></p> <p><strong>CUSTOMER-FACING RAG / LLM</strong></p> <p><strong>An assistant your users talk to</strong></p> <p><span>Quality, safety, and steady latency matter more than saving idle cost.</span></p> <ul> <li style="font-weight: 400;"><strong>Provisioned throughput, not pay-per-token. </strong><span>Reserve dedicated capacity sized to your measured peak, with the minimum above zero so it never cold-starts.</span></li> <li style="font-weight: 400;"><strong>Budget for guardrails. </strong><span>PII masking and toxicity filters run their own moderation pass and add latency. It is worth it, but put it in the latency budget rather than discovering it in production.</span></li> <li style="font-weight: 400;"><strong>Rate-limit per team in tokens-per-minute </strong><span>through Unity AI Gateway, so one noisy client cannot drain the shared budget.</span></li> <li style="font-weight: 400;"><strong>Fallbacks and traffic splitting </strong><span>for safe rollouts; mind the </span><a href="https://docs.databricks.com/gcp/machine-learning/model-serving/model-serving-limits" rel="noopener" target="_blank"><span>4 MB payload limit on foundation endpoints</span></a><span>.</span></li> </ul> <p><strong>EXPERIMENTATION / INTERNAL TOOLS</strong></p> <p><strong>Spiky, low-volume, cost-sensitive</strong></p> <p><span>Here, paying for idle capacity is the mistake, not the occasional cold start.</span></p> <ul> <li style="font-weight: 400;"><strong>Pay-per-token, or CPU with scale to zero on. </strong><span>Accept the first-request wake-up to keep idle spend at zero.</span></li> <li style="font-weight: 400;"><strong>Cap spends at the gateway </strong><span>with queries- and tokens-per-minute limits per user or group, before someone’s notebook loop runs up a bill.</span></li> <li style="font-weight: 400;"><strong>Promote on evidence. </strong><span>Use the usage system tables to see which experiments actually get traffic, and move only those to provisioned capacity.</span></li> </ul> <h2><strong>09 </strong><strong>The pre-production checklist</strong></h2> <ul> <li><strong>Load test with representative payloads. </strong><span>Derive concurrency and throughput from measured execution time, not estimates.</span></li> </ul> <ul> <li><strong>Disable scale to zero </strong><span>for anything user-facing, and keep a warm minimum.</span></li> </ul> <ul> <li><strong>Turn on route optimization </strong><span>for custom models </span><a href="https://docs.databricks.com/gcp/machine-learning/model-serving/model-serving-limits" rel="noopener" target="_blank"><span>above 200 QPS</span></a><span> or with low-overhead requirements, and move clients to OAuth.</span></li> </ul> <ul> <li><strong>Enable inference tables and usage tracking</strong><span>, build a dashboard, and alert on 429s, p95 and p99 latency, and error rate.</span></li> </ul> <ul> <li><strong>Set rate limits and guardrails </strong><span>with Unity AI Gateway before launch.</span></li> </ul> <ul> <li><strong>Plan fallbacks and a rollback path</strong><span>, and use traffic splitting for canary releases.</span></li> </ul> <ul> <li><strong>Right-size compute. </strong><span>Start on CPU, move to GPU only when the numbers justify it, and revisit the GPU tier after testing.</span></li> </ul> <p></p> <p> </p> <p><span>It is the difference between an endpoint that demos well and one that survives a Monday morning traffic spike at a price you can defend. Pick the endpoint type from your workload, keep production capacity warm, size scaling from real measurements, and let Unity Catalog and Unity AI Gateway give you the governance and visibility to keep tuning. Decide which lever your workload cares about most and then make the other two earn their cost.</span><strong> </strong></p> <h2><strong>Related articles</strong></h2> <p><span>Every figure in this post links to its source. The full list:</span></p> <ul> <li style="font-weight: 400;"><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/glossary" rel="noopener" target="_blank"><span>Mosaic AI Model Serving concepts: scale to zero, route optimization, the concurrency formula</span></a></li> <li style="font-weight: 400;"><a href="https://docs.databricks.com/gcp/machine-learning/model-serving/model-serving-limits" rel="noopener" target="_blank"><span>Model Serving limits and regions: QPS ceiling, payload sizes, concurrency</span></a></li> <li style="font-weight: 400;"><a href="https://docs.databricks.com/aws/en/machine-learning/model-serving/production-optimization" rel="noopener" target="_blank"><span>Optimize Model Serving endpoints for production: route optimization, OAuth, batching, pooling</span></a></li> <li style="font-weight: 400;"><a href="https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/deploy-prov-throughput-foundation-model-apis" rel="noopener" target="_blank"><span>Provisioned throughput Foundation Model APIs</span></a></li> <li style="font-weight: 400;"><a href="https://www.databricks.com/product/pricing/serverless-realtime-inference" rel="noopener" target="_blank"><span>Model Serving pricing: GPU DBU rates</span></a></li> <li style="font-weight: 400;"><a href="https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/overview-serving-endpoints" rel="noopener" target="_blank"><span>Unity AI Gateway for serving endpoints: feature support and route-optimized limits</span></a></li> <li style="font-weight: 400;"><a href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/inference-tables" rel="noopener" target="_blank"><span>Inference tables for monitoring: captured fields and the 128 concurrency cap</span></a></li> <li style="font-weight: 400;"><a href="https://docs.databricks.com/aws/en/ai-gateway" rel="noopener" target="_blank"><span>Unity AI Gateway overview</span></a></li> <li style="font-weight: 400;"><a href="https://www.databricks.com/blog/best-practices-high-qps-model-serving-databricks" rel="noopener" target="_blank"><span>Best Practices for High QPS Model Serving on Databricks: 300K+ QPS</span></a></li> <li style="font-weight: 400;"><a href="https://www.databricks.com/blog/whats-new-unity-ai-gateway-service-policies-guardrails-observability-and-cost-controls-ai" rel="noopener" target="_blank"><span>What’s new in Unity AI Gateway</span></a></li> </ul> <p> </p>
July 2026 release notes for new Databricks features and improvements.
