Agents in the cloud, inference on your floor. The platform decides which model and which endpoint — per use case.

If your agents are built in the cloud and some of your inference has to stay on your own floor, the hard part is not the hardware. It is the decision that gets taken on every single request: which model answers it, and which endpoint serves it. We join ATAILA GPU nodes to your EKS cluster as Amazon EKS Hybrid Nodes — your hardware, an AWS control plane — and put ATAILA's AI Gateway in front of both sides, so each use case is routed on sensitivity, cost, latency and data residency. One governance and observability plane over the whole thing. We are building it now, on a real customer case, as a fixed-price six-week pilot.

Two places, one request

A European customer came to us with a shape we now see every month. The agents are built in the cloud, because that is where the developers, the frameworks and the managed runtimes already are. Parts of the inference cannot go there: a contract, a patient record, a production log, a drawing that never leaves the building. They want an immutable repository with tests run on on-premises hardware, a gateway that routes inference, and one observability and governance plane across both sides. We said yes the same morning.

That is a hybrid, and most hybrids stop at the wiring: a VPN, a cluster, two endpoints, and a developer left to hardcode which one a given call goes to. Hardcoding it is what fails. The classification changes, the model list changes, a cheaper model becomes good enough for summarisation, a regulator tightens one data category — and every change means a code change in a dozen agents.

So the routing is the product. ATAILA''s AI Gateway sits in front of both sides and takes the decision per use case, from policy rather than from source code:

Sensitivity. A use case is classified once. Confidential categories are pinned to the on-premises endpoint, with a network block behind the policy, not a promise in a README.
Cost. Public, non-sensitive, high-volume work goes to whichever model is cheapest for that task — often a small one, often in the cloud.
Latency. Interactive paths go to the endpoint that answers fastest from where the user is; batch work goes where there is capacity.
Data residency. Where a jurisdiction is part of the requirement, the endpoint is part of the answer, and the evidence is per-tenant and exportable.

The agent asks one OpenAI-compatible endpoint. The platform decides what actually served it, and writes down why.

Per use case

How the model and the endpoint are chosen — sensitivity, cost, latency and data residency, from policy rather than from source code

Dec 2024

Amazon EKS Hybrid Nodes generally available — on-premises GPU nodes under an EKS control plane

Oct 2025

Amazon Bedrock AgentCore generally available — a gateway with guardrails and policy that routes inference to OpenAI-compatible endpoints

6 weeks

The pilot — design, hybrid substrate, agent baseline, decision — fixed price, credited against a Build

What we build, block by block

Agents in the cloud. Built with Strands Agents (Apache-2.0) or the framework the customer already uses; running on Bedrock AgentCore Runtime or on EKS. Nothing about the agent changes to make this work — it keeps calling an OpenAI-compatible endpoint.
Inference on ATAILA''s GPU nodes. vLLM — or NVIDIA NIM where a model licence asks for it — serving OpenAI-compatible endpoints from our bare-metal GPU nodes, joined to the customer''s EKS cluster as Amazon EKS Hybrid Nodes: the customer''s hardware under an AWS control plane, which is AWS''s own published mechanism for exactly this.
The routing layer, which is the point. ATAILA''s AI Gateway holds the model catalogue, the per-use-case policy and the per-tenant metering, and picks the model and the endpoint per request. Amazon Bedrock AgentCore Gateway can sit at the perimeter with Bedrock Guardrails and AgentCore Policy where the customer wants it. Plan B needs nothing from AWS: the ATAILA gateway alone, which is what our platform runs today.
One image, two registries. Built once, pushed to Amazon ECR and mirrored to Harbor on-premises with immutable tags — tested in the cloud for the baseline, tested on the on-premises nodes for the comparison.
One observability plane. AWS Distro for OpenTelemetry on every node, cloud and on-premises, into Amazon Managed Grafana and into ATAILA''s Grafana; AgentCore observability for the agent traces; ATAILA''s per-tenant usage evidence for the governance report. One dashboard answers "which model served this, where, at what cost" for both sides.
Hybrid network. An AWS Site-to-Site VPN to the customer''s VPC, or WireGuard — a project zone per tenant on our side, exactly as the platform isolates everything else.

The deliverable of the pilot is a comparison, not a slide: the same agent, the same test suite, run against the cloud model and against the on-premises GPUs, with latency, cost, quality and data residency on one dashboard. That is what a board decides on — and it is also what calibrates the routing policy, because the thresholds you route on should come from your own measurements rather than from a vendor''s.

Amazon Web Services (AWS) — Amazon Web Services, AWS and the AWS logo are trademarks of Amazon.com, Inc. or its affiliates. ATAILA is not affiliated with, sponsored by or endorsed by AWS.

Week 1

Design workshop.

Target architecture agreed, the model list, the use-case classification and the routing policy that follows from it, the test suite and its acceptance criteria, the network design, the success criteria. Remote, or in our Executive Briefing Center.

Weeks 2–3

The hybrid substrate.

VPN up; an ATAILA GPU node joined to the customer's EKS cluster as a hybrid node; vLLM serving the first model; the gateway routing by policy end to end; OpenTelemetry into both Grafanas.

Weeks 4–5

The agent baseline.

The customer's first agent containerised, in ECR and Harbor, tested in the cloud for the baseline and on-premises for the comparison; guardrails and policy on the gateway; the routing rules tuned against the measurements rather than against assumptions.

Week 6

The decision.

A report the board can read: quality, latency, cost and data residency per model and per location, the routing policy that follows from it, and the go/no-go for production on the customer's own hardware or on ATAILA Cloud. Fixed price for the six weeks, credited against a Build if you continue.

Where this sits in the mission

The Service Provider mission promised connectors to the hyperscalers as an option, never a dependency. AWS is the first, and it is a hybrid rather than a connector because the customer''s need is a hybrid: the agents live where the developers are, the inference lives where the data is, and something has to decide which is which. Azure, Oracle Cloud and Huawei Cloud follow on the roadmap — each when a customer asks or a vendor commits. Every ATAILA edition runs without any of them, and the gateway routes the same way whether the other side is a hyperscaler or a second room of your own.

The pilot''s comparison is handed to the customer whatever it shows — including the case where the honest answer is that the cloud model wins for that use case. A routing platform that can only ever route one way is not a routing platform; it is a sales argument with a config file.

Sources

AWS: Amazon EKS Hybrid Nodes (December 2024); Run GenAI inference across environments with Amazon EKS Hybrid Nodes.
AWS: Amazon Bedrock AgentCore generally available (October 2025); AgentCore Gateway inference targets; Strands Agents SDK.

Talk to us

If your agents run in the cloud and some of your inference has to stay on your premises — or you are the service provider whose customers are in that position — write two sentences on the form. It reaches the founder directly.

Talk about a hybrid AI pilot The Service Provider mission

← Back to the Newsroom Press enquiries: contact us