Add `cacheMessageOffset` field to `PromptCachingConfig`. When set,
shifts the Bedrock message cache point further back from the default
second-to-last position.
- `cacheMessageOffset: 0` (default): no change in behavior
- `cacheMessageOffset: N`: cache point moves N additional messages
toward the start, clamped to bounds
This gives callers control over which portion of the conversation is
covered by the cache prefix — useful for agentic workloads where
mid-conversation compaction may mutate recent messages.
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
With -race, this took 53s on my fast machine before. now it takes 3.
The problem was we parsed all CRDs each time, now we do it once (and
ruse more code)
Currently, mcpAuthentication is a backend level policy. This sort of
made sense at the time, as its only applicable for MCP, and "is MCP" is
a property of the backend. However, in practice this causes issues.
McpAuth is a glorified JWT validator with some additional special path
handling. One of the things you might want to use JWT/MCPAuth for is
apply other policies based on the claims. Authorization, rate limit,
transformations, etc
all based on the jwt claims are quite handy and common.
The alternative to this is to start copying every single policy to an
mcpXYZ policy which sucks.
----
This would be the first time the project faces a serious "deprecating an
API" so warrants some thinking. In this case I propose:
* The old backend policy is kept around in Kubernetes and marked as
deprecated. We can keep the field around for a very long time (forever?)
without too much burden on the project as its pretty much the same
translation regardless of where it is
* Standalone is not impacted; we already had users set it in the route
level and just translated it down to a backend policy.
* If a user sets both, we reject the backend one (TODO: this is not done
in this PR)
* If a user sets JWT and MCP auth, we prefer whichever is more precise.
(TODO: this is not done in this PR)
Fixes https://github.com/agentgateway/agentgateway/issues/870
This policy is equivalent to the HTTP authorization policy but runs at
the L4 level. This means you can use it for non-HTTP traffic, and you
can layer it on top of HTTP traffic to have L4+L7 rules. For example,
you might deny banned CIDR ranges at L4 and then have L7 rules on top of
that.
These are independant rule sets, so an allow in L4 does not allow at L7.
The policy runs _after_ TLS handshake and de-tunneling. This ensures the
properties of the handshakes can be used in policy decision making. In
theory, you may want to block users earlier to avoid the cost of a TLS
handshake, but IMO this is the wrong tradeoff here; if you are trying to
prevent things like a DOS you want something at L3 in front of the
gateway that prevents the cost of the TCP handshake too.
`Deny` mode is a bad API with a useful goal: to ensure that policies
MUST match the rule, essentialy adding AND semantics on top of the ALLOW
policies OR semantics.
Deny is trick, because you often end up having double negatives.
For example `deny: !(port in [443, 8443])`. Or worse, somethign like
`deny: jwt.group != 'eng'`; if the jwt doesn't have a `group` this rule
will not deny.
Instead, this proposes adding a new Require mode which is like deny but
inverted.
The above rules would be `require: port in [443, 8443]` or `require:
jwt.group == 'eng'`.
These are useful for admin level rules that apply on top of user level
rules like `require: source.address not in [banned country cidr ranges]`
etc
- Handle 'global' region explicitly in get_host(), returning
aiplatform.googleapis.com instead of global-aiplatform.googleapis.com
- Make VertexAI Region field optional with default 'global' in CRD
- Add unit tests for get_host() covering all region cases
---------
Signed-off-by: Rajat Vig <rvig@etsy.com>
This hardens MCP degraded-mode behavior so multiplexed backends can keep
serving healthy targets without hanging, looping, or corrupting
resumable session state.
The original `failureMode` work was meant to cover three concrete
failure cases:
- `#1162`: zero-target MCP backends should not trap clients in a
reconnect loop
- `#1029`: `failOpen` should skip bad targets, but still fail if every
configured target is unusable
- `#962`: one bad upstream should not take down the rest of a
multiplexed MCP session
This change makes that behavior consistent across init, runtime fanout,
and session resume.
At init time, `failOpen` now allows partial success but still returns
`NoBackends` when all configured targets fail. Zero configured targets
remain a special case so we do not regress the zero-target handling
behind `#1162`. The `failureMode` setting is also threaded through the
controller, local config, schema, and MCP runtime so the configured
behavior is preserved end to end.
At runtime, multiplexed fanout now degrades correctly when individual
upstreams fail. That includes the legacy SSE path, which was the sharp
edge here: a dead legacy SSE transport could leave pending requests
orphaned and hang the whole downstream request. This change makes
wrapped transports fail pending work, fail fast once dead, and treat
dead legacy SSE upstreams as unavailable for the rest of that downstream
session instead of trying to silently rejoin mid-session. That keeps
healthy stdio, SSE, and streamable HTTP targets serving without
introducing questionable session semantics for non-resumable transports.
For resumable streamable HTTP MCP sessions, persisted upstream session
state is now keyed by target name. New session handles include an
optional target field, and resume restores by target identity instead of
positional zip. Old handles without target names still use the legacy
positional path, so existing session IDs remain readable. This avoids
misbinding persisted sessions when `failOpen` changes the initialized
target set.
Most of the diff is test coverage plus generated/schema updates from
wiring the config surface through. The behavior changes themselves are
concentrated in the MCP relay, upstream transport wrappers, and resume
logic.
Fixes#1162Fixes#1029Fixes#962
---------
Co-authored-by: John Howard <john.howard@solo.io>
Like https://github.com/agentgateway/agentgateway/pull/1185 but split
out some unrelated changes
This is like setting HTTP_PROXY=xyz but done with explicit policies not
implicit.
We support a subset of policies only on the tunnel connection: HTTP/1.1
only, authentication, and TLS.
This introduces a new LLM route type, 'Detect'.
Detect is similar to passthrough, but we attempt to detect the
underlying type. This gives us the following types..
* Conversion: complete translation from X to Y. Highest risk since we
need to support every field. Full API support and reporting
* Native types: we passthrough X to X, but have native understanding of
it. Lower risk since we don't need to know about every field, only
things we need to process. Full API support and reporting
* Passthrough: raw passthrough. No API support or reporting at all, this
is mostly so you can easil;y hit unsupported endpoints on the same
backend
* Detect: passthrough the request as is **no matter what** (even invalid
json, invalid SSE, etc). Attempt to detect token usage stuff. Supports
rate limiting and reporting, but not request manipulation.
TODO: today we support `defaults` and `overrides`, but not `model`. we
should support all or none.
Move session key management into the controller-managed object set
instead of creating Secrets as a side effect of Helm value generation.
This makes the per-Gateway Secret the authoritative source of truth for
managed Gateways:
- build and append a managed Secret child object during post-processing
- watch Secret changes so Gateway reconciliation repairs drift and
deletion
- validate existing keys before reuse and generate deterministic test
keys
- reserve SESSION_KEY and SESSION_KEY_FILE in AGWP env for managed
Gateways
- prefer the mounted session key file over env and inline fallback in
the dataplane
Refresh deployer goldens and generated CRD/schema output.
Motivation for this is ergonomics for this :
https://github.com/agentgateway/agentgateway/pull/878
Within the AgentgatewayPolicy CRD there is a bug with the HostRewrite
optional property.
It is simultaneously an `enum` for `Auto`/`None` and of type `object`
with a `mode` property, which are not simultaneously satisfiable.
I believe an adequate fix would be to just move the `enum` into the
`mode`, as it was probably just an indentation bug to begin with.
I'm not sure if there are any considerations to be had about
backwards-compability since, in this case, it goes from not working
(unsatisfiable constraints) to functional (at least on the CRD side).
---------
Co-authored-by: John Howard <john.howard@solo.io>
Evict on 5xx / connection failure so the next request can fail over for
model failover.
Now with cel-based backend policy:
```
binds:
- port: 3000
listeners:
- name: llm-failover
protocol: HTTP
routes:
- backends:
- ai:
groups:
# Priority 0 – primary: try OpenAI first
- providers:
- name: openai
provider:
openAI:
model: gpt-4
# Priority 1 – fallback: local Llama via Ollama
- providers:
- name: llama
hostOverride: "localhost:11434"
pathOverride: /v1/chat/completions
provider:
openAI:
model: llama3.2
policies:
health:
unhealthyExpression: "response.code >= 500"
eviction:
duration: 30s
policies:
backendAuth:
key: "${OPENAI_API_KEY}"
```
Currently, the behavior will still allow the first request to fail, but
then the second request will use the failover provider:
```
# error first time
❯ curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "any", "messages": [{"role": "user", "content": "Say hello in one sentence."}]}'
upstream call failed: Connect: Connection refused (os error 61)%
# now failover to local model
❯ curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "any", "messages": [{"role": "user", "content": "Say hello in one sentence."}]}'
{"model":"llama3.2","usage":{"prompt_tokens":31,"completion_tokens":3,"total_tokens":34},"choices":[{"message":{"content":"Hello!","role":"assistant"},"index":0,"finish_reason":"stop"}],"id":"chatcmpl-498","object":"chat.completion","created":1772382504,"system_fingerprint":"fp_ollama"}%
```
Fixes: https://github.com/kgateway-dev/kgateway/issues/13216
---------
Signed-off-by: npolshakova <nina.polshakova@solo.io>
- updates kgateway-system -> agentgateway-system in dev docs and unit
tests
- adds unit tests for backend AgentgatewayPolicy with multiple
targetRefs
Signed-off-by: npolshakova <nina.polshakova@solo.io>
Part of https://github.com/agentgateway/agentgateway/issues/1078
* Deprecate `config.logging` and `config.tracing` which has been
superseded by frontendPolicies
* Remove reading of those, instead convert to frontendPolicies in local
config processing
* Support path in tracing
* Move MCP to the unified tracing config (for frontendPolicies support)
* Add support to add headers on the backend request
Pulls in part of https://github.com/agentgateway/agentgateway/pull/765/
(the MCP part only; that PR has more)
Workloads currently don't have a trust domain set. This results in the
data plane falling back to using `cluster.local` as the trust domain.
Setting Istio to use a different trust domain didn't affect the
Workloads.
The PR is basically replacing the dummy empty mesh config with the real
Istio one (if exists) and by that causing the builder to set the
workloads with the trust domain from the Istio Mesh Config.
This PR replaces https://github.com/agentgateway/agentgateway/pull/1064
following important
[comment](https://github.com/agentgateway/agentgateway/pull/1064#issuecomment-3967999405).
Reverts agentgateway/agentgateway#1065 because the unittests didn't use
a real k8s API server. A real k8s rejects any CRD that attempts to use a
null map value, and you cannot easily change the CRD because
controller-tools is opinionated about that. I don't want to add new
'remove' APIs for this edge case or kustomize a CRD.
Fixes a TODO:
```
metadata:
labels:
# You'd think this would remove the app.kubernetes.io/instance label,
# but it just sets the label to the empty string. TODO(chandler):
# Figure out the incantation to cause removal of the label.
app.kubernetes.io/instance: null
```
---------
Signed-off-by: David L. Chandler <david.chandler@solo.io>
This lets you write policies on the LLM request directly. For example:
```yaml
transformation:
max_tokens: min(llmRequest.max_tokens, 1024) # I am cheap, don't let my users burn tokens
```
gateway api is going to make listener optional:
https://github.com/kubernetes-sigs/gateway-api/issues/4425. The problem
is, if you go 0 listeners, the service is invalid. Worse, if you go
0->1->0 we would lose the service IP!
So we put a dummy one.
Updating links in the docs and found some issues with links in the
auto-generated reference docs from this repo. Thanks!
Signed-off-by: Kristin Brown <kristin.brown@solo.io>
The release workflow now publishes the controller package too. Tested it
on my fork: https://github.com/npolshakova/agentgateway/packages
Currently tested:
- [x] make generate-all
- [x] make analyze
- [x] make fmt
- [x] make unit - passing
- [x] make run
- [x] make package-agentgateway-charts
- [x] make kind-build-and-load
- [x] make kind-load-agentgateway
---------
Signed-off-by: npolshakova <nina.polshakova@solo.io>
JWT, BasicAuthentication, and APIKeyAuthentication policies were
ignoring the `phase` parameter. The data plane already supports these at
gateway level, but the control plane wasn't threading the phase
information through.
## Changes
- **Validation rule**: Updated to allow `jwtAuthentication`,
`basicAuthentication`, and `apiKeyAuthentication` in PreRouting phase
alongside existing policies (extAuth, transformation, extProc)
- **Policy processing**: Added `policyPhase` parameter to
`processJWTAuthenticationPolicy`, `processBasicAuthenticationPolicy`,
and `processAPIKeyAuthenticationPolicy` functions. Each now calls
`phase(policyPhase)` when building the TrafficPolicySpec.
- **Translation layer**: Updated `translateTrafficPolicyToAgw` to pass
`traffic.Phase` to all three authentication policy processors.
## Example
```yaml
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
spec:
targetRefs:
- kind: Gateway
name: my-gateway
traffic:
phase: PreRouting # Now respected by auth policies
jwtAuthentication:
mode: Strict
providers: [...]
```
The phase now correctly propagates: `PreRouting` → `GATEWAY` in the
proto output.
> [!WARNING]
>
> <details>
> <summary>Firewall rules blocked me from connecting to one or more
addresses (expand for details)</summary>
>
> #### I tried to connect to the following addresses, but was blocked by
firewall rules:
>
> - `cgr.dev`
> - Triggering command: `/usr/libexec/docker/cli-plugins/docker-buildx
/usr/libexec/docker/cli-plugins/docker-buildx buildx build --build-arg
VERSION=eaeb1fb-dirty --build-arg
GIT_REVISION=eaeb1fb553685bc5442de8bf8b2e08149edd4993 -t
ghcr.io/agentgateway/agentgateway:eaeb1fb-dirty . --progress=plain
0.1-go1.25.1.lin/tmp/cc60Bcwp.o -ato -bool t
rg/toolchain@v0.0.1-go1.25.1.linux-amd64/pkg/tool/linux_amd64/compile
-errorsas -ifaceassert t
rg/toolchain@v0.0.1-go1.25.1.lin/tmp/ccQMzFBl.o` (dns block)
>
> If you need me to access, download, or install something from one of
these locations, you can either:
>
> - Configure [Actions setup
steps](https://gh.io/copilot/actions-setup-steps) to set up my
environment, which run before the firewall is enabled
> - Add the appropriate URLs or hosts to the custom allowlist in this
repository's [Copilot coding agent
settings](https://github.com/agentgateway/agentgateway/settings/copilot/coding_agent)
(admins only)
>
> </details>
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
>
> ----
>
> *This section details on the original issue you should resolve*
>
> <issue_title>Some pre routing policies are missing control plane
support</issue_title>
> <issue_description>in the traffic policies implementation, we ignore
the phase for some policies. We need this for any that support being pre
routing. I know jwt has this issue but there may be some others
too</issue_description>
>
> ## Comments on the Issue (you are @copilot in this section)
>
> <comments>
> </comments>
>
</details>
<!-- START COPILOT CODING AGENT SUFFIX -->
- Fixesagentgateway/agentgateway#933
<!-- START COPILOT CODING AGENT TIPS -->
---
✨ Let Copilot coding agent [set things up for
you](https://github.com/agentgateway/agentgateway/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot)
— coding agent works faster and does higher quality work when set up for
your repo.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: howardjohn <623453+howardjohn@users.noreply.github.com>