Runbook: credential brokering (ADR-040)¶
Moves bedrock:Invoke* off developer/node IAM entirely: inferplaned holds
one STS broker role and vends ≤1-hour session credentials to each mayu over
POST /v1alpha1/credentials; mayu's bedrock provider opts in with
auth.mode: "broker". On hosts where mayu's environment is not readable by
its users (K8s nodes, dedicated proxy hosts), bypassing mayu then yields no
Bedrock credentials at all. On a developer-owned machine the developer can
read the broker token out of mayu's env — there this is an IAM-hygiene win
(no standing node grant), not a bypass control. See ADR-040 for the full
threat model.
1. IAM wiring (do this first — it is two-sided)¶
Create the broker role (example name inferplane-bedrock-broker):
Identity policy — Bedrock invoke only, ideally network-conditioned:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
"Resource": "*",
"Condition": {
"IpAddress": { "aws:SourceIp": ["<your corporate/VPC CIDRs>"] }
}
}]
}
Strongly recommended: the
aws:SourceIp(oraws:SourceVpcefor VPC endpoints) condition makes an exfiltrated credential triplet dead on arrival outside your network — the cheapest mitigation for the portable-credentials risk ADR-040 accepts. Without it, anyone holding a minted triplet has Bedrock access from anywhere for up to 1 hour.
Trust policy — must allow all three actions, or every AssumeRole fails
(denying sts:TagSession/sts:SetSourceIdentity in the trust policy fails
the whole call, not just the tag):
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::<account>:role/<inferplaned task role>" },
"Action": ["sts:AssumeRole", "sts:TagSession", "sts:SetSourceIdentity"]
}]
}
Then add to inferplaned's task role the matching identity-policy grant:
{
"Effect": "Allow",
"Action": ["sts:AssumeRole", "sts:TagSession", "sts:SetSourceIdentity"],
"Resource": "arn:aws:iam::<account>:role/inferplane-bedrock-broker"
}
Session lifetime is capped at 1 hour regardless of the role's configured
maximum — inferplaned's task role is itself an assumed role, so this is
STS role chaining. Do not raise the role's MaxSessionDuration expecting
longer broker sessions; it has no effect on chained sessions.
2. Enable it on inferplaned¶
Two env vars (inferplaned has no config file):
INFERPLANED_BROKER_ROLE_ARN=arn:aws:iam::<account>:role/inferplane-bedrock-broker
INFERPLANED_BROKER_TOKEN=<long random secret — NOT the value of INFERPLANED_TOKEN>
| Var | Required | Notes |
|---|---|---|
INFERPLANED_BROKER_ROLE_ARN |
opt-in switch | Unset ⇒ POST /v1alpha1/credentials does not exist (404); behavior is byte-identical to before ADR-040. |
INFERPLANED_BROKER_TOKEN |
when the ARN is set | Boot fails if missing, if JWT-shaped, or if equal to INFERPLANED_TOKEN — the heartbeat token sits in every node's env and grants policy reads; sharing it would turn any one node compromise into a Bedrock grant without mayu. Keep this secret in strictly fewer places than the heartbeat token. |
The endpoint authenticates with the broker token ONLY. A console SSO session (OIDC bearer) is rejected with 403 — a browser must never be able to mint AWS credentials.
3. Enable it on mayu¶
{
"control_plane": {
"url": "https://inferplaned.example.com",
"token_ref": {"env": "INFERPLANED_TOKEN"},
"broker_token_ref": {"env": "INFERPLANE_BROKER_TOKEN"},
"dataplane": "build-farm-node-07"
},
"providers": {
"bedrock-global": {
"type": "bedrock",
"region": "ap-northeast-2",
"auth": {"mode": "broker"}
}
}
}
Config load rejects, with a clear message, any of:
- auth.mode outside default|irsa|pod_identity|profile|static|broker
(previously any string silently meant "default" — a typo now fails boot
instead of silently deploying the bypassable local-IAM posture);
- broker without a control_plane block or without broker_token_ref;
- broker_token_ref resolving to the same value as token_ref;
- broker with a non-loopback plain-http control_plane.url —
credentials must not transit plaintext (the heartbeat tolerates http;
this channel does not).
Set control_plane.dataplane explicitly. The default id is
per-boot (hostname-ULID), so every mayu restart starts a new CloudTrail
session-name trail. A stable id gives you a stable audit line:
CloudTrail's AssumeRole events carry RoleSessionName, SourceIdentity,
and a dataplane session tag, all set to the sanitized id.
4. Make bypass prevention real (the part inferplane cannot do for you)¶
Remove bedrock:InvokeModel* from the node/developer roles that previously
carried it. Until you do, brokering only adds a path — the old direct
path still works and can still be used to skip mayu. After you do, the only
Bedrock credentials in the fleet are ≤1h broker sessions, revocable
fleet-wide by disabling one role.
5. Failure modes¶
| Symptom | Meaning / fix |
|---|---|
mayu boot/SIGHUP fails: credential source ... Retrieve |
The eager first fetch failed — broker unreachable, wrong broker token, or inferplaned's ARN/permissions wrong. This is fail-closed by design: a bad broker config never silently falls back to node IAM. On SIGHUP the OLD topology stays live (ADR-006 rollback). |
| Bedrock calls start failing ~1h after a control-plane outage | Cached session expired and no renewal is possible. Same posture as hard-cap leases: no control plane ⇒ (eventually) no credentials. Restore inferplaned; mayu recovers on the next signing. |
| inferplaned boot fails: broker token missing/equal/JWT-shaped | See §2 table. |
502 from /v1alpha1/credentials with a fixed body |
STS AssumeRole failed server-side; the real error (which can name the role ARN) is in inferplaned's log, deliberately not in the response. Check the two-sided IAM wiring in §1 — the most common miss is sts:TagSession/sts:SetSourceIdentity absent from the broker role's trust policy. |
CloudTrail shows sessions without SourceIdentity |
inferplaned's own task-role session already carried an inherited SourceIdentity (AWS forbids changing it on the next hop). The broker retried without it; the dataplane session tag still attributes the session. |
| Sessions in CloudTrail under an id you don't recognize | v1 limitation (ADR-040 §4): the dataplane id is claimed by the caller, and any broker-token holder can claim any id. Attribution is "which id was claimed," not "which machine called" — per-dataplane identity is the v2 follow-up. Rotate INFERPLANED_BROKER_TOKEN if you suspect misuse. |
What this does NOT do¶
- Broker 1P Anthropic API keys — no temporary-token mechanism exists for
api.anthropic.com; that channel keeps using local
env:/file:refs. - Restrict which models a brokered session may invoke — v1 sessions carry
the broker role's full
bedrock:Invoke*. Model allow-lists are still enforced by mayu's policy layer for traffic that goes through mayu; per-team session policies at the IAM layer are the v2 candidate. - Prove which machine called — see the last failure-mode row.