Summary
I built a multi-agent platform that lives inside Microsoft Teams and puts two production workflows behind one intent router. The first is enterprise RAG: grounded, cited answers over an enterprise SharePoint corpus using a Bedrock Knowledge Base. The second is agentic AWS account vending: a Bedrock AgentCore runtime that gathers intent from a conversation, proposes network configuration, and — only after a fail-closed human approval — performs a validated commit and workflow dispatch into an internal account-factory pipeline. The whole thing is serverless. The interesting part is not the service list; it is the set of constraints I hit and the boundaries I drew around them.
The problem
Two costs motivated this.
The first is repeated human answering. An enterprise IT/cloud platform team fields the same questions continuously: how do I request access, what is the standard for X, which pipeline owns Y. The answers exist, written down, in SharePoint. They are just not reachable at the moment of the question. Engineers ask a human because search is worse than asking a human. That converts a documentation problem into a permanent staffing cost, and it degrades every time the docs change without anyone re-reading them.
The second is provisioning latency. Getting a new AWS account is a ticket. A human reads the request, checks whether the proposed CIDR collides with anything already deployed across the organization, checks whether the SSO entities exist, then hand-edits a config file and triggers a pipeline. The work is mechanical but the verification is genuinely hard — CIDR collision requires an org-wide read — so it sits in a queue. Days of lead time for an operation whose actual decision content is small.
Both are conversational problems with a deterministic core. That shape is what the platform is built for.
Architecture
One entry point, one router, two workflows.
The text flow below is authoritative; the diagram is a reading aid.
Microsoft Teams
|
Azure Bot Service
|
API Gateway (HTTP API)
|
Router Lambda [Terraform] <- validates Bot Framework JWT, classifies intent
|
+--(a) RAG query Lambda
| -> Bedrock Knowledge Base
| -> OpenSearch Serverless vector collection
| -> S3 data source (synced from SharePoint via Microsoft Graph)
|
+--(b) Account Vending AgentCore Runtime [AgentCore CLI -> CDK -> CloudFormation]
-> AgentCore Gateway (MCP, AWS_IAM / SigV4)
-> org-reads Lambda target
(list_accounts, list_vpc_cidrs, check_sso_entities)
-> GitHub contents API + workflow_dispatch
API Gateway is internet-reachable. It has to be: Azure Bot Service is the caller and it originates outside the VPC. The router Lambda validates Bot Framework JWTs before any dispatch. That is a deliberate, documented tradeoff rather than an accident — the alternative is a private ingress path that Bot Service cannot use.
The RAG pipeline:
SharePoint folder
-> Microsoft Graph sync Lambda (Sites.Selected)
-> S3 source bucket (per-file .metadata.json sidecar carrying citation source URLs)
-> Bedrock KB S3 data source
-> amazon.titan-embed-text-v2:0 (1024 float dims)
-> OpenSearch Serverless, FAISS / HNSW index
-> hybrid retrieval, top 25
-> cohere.rerank-v3-5:0, top 5
-> Claude, grounded answer + source citations (guardrails applied)
Nightly EventBridge schedule re-runs the sync and starts a fresh ingestion job.
Design decisions and their tradeoffs
I rejected Bedrock's native SharePoint connector and built a Graph → S3 → Bedrock S3 data source pipeline instead. The native connector wants Sites.Read.All, which is tenant-wide read on every site. The application holds a scoped Sites.Selected grant on exactly one folder, and widening that to satisfy a connector is a real least-privilege regression for a convenience win. The other supported path used a retired Azure ACS app-only mechanism. The managed ENTRA_ID_APP_ONLY connector is not expressible in the Terraform AWS provider, which would have forced a click-ops or custom-resource seam into an otherwise declarative stack. Cost of my choice: I own a sync Lambda and its failure modes. Benefit: the scoped grant survives and the entire pipeline stays in Terraform.
The embedding model is pinned to amazon.titan-embed-text-v2:0 at 1024 dimensions because it must be Knowledge-Base-supported, not merely available in Bedrock. cohere.embed-v4:0 is direct-InvokeModel only and fails CreateKnowledgeBase. There is no way to learn this from the model catalog; you learn it from a failed API call. I added a CI preflight that verifies model availability *for the intended integration* before a deploy runs, so the next person hits a build failure with a clear message instead of a confusing control-plane error.
Retrieval is two-stage: hybrid search to top 25, then rerank to top 5. Raw vector top-k puts too much marginal context in front of the generator, and marginal context is where ungrounded answers come from. Reranking costs a model call per query and buys precision at the point where it matters — before generation, not after.
I migrated the vending half off Terraform and onto the AgentCore CLI. The original shape was a FastMCP container behind a Lambda Function URL, wired by a Terraform module. The CLI shape is a declarative agentcore.json that compiles to CDK and then CloudFormation. Three things fell out of that. The org-reads tool target became a plain Python Lambda — no container image, no Function URL, no ECR lifecycle. Deployment became one declarative artifact instead of hand-assembled resources. And I got ADOT/OpenTelemetry tracing that the container build never had.
That leaves two infrastructure owners in one system: Terraform owns the RAG and platform half, the AgentCore CLI owns the vending half. I consider that acceptable rather than a purity violation, because the seam is clean and the lifecycles differ. The two halves share no state, only an invocation contract. The agent runtime changes on the cadence of agent behavior; the platform changes on the cadence of infrastructure. Forcing them into one tool would mean re-implementing the CLI's CDK output by hand and losing the parts that made the migration worth doing. I removed the legacy Terraform module only after it was scaled to zero and held no state, so the deletion was a genuine no-op to terraform plan.
AgentCore Gateway namespaces tools as <TargetName>___<tool>, and the agent refuses to boot unless every required tool resolves exactly once. Renaming a target breaks startup. Worse, the CLI *derives* runtime environment variable names from resource names, so renaming a resource silently renames its env var. Configuration resolves SSM → environment → default, and the code tries the derived name first and falls back to a flat name. That fallback is not elegance; it is defense against a rename doing something invisible in a place I would not look.
Cedar is attached in LOG_ONLY, default-deny, with no permits yet. This is staged, not unfinished. A tool-specific permit must scope its resource to the concrete Gateway ARN — an unscoped resource is rejected at CreatePolicy — and that ARN is only knowable after the first deploy. The sequence is: deploy, observe decision logs to learn the real call shapes, write permits against them, then flip to ENFORCE. Writing permits before seeing traffic would just produce guesses that get relaxed under pressure.
Gateway authorizer configuration cannot be changed in place. Moving off AWS_IAM means remove → deploy → recreate → re-add every target → deploy. I treat the authorizer choice as an irreversible decision and plan around it rather than assuming a later migration is cheap.
Access control is coarse today, and I want to be explicit about it. By default every authenticated user queries the same shared corpus. A user allowlist exists, but that is a gate on the door, not per-document SharePoint ACL enforcement. This is safe for a single-sensitivity corpus and is *not* safe for mixed-sensitivity content. An ACL-aware retrieval design is a prerequisite before indexing anything else, and I would rather state that than let the current scope imply a guarantee it does not make.
The safety model
Exactly one tool in the vending agent mutates anything: commit_and_dispatch. It is the only interrupted tool.
conversation -> model gathers intent, proposes a CIDR
-> HITL approval card in Teams
-> user approves (router binds approval to the ORIGINAL requester)
-> INSIDE commit_and_dispatch:
fresh Gateway MCP reads (fresh=true, strict=true)
list_accounts / list_vpc_cidrs / check_sso_entities
deterministic validation:
account name, email, CIDR, subnets, SSO entities
-> GitHub contents write
-> workflow_dispatch
The threat model is specific. An LLM will propose a plausible CIDR. Plausible is not the same as free. If the model's proposal, or a read cached earlier in the conversation, is what gets written, then a hallucination or a few minutes of drift becomes a real VPC collision in a real organization. So the reads that authorize the write are executed inside the mutating call, at commit time, bypassing the advisory cache entirely. An incomplete org scan is an explicit error, never a short list — a partial read that looks like a clean read is the worst possible failure here. Cross-account CIDR enumeration uses an org-wide read-only role deployed by StackSet.
The pattern generalizes: the model is allowed to be probabilistic while it is exploring, and determinism is reintroduced at the mutation boundary. Human approval authorizes *intent*; fresh deterministic validation authorizes the *values*. Those are not the same check and collapsing them is the bug.
Operating it
304 unit tests run across two dependency tiers, so the platform half and the agent half can be tested without either one dragging in the other's runtime.
CI gates two things that are cheap to check and expensive to discover late. IAM Access Analyzer validates policy documents before they are applied. A Bedrock model preflight confirms the configured models exist and are usable for the integration they are configured for — the lesson from the embedding-model constraint above, encoded so it cannot recur silently. Deploys authenticate via OIDC; there are no long-lived deploy credentials.
Observability is LangSmith tracing plus AgentCore observability with ADOT/OpenTelemetry from the CLI build. AgentCore online evaluations run GoalSuccessRate, ToolSelectionAccuracy, and ToolParameterAccuracy at 100% sampling. One operational trap worth naming: CloudWatch Transaction Search must be enabled, or the spans never land — and because the online evaluations read those spans, the evaluations return nothing too. It fails quietly in both places at once. AWS X-Ray is intentionally disabled for the RAG and router Lambdas; the tracing that matters for those paths is already captured elsewhere.
What I'd do differently, and what's next
- The Teams router is not yet bound to the CLI-deployed vending runtime. The migration changed the invocation target and the router still points at the old shape, so vending mode stays hidden in Teams until that binding is added. This is the top item.
- Cedar is LOG_ONLY, not ENFORCE. The staged plan is deliberate, but until permits are written and enforcement is on, authorization is advisory.
- No per-document ACL enforcement. As above: fine for one corpus, blocking for mixed sensitivity.
- A non-production ingestion run and an end-to-end smoke test are still required before a production RAG cutover. I would not call the retrieval path production-ready on unit tests alone; the failure modes that matter here are data-shaped.
If I were starting again, the one thing I would change early is the model-capability preflight. Nearly every expensive surprise in this build was a service supporting something in one integration path and not another — the SharePoint connector, the embedding model, the authorizer immutability. Those are all the same class of problem, and I found each one the slow way.