Skip to content

Commit 3d50e04

Browse files
cgwalterscgwalters
committed
docs(adr): add ADR-0006 for workspace container architecture
Increase flexibility by allowing users to configure the container used by the agent's tools (separate from the container running the agent itself). Assisted-by: Claude Code (Opus 4.5)
1 parent 6c210f1 commit 3d50e04

File tree

1 file changed

+252
-0
lines changed

1 file changed

+252
-0
lines changed

‎docs/adr/0006-agent-injection-architecture.md‎

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# ADR-0006: Workspace Container Architecture
2+
3+
**Status**: Pending
4+
**Date**: 2025-11-24 (Updated: 2025-12-04)
5+
**Deciders**: Platform Architecture Team
6+
**Related**: [ADR-0001 Kubernetes Native Architecture](0001-kubernetes-native-architecture.md)
7+
8+
## Context
9+
10+
Agents need access to tools specific to their task: compilers, test frameworks, language runtimes. Previously, the agent ran directly in the runner container with limited, fixed tooling.
11+
12+
This ADR introduces **workspace containers**: separate pods where agents execute commands, with user-configurable container images.
13+
14+
## Decision
15+
16+
### Workspace Containers
17+
18+
Each agentic session runs with two pods:
19+
20+
1. **Runner Pod**: Contains the Claude Code agent and a session proxy sidecar. The agent has no direct shell access—all command execution goes through the proxy.
21+
2. **Workspace Pod**: A user-configurable container where commands actually execute. Shares a PVC with the runner for file access.
22+
23+
```mermaid
24+
graph LR
25+
subgraph "Runner Pod"
26+
Runner[Runner Container<br/>Claude Code Agent]
27+
Proxy[Session Proxy<br/>Sidecar]
28+
end
29+
subgraph "Workspace Pod"
30+
Workspace[Workspace Container<br/>User's Image]
31+
end
32+
PVC[(Shared PVC<br/>/workspace)]
33+
34+
Runner -->|localhost:8080| Proxy
35+
Proxy -->|kubectl exec| Workspace
36+
Runner -.->|reads/writes| PVC
37+
Workspace -.->|reads/writes| PVC
38+
```
39+
40+
By default, the workspace pod uses a base development image (`quay.io/ambient_code/vteam_workspace`). Users can customize this per-project via ProjectSettings.
41+
42+
### Configuration
43+
44+
Users configure workspace containers in the ProjectSettings UI under "Workspace Container". The **Container Image** field accepts a custom image with required tooling (e.g., `node:20`, `rust:1.75`, `python:3.12`); leaving it empty uses the platform default. The **Resource Limits** field allows setting CPU and memory requests/limits for the workspace pod.
45+
46+
The ProjectSettings CR schema:
47+
48+
```yaml
49+
apiVersion: vteam.ambient-code/v1alpha1
50+
kind: ProjectSettings
51+
metadata:
52+
name: settings
53+
namespace: my-project
54+
spec:
55+
workspaceContainer:
56+
image: "node:20-slim" # Optional - defaults to runner image
57+
resources: # Optional
58+
cpuRequest: "500m"
59+
cpuLimit: "2"
60+
memoryRequest: "512Mi"
61+
memoryLimit: "4Gi"
62+
```
63+
64+
### Session Proxy Sidecar
65+
66+
The session proxy is a lightweight Go binary that runs as a sidecar container within the runner pod. It listens on localhost:8080 and provides an HTTP streaming endpoint for command execution. The agent's MCP workspace tool makes streaming HTTP calls to the proxy, which uses kubectl exec to run commands in the workspace pod and streams output back in real-time.
67+
68+
```mermaid
69+
sequenceDiagram
70+
participant Runner as Runner Container
71+
participant Proxy as Session Proxy
72+
participant Workspace as Workspace Pod
73+
74+
Runner->>Proxy: POST /exec (streaming)<br/>{command: "npm i", cwd: "/workspace"}
75+
76+
Proxy->>Workspace: kubectl exec <workspace-pod> -- sh -c "npm i"
77+
78+
Workspace-->>Proxy: stdout: "npm WARN..."
79+
Proxy-->>Runner: chunk: "npm WARN..."
80+
81+
Workspace-->>Proxy: stdout: "added 50 packages"
82+
Proxy-->>Runner: chunk: "added 50 packages"
83+
84+
Workspace-->>Proxy: exit code: 0
85+
Proxy-->>Runner: {exit: 0}
86+
```
87+
88+
**Why a sidecar instead of the operator?** Keeping the exec API within the runner pod eliminates a network hop to the operator and avoids exposing an HTTP endpoint on the operator. The operator remains a pure watch-based controller. The proxy holds the credentials for pod exec—the runner container has no Kubernetes API access.
89+
90+
### Why Streaming
91+
92+
For long-running commands (npm install, cargo build, pytest), users need real-time feedback:
93+
94+
| Approach | Latency | Streaming | UX |
95+
|----------|---------|-----------|-----|
96+
| CRD polling | ~200ms + poll interval | No | Wait for completion, then see all output |
97+
| Direct streaming | ~10ms | Yes | See output as it happens |
98+
99+
The MCP protocol supports streaming tool responses, enabling the agent to observe build output, test results, and errors in real-time.
100+
101+
### Pod Isolation
102+
103+
The runner and workspace run as separate pods. Within the runner pod, the agent container and proxy sidecar share a network namespace (allowing localhost communication) but have separate filesystem namespaces. The agent cannot directly access the proxy's credentials or make Kubernetes API calls.
104+
105+
**Session Isolation**: The proxy discovers its workspace pod by label selector (`session=<name>,type=workspace`), not by user-provided pod names. This prevents an agent from targeting another session's workspace even if it could somehow influence the proxy's requests.
106+
107+
**Privilege Separation**: The runner pod disables automatic service account token mounting (`automountServiceAccountToken: false`). The proxy sidecar alone receives the token via a projected volume mount, giving it `pods/exec` permission scoped to workspace pods in the namespace. The runner container has no token and cannot make Kubernetes API calls. The workspace pod has no special permissions.
108+
109+
### Disabling Native Bash
110+
111+
The agent MUST NOT have access to the native Bash tool. This prevents bypassing the proxy, direct execution in the runner container, and running kubectl or other cluster tools.
112+
113+
The runner wrapper configures allowed tools explicitly, excluding Bash:
114+
115+
```python
116+
allowed_tools = ["Read", "Write", "Glob", "Grep", "Edit", "WebSearch", "WebFetch"]
117+
# Bash is NOT in the list - all execution goes through workspace MCP tool
118+
```
119+
120+
### Security Model
121+
122+
| Layer | Enforcement |
123+
|-------|-------------|
124+
| **Network isolation** | Proxy listens only on localhost; no external network access to exec API |
125+
| **Session isolation** | Proxy discovers workspace by label; cannot target other sessions |
126+
| **Token isolation** | Only proxy sidecar receives SA token via projected volume |
127+
| **Privilege separation** | Runner has no K8s API access; proxy has only `pods/exec` |
128+
| **No Bash** | Agent cannot execute commands outside the proxy workflow |
129+
130+
### Threat Mitigations
131+
132+
| Threat | Mitigation |
133+
|--------|------------|
134+
| Agent creates arbitrary pods | Runner has no SA token; proxy SA only has `pods/exec` |
135+
| Agent accesses other session's workspace | Proxy uses label selector; validates namespace match |
136+
| Agent escalates privileges in workspace | SecurityContext enforces non-root, dropped capabilities |
137+
| Agent steals proxy token | Token mounted only in proxy container, not runner |
138+
| Agent bypasses proxy | No Bash tool; no kubectl binary in runner container |
139+
140+
### Transparent Version Upgrades
141+
142+
The operator handles platform upgrades transparently without disrupting running sessions. When the operator deployment is upgraded with new workspace container images or session proxy versions, the system maintains continuity by allowing running sessions to continue with their current images while new sessions automatically receive the updated components.
143+
144+
This separation of concerns is made possible by the pod-based architecture. Each AgenticSession creates its own runner and workspace pods with image references resolved at creation time. Upgrading the operator deployment updates the default image references in the operator's configuration but does not affect pods that already exist in the cluster. When a running session's runner pod continues executing, it uses the workspace container image and proxy sidecar version that were current when the session started.
145+
146+
The workspace pod and runner pod are separate entities, enabling independent lifecycle management. In theory, workspace container images could be upgraded without touching the runner pod, though this would require additional controller logic to detect configuration changes and recreate only the workspace pod. The current implementation treats both pods as immutable—upgrading either component means creating a new session.
147+
148+
Session proxy upgrades require restarting the runner pod because the proxy runs as a sidecar container within that pod. Kubernetes does not support in-place container updates; the entire pod must be replaced. However, this limitation has minimal impact because AgenticSessions are typically short-lived (minutes to hours, not days). Long-running interactive sessions could theoretically be migrated gracefully by checkpointing state and recreating the pod, but this complexity is not justified given current usage patterns. Users can always resume work by starting a new session if an upgrade interrupts a running one.
149+
150+
The operator tracks component versions through pod annotations and labels. Each runner and workspace pod receives annotations indicating which operator version created it and which image versions are in use. This metadata enables debugging (identifying which sessions run old vs. new images), monitoring (tracking rollout progress as old sessions complete and new ones start), and potential future migration logic (triggering graceful session termination when a deprecated image version reaches end-of-life).
151+
152+
Version tracking also prevents accidental downgrades. If an operator deployment rolls back to an earlier version, new sessions will use older images, but the operator will not attempt to "fix" newer sessions by restarting them with downgraded images. Each session's image references are immutable once created, stored in the pod spec rather than dynamically resolved.
153+
154+
This approach balances operational simplicity with user experience. Platform administrators can deploy operator upgrades at any time without coordinating with active users. Running sessions continue uninterrupted, and users benefit from improvements when they start their next session. The upgrade window is effectively zero—there is no maintenance downtime where new sessions cannot be created.
155+
156+
### AgenticTask CRD (Tekton-inspired)
157+
158+
Create a Custom Resource for each command, with the operator reconciling execution.
159+
160+
**Rejected** because it provides no streaming (the agent must poll for final results), adds significant latency (CR creation + etcd write + reconciliation adds ~200ms+ overhead), and causes CR pollution (completed tasks accumulate, requiring TTL/GC logic).
161+
162+
### Namespace-per-Session
163+
164+
Create an ephemeral namespace for each AgenticSession.
165+
166+
**Deferred**: Adds ~200-500ms namespace creation overhead. Current pod-level isolation is sufficient. May revisit for stricter multi-tenancy requirements—see Future Directions.
167+
168+
### Operator Exec API
169+
170+
Have the operator expose an HTTP endpoint that the runner calls to execute commands in the workspace.
171+
172+
**Rejected** because it adds a network hop and latency, requires the operator to expose an HTTP API (moving beyond pure watch-based design), and creates a single point of failure for all command execution across the cluster.
173+
174+
### Direct kubectl exec from Runner
175+
176+
Give runner SA `pods/exec` permission and exec directly into a separate workspace pod.
177+
178+
**Rejected** because the runner could exec into any pod in the namespace and it would require mounting a service account token, expanding the attack surface.
179+
180+
## Consequences
181+
182+
### Positive
183+
184+
This architecture delivers real-time feedback through streaming output for builds, tests, and other long-running commands. The localhost HTTP call within the pod provides minimal latency. The operator remains a pure watch-based controller with no HTTP API exposure. Each session is self-contained—the proxy sidecar requires no cluster-wide coordination. Since there are no command CRs, no garbage collection is needed.
185+
186+
### Negative
187+
188+
The sidecar adds resource overhead to each runner pod (though the proxy is lightweight). The proxy requires a service account with `pods/exec` permission, adding RBAC complexity. In-flight commands are lost if the pod restarts, though this is inherent to any streaming approach.
189+
190+
## Implementation Status
191+
192+
| Phase | Status |
193+
|-------|--------|
194+
| Session proxy sidecar | In Progress |
195+
| Runner pod with sidecar spec | In Progress |
196+
| Proxy SA and RBAC | In Progress |
197+
| MCP workspace tools | In Progress |
198+
| Hardening (rate limits, resource limits) | Pending |
199+
200+
## Future Directions
201+
202+
### Agent-Managed Namespaces
203+
204+
The current architecture restricts agents to executing commands in a single workspace container. A natural evolution would allow agents to create and manage their own Kubernetes namespaces with full infrastructure capabilities:
205+
206+
```mermaid
207+
graph TB
208+
subgraph "Project Namespace"
209+
Session[Session Pod]
210+
Operator[Operator]
211+
end
212+
213+
subgraph "Agent Namespace (ephemeral)"
214+
WS[Workspace Pod]
215+
DB[(Database Pod)]
216+
Cache[Redis Pod]
217+
API[API Service Pod]
218+
end
219+
220+
Session -->|requests via CR| Operator
221+
Operator -->|creates/manages| WS
222+
Operator -.->|future: create/manage| DB
223+
Operator -.->|future: create/manage| Cache
224+
Operator -.->|future: create/manage| API
225+
```
226+
227+
**Use cases** include running integration tests that need a database, cache, and API services; deploying and testing microservices architectures; and spinning up ephemeral preview environments.
228+
229+
**Design considerations**: The system would enforce strict resource quotas on namespace resources (CPU, memory, pods, services) and network policies for isolation from other namespaces with controlled egress. Automatic namespace deletion when the session ends (TTL/cleanup) ensures resources don't accumulate. Pods within the agent namespace would be able to discover and communicate with each other, and scoped secrets would handle database credentials and API keys.
230+
231+
**Security model**: The operator creates the namespace with predefined RBAC and quotas. The agent receives a scoped ServiceAccount within that namespace but cannot modify namespace-level resources such as quotas or network policies. All resources inherit OwnerReference to the AgenticSession for cleanup.
232+
233+
This approach maintains the security boundary (agent never directly touches the host cluster) while enabling more sophisticated workloads.
234+
235+
### Devcontainer Auto-Detection
236+
237+
Many repositories include a `.devcontainer/devcontainer.json` file that specifies the development environment—container image, features, extensions, and post-create commands. A natural enhancement would have the operator automatically detect and honor this configuration when creating workspace containers.
238+
239+
When a session starts, the operator would check the cloned repository for `.devcontainer/devcontainer.json` (or `.devcontainer.json` at the root). If found, the operator extracts the `image` field (or builds from `dockerfile`/`build.dockerfile`) and applies relevant configuration such as environment variables, mount points, and post-create commands. This eliminates the need for users to manually configure workspace containers in ProjectSettings when the repository already defines its environment.
240+
241+
**Precedence**: Explicit ProjectSettings configuration would override devcontainer detection, allowing users to force a specific image when needed. Repositories without devcontainer configuration fall back to the platform default.
242+
243+
**Scope limitations**: Not all devcontainer features translate directly to Kubernetes. VS Code extensions, port forwarding rules, and certain lifecycle hooks would be ignored or adapted. The operator would support a subset focused on container image, environment variables, and initialization commands.
244+
245+
**Security considerations**: The operator must validate devcontainer configurations before applying them. Arbitrary image references from untrusted repositories pose supply chain risks, so organizations may want to restrict allowed registries or require image allowlisting. Post-create commands execute in the workspace container's security context, inheriting all existing sandboxing.
246+
247+
## References
248+
249+
- [Kubernetes Pod Security](https://kubernetes.io/docs/concepts/security/pod-security-standards/)
250+
- [Projected Volumes](https://kubernetes.io/docs/concepts/storage/projected-volumes/)
251+
- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)
252+
- [Dev Container Specification](https://containers.dev/implementors/json_reference/)

0 commit comments

Comments
 (0)