diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index d0aea9d..2890165 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -15,8 +15,8 @@ {"id":"forgejo-mcp-broker-9jh","title":"Phase 1: internal/store with SQLite open and embedded schema migrations","description":"Implement internal/store: wraps a modernc.org/sqlite connection, applies embedded schema migrations in order via a schema_migrations table, exposes a *sql.DB and a Close method. Phase 1 schema is just the migrations table itself plus a health_check row — real tables (clients, auth_codes, access_tokens, refresh_tokens) ship in phase 2. Store_path from config; creates parent dirs if missing; fails fast on unwritable path. Migrations embedded via embed.FS under internal/store/migrations/.","acceptance_criteria":"go test ./internal/store passes; opening a fresh db file applies migrations; re-opening is idempotent (no re-application, no errors); corrupt/locked files yield a clear error; Close() leaves no file handles open.","status":"closed","priority":1,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T14:46:19Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-24T15:22:53Z","started_at":"2026-04-24T15:11:36Z","closed_at":"2026-04-24T15:22:53Z","close_reason":"Store package shipped: modernc.org/sqlite, embed.FS migrations, WAL + FK pragmas, idempotent reopen, 90.1% coverage including bad-SQL rollback and record-step PK conflict.","dependencies":[{"issue_id":"forgejo-mcp-broker-9jh","depends_on_id":"forgejo-mcp-broker-n84","type":"blocks","created_at":"2026-04-24T16:46:19Z","created_by":"Ole-Morten Duesund","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"id":"forgejo-mcp-broker-9nq","title":"Phase 1: internal/config package with flag + env parsing and validation","description":"Implement internal/config: a Config struct populated from CLI flags and environment variables (flags win), with validation at startup. Parse public-url, listen addr, forgejo-url, forgejo-oauth-client-id/secret/scopes, forgejo-mcp-binary, store-path, max-sessions, session-idle-timeout, debug. Validation: required fields present and non-empty; public-url parses as an https URL; store-path writable; idle-timeout positive; max-sessions positive. Unit tests cover happy path + every validation error branch.","acceptance_criteria":"go test ./internal/config passes with \u003e=90% coverage; missing required env produces a clear error message listing all missing fields; flag values override env; --help prints all config options.","status":"closed","priority":1,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T14:46:19Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-24T15:10:26Z","started_at":"2026-04-24T15:05:57Z","closed_at":"2026-04-24T15:10:26Z","close_reason":"Config package shipped with 94.1% coverage, handles all required-vs-optional fields, env/flag precedence, URL validation with loopback-http exception, and store-path writability. Full test suite green.","dependencies":[{"issue_id":"forgejo-mcp-broker-9nq","depends_on_id":"forgejo-mcp-broker-n84","type":"blocks","created_at":"2026-04-24T16:46:18Z","created_by":"Ole-Morten Duesund","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"id":"forgejo-mcp-broker-n84","title":"Phase 1: bootstrap Go project layout","description":"Set up the Go project skeleton so all subsequent phase 1 packages have somewhere to live. Initialize go.mod with module path kode.naiv.no/olemd/forgejo-mcp-broker, create the directory layout (cmd/broker, internal/config, internal/log, internal/store, internal/httpserver), add a Makefile with build/test/lint targets, and wire build-info injection (version, git revision, build date) via -ldflags.","acceptance_criteria":"go.mod present with correct module path; make build produces ./fjmcp-broker binary; make test and make lint targets exist and pass against an empty codebase; binary prints --version with injected build info.","status":"closed","priority":1,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T14:45:44Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-24T14:55:39Z","started_at":"2026-04-24T14:50:57Z","closed_at":"2026-04-24T14:55:39Z","close_reason":"Bootstrap complete: go.mod, Makefile, directory layout, ldflags-injected build info, --version flag all working. make build/test/lint pass.","dependency_count":0,"dependent_count":4,"comment_count":0} -{"id":"forgejo-mcp-broker-q6n","title":"Phase 7a: Claude.ai end-to-end validation on staging","description":"Stand up staging Forgejo + broker on a public hostname with real TLS. Register as a Claude.ai MCP connector. Walk through OAuth, tool discovery, tool invocation, session timeout, reconnect, Forgejo token refresh mid-session. Capture findings.","acceptance_criteria":"Claude.ai completes OAuth and lists tools. All forgejo-mcp tools invocable. 30-min idle session reconnects. Token refresh during active session invisible to user. docs/phase7-findings.md written.","status":"open","priority":2,"issue_type":"task","owner":"olemd@glemt.net","created_at":"2026-04-24T15:45:19Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-24T15:45:19Z","dependencies":[{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-8yd","type":"blocks","created_at":"2026-04-24T17:45:34Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-q4x","type":"blocks","created_at":"2026-04-24T17:45:34Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-r2c","type":"blocks","created_at":"2026-04-24T17:45:35Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-t81","type":"blocks","created_at":"2026-04-24T17:45:33Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-ytw","type":"blocks","created_at":"2026-04-24T17:45:33Z","created_by":"Ole-Morten Duesund","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} -{"id":"forgejo-mcp-broker-r2c","title":"Phase 6a: Caddy front-end example + docs/deploy-caddy.md","description":"deploy/caddy/Caddyfile example and docs/deploy-caddy.md: TLS via Let's Encrypt, flush_interval -1 for SSE, X-Forwarded-* config (issuer URL still from broker config, never inbound headers).","acceptance_criteria":"Caddyfile works against a local broker (curl --resolve test). docs/deploy-caddy.md linked from README and deploy-podman.md. Explains host-header attack defense.","status":"in_progress","priority":2,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T15:45:19Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-27T15:42:37Z","started_at":"2026-04-27T15:42:37Z","dependencies":[{"issue_id":"forgejo-mcp-broker-r2c","depends_on_id":"forgejo-mcp-broker-8yd","type":"blocks","created_at":"2026-04-24T17:45:32Z","created_by":"Ole-Morten Duesund","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"id":"forgejo-mcp-broker-q6n","title":"Phase 7a: Claude.ai end-to-end validation on staging","description":"Stand up staging Forgejo + broker on a public hostname with real TLS. Register as a Claude.ai MCP connector. Walk through OAuth, tool discovery, tool invocation, session timeout, reconnect, Forgejo token refresh mid-session. Capture findings.","acceptance_criteria":"Claude.ai completes OAuth and lists tools. All forgejo-mcp tools invocable. 30-min idle session reconnects. Token refresh during active session invisible to user. docs/phase7-findings.md written.","status":"in_progress","priority":2,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T15:45:19Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-27T15:50:05Z","started_at":"2026-04-27T15:50:05Z","dependencies":[{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-8yd","type":"blocks","created_at":"2026-04-24T17:45:34Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-q4x","type":"blocks","created_at":"2026-04-24T17:45:34Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-r2c","type":"blocks","created_at":"2026-04-24T17:45:35Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-t81","type":"blocks","created_at":"2026-04-24T17:45:33Z","created_by":"Ole-Morten Duesund","metadata":"{}"},{"issue_id":"forgejo-mcp-broker-q6n","depends_on_id":"forgejo-mcp-broker-ytw","type":"blocks","created_at":"2026-04-24T17:45:33Z","created_by":"Ole-Morten Duesund","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"id":"forgejo-mcp-broker-r2c","title":"Phase 6a: Caddy front-end example + docs/deploy-caddy.md","description":"deploy/caddy/Caddyfile example and docs/deploy-caddy.md: TLS via Let's Encrypt, flush_interval -1 for SSE, X-Forwarded-* config (issuer URL still from broker config, never inbound headers).","acceptance_criteria":"Caddyfile works against a local broker (curl --resolve test). docs/deploy-caddy.md linked from README and deploy-podman.md. Explains host-header attack defense.","status":"closed","priority":2,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T15:45:19Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-27T15:49:34Z","started_at":"2026-04-27T15:42:37Z","closed_at":"2026-04-27T15:49:34Z","close_reason":"deploy/caddy/Caddyfile + docs/deploy-caddy.md shipped. Caddyfile validated with caddy validate; flush_interval -1 documented as critical for SSE; host-header attack defense cross-referenced to existing regression test. Unblocks 7a (Claude.ai E2E).","dependencies":[{"issue_id":"forgejo-mcp-broker-r2c","depends_on_id":"forgejo-mcp-broker-8yd","type":"blocks","created_at":"2026-04-24T17:45:32Z","created_by":"Ole-Morten Duesund","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"id":"forgejo-mcp-broker-8yd","title":"Deploy via rootless podman with persistent state","description":"Operators will deploy fjmcp-broker via rootless podman (user's stated preference over docker). The broker needs: (1) a working Containerfile that bundles both fjmcp-broker and a pinned forgejo-mcp binary so the broker can spawn subprocesses; (2) deployment artefacts that use a named podman volume for /data so the SQLite store persists across container restarts and image updates; (3) documentation an operator can follow end-to-end. Container must run as non-root, carry OCI labels (org.opencontainers.image.created, .revision) and /etc/build-info per the user's global container standards. Note: full usefulness depends on OAuth (phase 2) being implemented, but writing the Containerfile + deploy docs early forces runtime concerns (volumes, env vars, non-root) to surface before they become harder to fix.","design":"Container: multi-stage build. Stage 1 compiles fjmcp-broker with -trimpath and ldflags for build info. Stage 2 fetches forgejo-mcp at a pinned version (pinning matters — the broker subprocess contract depends on flags/env). Final stage uses distroless/static or alpine-nonroot, copies both binaries, USER 65532, EXPOSE 8080, ENTRYPOINT ['/usr/local/bin/fjmcp-broker']. Volume strategy: /data is a podman named volume ('fjmcp-state') mounted with the right uid. SQLite WAL files live inside it and survive restarts. Env vars for secrets via --env-file pointing to a 0600-permissioned file outside the repo. Build args: BUILD_DATE and GIT_REVISION, wired into labels and /etc/build-info. Quadlet unit: use Podman Quadlet (not the deprecated podman generate systemd) — drop fjmcp-broker.container under ~/.config/containers/systemd/, daemon-reload, start.","acceptance_criteria":"make image builds locally under rootless podman; podman run -p 8080:8080 -v fjmcp-state:/data --env-file .env fjmcp-broker:latest starts the broker and /healthz returns 200 with store=ok; stopping the container and starting a new one against the same fjmcp-state volume preserves schema_migrations rows; pulling a newer image and recreating the container preserves state; a Quadlet .container unit example brings the broker up under systemd --user; docs/deploy-podman.md walks a fresh operator through build → run → persistent state → secret injection → image upgrade → troubleshooting.","notes":"Related to plan.md Phase 6 (packaging). If we build a compose example later for non-podman users, it can piggyback off the same Containerfile. Caddy sits in front of this container — include a brief note about Caddy+flush_interval requirements for SSE, referencing design.md §7.2.","status":"closed","priority":2,"issue_type":"task","assignee":"Ole-Morten Duesund","owner":"olemd@glemt.net","created_at":"2026-04-24T15:30:50Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-27T15:42:17Z","started_at":"2026-04-27T15:37:37Z","closed_at":"2026-04-27T15:42:17Z","close_reason":"Containerfile + Quadlet unit + docs/deploy-podman.md shipped. Image builds under rootless podman; verified end-to-end that fjmcp-state volume preserves broker.db + WAL + SHM across container swap and that /healthz returns store: ok on the new container. Unblocks 6a (Caddy front-end docs).","dependency_count":0,"dependent_count":2,"comment_count":0} {"id":"forgejo-mcp-broker-0sq","title":"Backlog: Forgejo Actions CI workflow","description":".forgejo/workflows/ci.yml running go vet, go test -race, go build, and golangci-lint on PRs and pushes to main. Block merge on failure.","acceptance_criteria":"CI green on main. PR failures block merge. Badge/status referenced in README.","status":"open","priority":3,"issue_type":"task","owner":"olemd@glemt.net","created_at":"2026-04-24T15:45:24Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-24T15:45:24Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"id":"forgejo-mcp-broker-ttl","title":"Backlog: rate limits on /oauth/register and /oauth/token","description":"design.md §8: Start with Caddy-level rate limits in the example Caddyfile (coarse but free). Add broker-side middleware only if operator experience shows Caddy is too coarse.","acceptance_criteria":"Caddy example extended with rate_limit directive. If broker-side added: unit tests cover quota exhaustion and reset windows.","status":"open","priority":3,"issue_type":"task","owner":"olemd@glemt.net","created_at":"2026-04-24T15:45:23Z","created_by":"Ole-Morten Duesund","updated_at":"2026-04-24T15:45:23Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/cmd/broker/e2e_test.go b/cmd/broker/e2e_test.go new file mode 100644 index 0000000..d826a1a --- /dev/null +++ b/cmd/broker/e2e_test.go @@ -0,0 +1,322 @@ +package main_test + +import ( + "crypto/sha256" + "encoding/base64" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/url" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "syscall" + "testing" + "time" +) + +// TestE2E_FullOAuthAndMCPFlow exercises every wired component end-to-end: +// +// 1. Build the broker binary (already done in TestMain). +// 2. Build forgejo-mcp from sibling source so the broker can spawn it. +// 3. Stand up a fake Forgejo (httptest.Server) covering the OAuth and +// OIDC endpoints plus the API surface forgejo-mcp probes at startup. +// 4. Run the broker with all that wired up. +// 5. Walk through: register → authorize → callback → token → +// /mcp (initialize via Bearer token) → tools/list. +// +// Skipped under -short and when the sibling forgejo-mcp source isn't +// present. This test is the closest local stand-in for the manual +// Claude.ai validation in docs/phase7-findings.md. +func TestE2E_FullOAuthAndMCPFlow(t *testing.T) { + if testing.Short() { + t.Skip("end-to-end test (~5s); rerun without -short") + } + + forgejoMCPBin := buildForgejoMCPSibling(t) + fakeForgejo := startFakeForgejo(t) + storePath := filepath.Join(t.TempDir(), "broker.db") + listenAddr := freePort(t) + publicURL := "http://" + listenAddr + + cmd := exec.Command(binPath, + "--public-url", publicURL, + "--forgejo-url", fakeForgejo.URL, + "--forgejo-oauth-client-id", "broker-app", + "--forgejo-oauth-client-secret", "broker-secret", + "--forgejo-mcp-binary", forgejoMCPBin, + "--listen", listenAddr, + "--store-path", storePath, + "--debug", + ) + cmd.Env = []string{"PATH=" + os.Getenv("PATH")} + stderr := &captureBuffer{} + cmd.Stderr = stderr + if err := cmd.Start(); err != nil { + t.Fatalf("start broker: %v", err) + } + defer func() { + if cmd.Process != nil { + _ = cmd.Process.Signal(syscall.SIGTERM) + _, _ = cmd.Process.Wait() + } + }() + waitListening(t, listenAddr, 5*time.Second) + + // Step 1 — register a client. + clientID := registerClient(t, publicURL, "https://app.example.com/cb") + + // Step 2 — authorize. We use a no-redirect HTTP client so we can + // inspect the redirect Location and pick out the Forgejo state. + verifier := "a-pkce-verifier-thats-quite-long-12345678" + challenge := pkceChallenge(verifier) + authQ := url.Values{ + "response_type": {"code"}, + "client_id": {clientID}, + "redirect_uri": {"https://app.example.com/cb"}, + "state": {"client-csrf"}, + "code_challenge": {challenge}, + "code_challenge_method": {"S256"}, + } + resp := getNoRedirect(t, publicURL+"/oauth/authorize?"+authQ.Encode()) + resp.Body.Close() + if resp.StatusCode != http.StatusFound { + t.Fatalf("authorize: status = %d, want 302", resp.StatusCode) + } + upstream, err := url.Parse(resp.Header.Get("Location")) + if err != nil { + t.Fatalf("parse Location: %v", err) + } + forgejoState := upstream.Query().Get("state") + if forgejoState == "" { + t.Fatalf("authorize did not produce a Forgejo state: %s", upstream) + } + + // Step 3 — fake Forgejo would have authenticated the user and + // redirected to /oauth/callback. Simulate that. + cb := getNoRedirect(t, publicURL+"/oauth/callback?code=upstream-code&state="+forgejoState) + cb.Body.Close() + if cb.StatusCode != http.StatusFound { + t.Fatalf("callback: status = %d, want 302", cb.StatusCode) + } + cbLoc, _ := url.Parse(cb.Header.Get("Location")) + brokerCode := cbLoc.Query().Get("code") + if brokerCode == "" { + t.Fatalf("callback did not return broker code: %s", cbLoc) + } + + // Step 4 — exchange broker code for access + refresh tokens. + tokForm := url.Values{ + "grant_type": {"authorization_code"}, + "code": {brokerCode}, + "client_id": {clientID}, + "redirect_uri": {"https://app.example.com/cb"}, + "code_verifier": {verifier}, + } + tokResp, err := http.PostForm(publicURL+"/oauth/token", tokForm) + if err != nil { + t.Fatalf("token: %v", err) + } + defer tokResp.Body.Close() + if tokResp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(tokResp.Body) + t.Fatalf("token status = %d: %s", tokResp.StatusCode, body) + } + var tokens struct { + AccessToken string `json:"access_token"` + } + if err := json.NewDecoder(tokResp.Body).Decode(&tokens); err != nil { + t.Fatalf("decode token: %v", err) + } + if tokens.AccessToken == "" { + t.Fatalf("token response missing access_token") + } + + // Step 5 — call /mcp with the bearer token. First the MCP `initialize` + // handshake; the broker spawns a forgejo-mcp child for this session. + initBody := `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"e2e","version":"1"}}}` + initReq, _ := http.NewRequest(http.MethodPost, publicURL+"/mcp", strings.NewReader(initBody)) + initReq.Header.Set("Authorization", "Bearer "+tokens.AccessToken) + initReq.Header.Set("Content-Type", "application/json") + initResp, err := http.DefaultClient.Do(initReq) + if err != nil { + t.Fatalf("initialize: %v", err) + } + defer initResp.Body.Close() + if initResp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(initResp.Body) + t.Fatalf("initialize status = %d: %s\n\nbroker stderr:\n%s", initResp.StatusCode, body, stderr.String()) + } + sid := initResp.Header.Get("Mcp-Session-Id") + if sid == "" { + t.Fatalf("initialize did not return Mcp-Session-Id") + } + body, _ := io.ReadAll(initResp.Body) + if !strings.Contains(string(body), `"protocolVersion"`) { + t.Errorf("initialize response missing protocolVersion: %s", body) + } + + // Step 6 — tools/list with the same sid. Different RPC, same forgejo-mcp child. + listBody := `{"jsonrpc":"2.0","id":2,"method":"tools/list"}` + listReq, _ := http.NewRequest(http.MethodPost, publicURL+"/mcp", strings.NewReader(listBody)) + listReq.Header.Set("Authorization", "Bearer "+tokens.AccessToken) + listReq.Header.Set("Content-Type", "application/json") + listReq.Header.Set("Mcp-Session-Id", sid) + listResp, err := http.DefaultClient.Do(listReq) + if err != nil { + t.Fatalf("tools/list: %v", err) + } + defer listResp.Body.Close() + listBytes, _ := io.ReadAll(listResp.Body) + if listResp.StatusCode != http.StatusOK { + t.Fatalf("tools/list status = %d: %s\n\nbroker stderr:\n%s", + listResp.StatusCode, listBytes, stderr.String()) + } + if !strings.Contains(string(listBytes), "get_forgejo_mcp_server_version") { + t.Errorf("tools/list missing expected tool. Body: %s", listBytes) + } + + // Step 7 — discovery surface returns issuer-rooted URLs derived from + // --public-url, not from the test server's address. + disc, err := http.Get(publicURL + "/.well-known/oauth-authorization-server") + if err != nil { + t.Fatalf("discovery: %v", err) + } + defer disc.Body.Close() + var md map[string]any + if err := json.NewDecoder(disc.Body).Decode(&md); err != nil { + t.Fatalf("decode discovery: %v", err) + } + if md["issuer"] != publicURL { + t.Errorf("issuer = %v, want %s", md["issuer"], publicURL) + } +} + +// captureBuffer is a thread-safe buffer for collecting child stderr. +type captureBuffer struct { + mu sync.Mutex + buf strings.Builder +} + +func (c *captureBuffer) Write(p []byte) (int, error) { + c.mu.Lock() + defer c.mu.Unlock() + return c.buf.Write(p) +} + +func (c *captureBuffer) String() string { + c.mu.Lock() + defer c.mu.Unlock() + return c.buf.String() +} + +func registerClient(t *testing.T, baseURL, redirectURI string) string { + t.Helper() + body, _ := json.Marshal(map[string]any{ + "redirect_uris": []string{redirectURI}, + "client_name": "e2e-test", + }) + resp, err := http.Post(baseURL+"/oauth/register", "application/json", strings.NewReader(string(body))) + if err != nil { + t.Fatalf("register: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusCreated { + b, _ := io.ReadAll(resp.Body) + t.Fatalf("register status = %d: %s", resp.StatusCode, b) + } + var r struct { + ClientID string `json:"client_id"` + } + if err := json.NewDecoder(resp.Body).Decode(&r); err != nil { + t.Fatalf("decode register: %v", err) + } + return r.ClientID +} + +func pkceChallenge(verifier string) string { + sum := sha256.Sum256([]byte(verifier)) + return base64.RawURLEncoding.EncodeToString(sum[:]) +} + +var noRedirectClient = &http.Client{ + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, +} + +func getNoRedirect(t *testing.T, url string) *http.Response { + t.Helper() + resp, err := noRedirectClient.Get(url) + if err != nil { + t.Fatalf("get %s: %v", url, err) + } + return resp +} + +// startFakeForgejo stands up the Forgejo API surface we need: SDK +// version probe, OIDC userinfo, OAuth token endpoint, plus an /api/v1/user +// for forgejo-mcp's own startup probe. +func startFakeForgejo(t *testing.T) *httptest.Server { + t.Helper() + mux := http.NewServeMux() + mux.HandleFunc("/api/v1/version", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"version":"11.0.0"}`) + }) + mux.HandleFunc("/api/v1/user", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"id":1,"login":"e2e-user","username":"e2e-user","full_name":"E2E"}`) + }) + mux.HandleFunc("/login/oauth/access_token", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, + `{"access_token":"fj-access-token","refresh_token":"fj-refresh-token","token_type":"bearer","expires_in":3600}`) + }) + mux.HandleFunc("/login/oauth/userinfo", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"sub":"42","preferred_username":"e2e-user","name":"E2E User"}`) + }) + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + t.Logf("fake forgejo: unexpected probe %s %s", r.Method, r.URL.Path) + w.WriteHeader(http.StatusNotFound) + }) + srv := httptest.NewServer(mux) + t.Cleanup(srv.Close) + return srv +} + +// buildForgejoMCPSibling locates / builds the forgejo-mcp binary the +// broker will spawn. Same search order as internal/bridge's integration +// test. +func buildForgejoMCPSibling(t *testing.T) string { + t.Helper() + if p := os.Getenv("FORGEJO_MCP_BIN"); p != "" { + if _, err := os.Stat(p); err == nil { + return p + } + t.Skipf("FORGEJO_MCP_BIN=%q does not exist", p) + } + if abs, err := filepath.Abs("../../../forgejo-mcp/forgejo-mcp"); err == nil { + if _, err := os.Stat(abs); err == nil { + return abs + } + } + if abs, err := filepath.Abs("../../../forgejo-mcp"); err == nil { + if _, err := os.Stat(filepath.Join(abs, "main.go")); err == nil { + bin := filepath.Join(t.TempDir(), "forgejo-mcp") + build := exec.Command("go", "build", "-o", bin, ".") + build.Dir = abs + build.Env = os.Environ() + out, err := build.CombinedOutput() + if err != nil { + t.Skipf("go build of sibling forgejo-mcp failed: %v\n%s", err, out) + } + return bin + } + } + t.Skip("forgejo-mcp binary not found: set $FORGEJO_MCP_BIN or place a sibling repo at ../forgejo-mcp") + return "" +} + diff --git a/cmd/broker/main.go b/cmd/broker/main.go index 5f3a76e..8adec57 100644 --- a/cmd/broker/main.go +++ b/cmd/broker/main.go @@ -5,19 +5,28 @@ package main import ( "context" + "database/sql" "errors" "flag" "fmt" "io" + "log/slog" + "net/http" "os" "os/signal" "syscall" + "time" + "kode.naiv.no/olemd/forgejo-mcp-broker/internal/bridge" "kode.naiv.no/olemd/forgejo-mcp-broker/internal/buildinfo" "kode.naiv.no/olemd/forgejo-mcp-broker/internal/config" + "kode.naiv.no/olemd/forgejo-mcp-broker/internal/forgejo" "kode.naiv.no/olemd/forgejo-mcp-broker/internal/httpserver" brokerlog "kode.naiv.no/olemd/forgejo-mcp-broker/internal/log" + "kode.naiv.no/olemd/forgejo-mcp-broker/internal/oauth" + "kode.naiv.no/olemd/forgejo-mcp-broker/internal/session" "kode.naiv.no/olemd/forgejo-mcp-broker/internal/store" + "kode.naiv.no/olemd/forgejo-mcp-broker/internal/supervisor" ) // Exit codes follow the usual convention: 0 success, 2 config/usage, 1 runtime. @@ -82,10 +91,31 @@ func run(args []string, out io.Writer) int { } }() + // childCtx outlives any one HTTP request — it's the broker-process- + // scoped context that supervisor.Start will associate with each + // spawned forgejo-mcp. Canceling it on shutdown is what tears the + // whole subprocess tree down. (Using a request context here would + // kill children when their /mcp request returns — exec.CommandContext + // ties process lifetime to the ctx.) + childCtx, cancelChildren := context.WithCancel(context.Background()) + defer cancelChildren() + + // Build the OAuth + MCP-session stack as a single http.Handler that we + // pass to the http server as ExtraHandler. /healthz keeps living in + // httpserver itself. + mux, sessReg, stopReaper, err := buildHandlers(cfg, logger, st, childCtx) + if err != nil { + logger.Error("wire handlers", "err", err.Error()) + return exitRuntime + } + defer stopReaper() + defer sessReg.Stop(context.Background()) + srv := &httpserver.Server{ - Addr: cfg.Listen, - Log: logger, - Store: st, + Addr: cfg.Listen, + Log: logger, + Store: st, + ExtraHandler: mux, } if err := srv.Run(ctx); err != nil { logger.Error("server exit", "err", err.Error()) @@ -94,3 +124,144 @@ func run(args []string, out io.Writer) int { logger.Info("broker stopped") return exitSuccess } + +// buildHandlers assembles the OAuth server, the bearer-gated /mcp endpoint, +// the session registry, and the reaper into one http.Handler. The returned +// stopReaper must be called at shutdown. childCtx is a long-lived context +// used to spawn forgejo-mcp subprocesses — it must outlive any single +// /mcp request, otherwise exec.CommandContext kills the child on +// request completion. +func buildHandlers( + cfg *config.Config, + logger *slog.Logger, + st *store.Store, + childCtx context.Context, +) (http.Handler, *session.Registry, func(), error) { + fjClient, err := forgejo.NewClient(forgejo.ClientConfig{ + BaseURL: cfg.ForgejoURL, + ClientID: cfg.ForgejoOAuthClientID, + ClientSecret: cfg.ForgejoOAuthClientSecret, + UserAgent: "fjmcp-broker/" + buildinfo.Version, + }) + if err != nil { + return nil, nil, nil, fmt.Errorf("forgejo client: %w", err) + } + + oauthSrv, err := oauth.NewServer(oauth.Config{ + Store: st, + Forgejo: fjClient, + Issuer: cfg.PublicURL, + Scopes: cfg.ForgejoOAuthScopes, + Log: logger.With("component", "oauth"), + }) + if err != nil { + return nil, nil, nil, fmt.Errorf("oauth server: %w", err) + } + + auth := &oauth.Authenticator{Store: st} + + spawn := func(_ context.Context, sess *oauth.Session) (*session.Backend, error) { + // Deliberately ignore the per-request context the registry hands + // us; pass childCtx so the spawned forgejo-mcp survives once the + // HTTP request that triggered the spawn returns. + return spawnForgejoMCP(childCtx, cfg, logger, sess) + } + + sessReg, err := session.New(session.Config{ + Spawn: spawn, + MaxSessions: cfg.MaxSessions, + Log: logger.With("component", "session"), + }) + if err != nil { + return nil, nil, nil, fmt.Errorf("session registry: %w", err) + } + + stopReaper := sessReg.StartReaper(session.ReaperConfig{ + IdleTimeout: cfg.SessionIdleTimeout, + RefreshForgejo: refreshFunc(fjClient, st, logger), + Respawn: spawn, + }) + + mux := http.NewServeMux() + // Mount OAuth + discovery at the root: /oauth/*, /.well-known/*. + // oauth.Server.Handler() declares method-routed patterns, so it + // composes safely with the gated /mcp below. + oauthHandler := oauthSrv.Handler() + mux.Handle("/oauth/", oauthHandler) + mux.Handle("/.well-known/", oauthHandler) + // Gated MCP endpoint. + mux.Handle("POST /mcp", auth.RequireBearer(sessReg.Handler())) + mux.Handle("GET /mcp", auth.RequireBearer(sessReg.Handler())) + + return mux, sessReg, stopReaper, nil +} + +// spawnForgejoMCP constructs a session.Backend by launching forgejo-mcp +// under the supervisor and wrapping its stdio with a bridge. The user's +// upstream Forgejo token is injected via FORGEJO_ACCESS_TOKEN — that's +// what makes per-session token isolation work. +func spawnForgejoMCP( + ctx context.Context, + cfg *config.Config, + logger *slog.Logger, + sess *oauth.Session, +) (*session.Backend, error) { + childLog := logger.With( + "component", "forgejo-mcp", + "user", sess.ForgejoUsername, + ) + child, err := supervisor.Start(ctx, supervisor.Config{ + Cmd: []string{cfg.ForgejoMCPBinary, "--transport", "stdio", "--url", cfg.ForgejoURL}, + Env: []string{ + "FORGEJO_ACCESS_TOKEN=" + sess.ForgejoToken, + "FORGEJO_USER_AGENT=fjmcp-broker/" + buildinfo.Version, + }, + OnStderr: func(line string) { + childLog.Debug("stderr", "line", line) + }, + }) + if err != nil { + return nil, err + } + + br := bridge.New(child.Stdin, child.Stdout, child.Done(), childLog) + br.Start() + + return &session.Backend{ + Handler: http.HandlerFunc(br.HandleSSE), + Stop: child.Stop, + Done: child.Done(), + }, nil +} + +// refreshFunc adapts the Forgejo OAuth client to the +// session.ReaperConfig.RefreshForgejo signature: refresh upstream tokens +// AND persist the new tokens back to the access_tokens row keyed by the +// session's broker-token hash. +func refreshFunc( + fj *forgejo.Client, + st *store.Store, + logger *slog.Logger, +) func(context.Context, *oauth.Session) (string, string, time.Time, error) { + return func(ctx context.Context, sess *oauth.Session) (string, string, time.Time, error) { + tok, err := fj.Refresh(ctx, sess.ForgejoRefresh) + if err != nil { + return "", "", time.Time{}, fmt.Errorf("forgejo refresh: %w", err) + } + expiresAt := time.Now().Add(time.Duration(tok.ExpiresIn) * time.Second) + _, dbErr := st.DB().ExecContext(ctx, + `UPDATE access_tokens + SET forgejo_access_token = ?, + forgejo_refresh_token = ?, + forgejo_token_expires_at = ? + WHERE token_hash = ? AND revoked_at IS NULL`, + tok.AccessToken, tok.RefreshToken, expiresAt.Unix(), sess.BrokerTokenHash) + if dbErr != nil && !errors.Is(dbErr, sql.ErrNoRows) { + logger.Warn("persist refreshed forgejo token", "err", dbErr.Error()) + // Persist failure is non-fatal — the rotator can still + // hand the new token to the respawned child; the next + // rotation cycle will retry the persist. + } + return tok.AccessToken, tok.RefreshToken, expiresAt, nil + } +} diff --git a/docs/phase7-findings.md b/docs/phase7-findings.md new file mode 100644 index 0000000..6b496e2 --- /dev/null +++ b/docs/phase7-findings.md @@ -0,0 +1,168 @@ +# Phase 7: end-to-end findings + +This document covers the validation that closes phase 7 (issue `-q6n`). +The work splits naturally into two halves: + +1. **Local end-to-end** — automated, runs against a fake Forgejo and a + real forgejo-mcp child spawned through the live broker binary. + Performed in this repo as `TestE2E_FullOAuthAndMCPFlow` under + `cmd/broker/e2e_test.go`. + +2. **Public Claude.ai validation** — manual, requires a real Forgejo + instance and a publicly-reachable broker. Documented below as a + step-by-step recipe so the operator can repeat it cleanly. + +The local automated half was the gating prerequisite: without it, every +manual step would hide both setup mistakes and broker bugs. With it +green, the manual run becomes mostly a configuration exercise. + +## What the wiring shipped + +`cmd/broker/main.go` now composes every component built across phases +2–5 into a single binary: + +``` +http.ServeMux +├── /healthz ← internal/httpserver +├── /oauth/* ← internal/oauth.Server.Handler() +├── /.well-known/* ← internal/oauth.Server.Handler() +└── /mcp ← internal/oauth.Authenticator.RequireBearer + (gates internal/session.Registry.Handler()) +``` + +The `SpawnFunc` passed to the session registry composes +`internal/supervisor` + `internal/bridge`: each new MCP session forks a +`forgejo-mcp --transport stdio` subprocess with the user's upstream +Forgejo token in env, wires its stdio through a bridge, and returns the +bridge's `HandleSSE` as the per-session HTTP handler. + +The reaper is wired with a `RefreshForgejo` callback that calls the +upstream Forgejo OAuth client and persists rotated tokens back to the +`access_tokens` row. When a token comes within the rotation lead time of +expiry, the rotator atomically swaps the session's child for one +spawned with the fresh token; the sid is preserved and the client +re-issues `initialize` on its next request. + +## Local end-to-end test + +```bash +go test -v -run TestE2E_FullOAuthAndMCPFlow ./cmd/broker/ +``` + +The test: + +1. Builds the broker binary in TestMain. +2. Builds `forgejo-mcp` from the sibling repo at `../../../forgejo-mcp` + (skipped if not present; alternatively set `FORGEJO_MCP_BIN`). +3. Stands up a fake Forgejo on `httptest.Server` covering + `/api/v1/version`, `/api/v1/user`, `/login/oauth/access_token`, and + `/login/oauth/userinfo`. +4. Runs the broker with all CLI flags wired against the fake. +5. Walks through register → authorize → callback → token → /mcp + initialize → tools/list. +6. Verifies that discovery (`/.well-known/oauth-authorization-server`) + advertises the configured issuer. + +It catches at least three classes of regression: + - **Wiring**: any component not plumbed correctly into main.go fails + a step rather than reporting "not implemented". + - **Subprocess-context**: an early version of the wiring used the + per-request HTTP context for `supervisor.Start`. Because + `exec.CommandContext` ties process lifetime to the ctx, this killed + the forgejo-mcp child as soon as the `initialize` request returned; + the subsequent `tools/list` came back as 502. Fixed by introducing + a long-lived `childCtx` in `run()`. + - **Session lifecycle**: 410 / 403 / 503 paths are exercised + in unit tests but the happy-path 200 with `Mcp-Session-Id` minted on + initialize, then reused on follow-up RPCs, is exercised here. + +## Public Claude.ai validation (manual) + +This is the part the operator runs once. It needs: + + - a publicly-reachable hostname (`mcp.example.com`) with a real TLS + cert (Caddy's automatic Let's Encrypt is sufficient — see + `deploy-caddy.md`); + - a real Forgejo instance you control; + - a Claude.ai workspace where you can register a custom MCP connector. + +### Setup + +1. Provision the broker container per `deploy-podman.md`. Confirm + `https://mcp.example.com/healthz` returns `{"status":"ok"}`. +2. Confirm discovery surface: + + ```bash + curl -fsS https://mcp.example.com/.well-known/oauth-authorization-server | jq .issuer + # → "https://mcp.example.com" + curl -fsS https://mcp.example.com/.well-known/oauth-protected-resource | jq .resource + # → "https://mcp.example.com/mcp" + ``` + +3. In Claude.ai, add a custom MCP connector pointing at + `https://mcp.example.com/mcp` (or whatever the Claude.ai UI asks for + — at the time of writing, Claude.ai detects the OAuth requirement + from the protected-resource discovery doc and walks the user + through the OAuth flow automatically). + +### Walkthrough checklist + +Run through these in order. Each is a check the operator marks done. + +- [ ] **OAuth completes.** Claude.ai redirects to the broker's + `/oauth/authorize`; the broker bounces to the Forgejo consent screen; + Forgejo redirects back to `/oauth/callback`; Claude.ai shows the + connector as authenticated. Expect the round trip to take a few + seconds. + +- [ ] **Tool discovery.** Claude.ai's UI should list every tool the + bundled `forgejo-mcp` exports — check at least + `get_forgejo_mcp_server_version`, `list_my_repos`, `list_repo_issues` + appear. + +- [ ] **Tool invocation.** Ask Claude something that exercises a tool + (e.g. "list issues open on ``"). Verify the tool runs and + the response cites real data from your Forgejo instance. + +- [ ] **Session reuse.** Run two queries in a row. The broker should + send one initialize per session, but successive calls should reuse + the same forgejo-mcp child — visible in the broker's logs as no new + "session reaped" / spawn entries between calls. + +- [ ] **Idle timeout.** Leave the conversation idle for 16 minutes + (idle timeout default is 15m + 30s reaper tick). Check broker logs + for `"session reaped"` matching your sid. The next query should + trigger a fresh spawn. + +- [ ] **Forgejo token refresh during active session.** Lower + `--session-idle-timeout` to something tolerable (say 1 h) and use a + Forgejo token TTL shorter than that. Run the connector continuously + until the rotator fires (`"session rotated"` in logs). Confirm the + next query still works — Claude.ai re-issues `initialize` on the + swapped child without user intervention. + +- [ ] **Revocation.** Hit `POST /oauth/revoke` with the access token. + The next /mcp request from Claude.ai should fail with 401; Claude.ai + should automatically re-authenticate. + +### Deliberately deferred + +Items not checked in this validation, tracked in the backlog: + +- **Rate limits** on `/oauth/register` and `/oauth/token` (`-ttl`). +- **`/proc//environ` token exposure** via the Forgejo token in + child env (`-1n2`). +- **At-rest encryption of Forgejo tokens** (`-sd4`). +- **Prometheus `/metrics`** for production observability (`-7fn`). +- **Forgejo Actions CI** (`-0sq`). + +All of these are P3 hardenings, not blockers for the Claude.ai +integration itself. + +## Closing notes + +Every phase-1-through-6 component is now exercised end-to-end against a +real `forgejo-mcp` child. The only wiring that can't be tested without +a real Claude.ai workspace is the actual MCP-discovery → OAuth-handoff +sequence in Claude.ai's client UI; that's left as a manual one-time +operator task using the checklist above.