Feat: Embedded HTTP API Server

[antweiss]

Feature Requirements

FR1: Embedded HTTP API Server

Objective: Embed an HTTP server within the khook controller to expose event data via REST API

Technical Requirements:

• Serve HTTP API on configurable port (default: 8082)
• Support graceful startup/shutdown with controller lifecycle
• Implement request routing, middleware, and error handling
• Add authentication and authorization mechanisms
• Include request logging and metrics collection

FR2: Event Query Endpoints

Objective: Provide REST endpoints for querying collected events with filtering capabilities

API Specification:

GET /api/v1/events
GET /api/v1/events/{namespace}
GET /api/v1/hooks/{namespace}/{name}/events
GET /api/v1/events/types/{eventType}

Query Parameters:

• startTime & endTime: Time-based filtering (RFC3339 format)
• eventType: Filter by event type (pod-restart, oom-kill, etc.)
• resourceName: Filter by Kubernetes resource name
• status: Filter by event status (firing, resolved, duplicate)
• limit & offset: Pagination support
• sort: Sort by timestamp, eventType, resourceName

FR3: Event Statistics and Aggregation

Objective: Provide aggregated event statistics and metrics via API

Endpoints:

GET /api/v1/stats/events/summary
GET /api/v1/stats/events/by-type
GET /api/v1/stats/hooks/{namespace}/{name}/metrics
GET /api/v1/stats/trends

Response Format:

{
  "summary": {
    "totalEvents": 1247,
    "activeEvents": 23,
    "duplicateEvents": 156,
    "timeRange": "24h"
  },
  "byType": {
    "pod-restart": {"count": 856, "percentage": 68.7},
    "oom-kill": {"count": 234, "percentage": 18.8},
    "pod-pending": {"count": 157, "percentage": 12.5}
  }
}

FR4: Real-time Event Streaming

Objective: Support real-time event streaming via WebSockets or Server-Sent Events

Technical Features:

• WebSocket endpoint: ws://khook:8082/api/v1/events/stream
• Server-Sent Events: GET /api/v1/events/stream with Accept: text/event-stream
• Configurable event filtering for streams
• Connection management and automatic reconnection
• Rate limiting and backpressure handling

FR5: Hook Configuration Management

Objective: Provide API endpoints for managing Hook configurations

Endpoints:

GET /api/v1/hooks
GET /api/v1/hooks/{namespace}/{name}
POST /api/v1/hooks/validate

Features:

• List all Hook configurations with status
• Validate Hook configurations before deployment
• Export Hook configurations in YAML format
• Display Hook processing statistics

FR6: Health and Diagnostics

Objective: Provide comprehensive health monitoring and diagnostic information

Endpoints:

GET /api/v1/health
GET /api/v1/diagnostics
GET /api/v1/metrics

Health Information:

• API server status and uptime
• Event processing pipeline health
• Kagent API connectivity status
• Plugin status (for pluggable event sources)
• Memory usage and performance metrics

[DmarshalTU]

@antweiss PRed my custom version

[DmarshalTU]

my internal usage

[EItanya]

This is a great idea, I do have a few thoughts/learnings I would like to take into account from the work we've already done on the kagent apiserver.

1. This server should be spec first. What I mean by that is that the swagger spec should be the source of truth, and then we should use an openapi golang codegen library to generate the client bindings. We can also go the proto route, but that's a whole can of worms.
2. Generally we should try and share as much code as possible with kagent. Specifically when it comes to middleware. If any of that code is not public today, we can make that change.

[DmarshalTU]

This is a great idea, I do have a few thoughts/learnings I would like to take into account from the work we've already done on the kagent apiserver.

1. This server should be spec first. What I mean by that is that the swagger spec should be the source of truth, and then we should use an openapi golang codegen library to generate the client bindings. We can also go the proto route, but that's a whole can of worms.
2. Generally we should try and share as much code as possible with kagent. Specifically when it comes to middleware. If any of that code is not public today, we can make that change.

I totally forgot that no one aware of our conversation with @antweiss and needed to mention in the PR that is code share of my custom setup and we agreed to look into and adjust it to Kagent guideline(like naming that related to me and not suitable for khook),
also @antweiss forgot to mention the two mode architecture for khook, I made it based on Redis but as you mentioned that Kagent have sql we can use it.

[EItanya]

So I definitely see why you went with this design, but I think there's a core thing here that we need to figure out. Right now you have a centralized khook controller, but that's really counter to the current design system of kagent generally. The kagent design language and system is specifically single cluster oriented. The APIs can only reference things in other Namespaces, not other clusters for examples. I understand at a high level why a design like this seems appealing, but I don't think it would work currently. For example, what if you want to turn off event listeners in a single cluster, how would you do that? I think instead we should have the khook controller/listener in the edge clusters, and then a separate data pipeline to get the data up. What do you think?

[antweiss]

Folks, I will join the conversation tomorrow. Shana tova to you both! 🍎 Anton Weiss http://otomato.io

…

On Mon, Sep 22, 2025, 7:08 PM Eitan Yarmush ***@***.***> wrote: *EItanya* left a comment (kagent-dev/khook#7) <#7 (comment)> So I definitely see why you went with this design, but I think there's a core thing here that we need to figure out. Right now you have a centralized khook controller, but that's really counter to the current design system of kagent generally. The kagent design language and system is specifically single cluster oriented. The APIs can only reference things in other Namespaces, not other clusters for examples. I understand at a high level why a design like this seems appealing, but I don't think it would work currently. For example, what if you want to turn off event listeners in a single cluster, how would you do that? I think instead we should have the khook controller/listener in the edge clusters, and then a separate data pipeline to get the data up. What do you think? — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAY7P3QWGBRE55VDRD33KKD3UANGVAVCNFSM6AAAAACG6AYRJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJZHE2DGMJVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DmarshalTU]

So I definitely see why you went with this design, but I think there's a core thing here that we need to figure out. Right now you have a centralized khook controller, but that's really counter to the current design system of kagent generally. The kagent design language and system is specifically single cluster oriented. The APIs can only reference things in other Namespaces, not other clusters for examples. I understand at a high level why a design like this seems appealing, but I don't think it would work currently. For example, what if you want to turn off event listeners in a single cluster, how would you do that? I think instead we should have the khook controller/listener in the edge clusters, and then a separate data pipeline to get the data up. What do you think?

What generally I got in mind that implement same technique as with metrics and logs, scrape all data to one place with labels and then based on khook use that,

Why?
In my case I already have tsdb and can write into or use redis to cache this data, secondly the collector mode will scrape only events that I label for him

I see your point that now khook is integrated part of kagent and must follow, but if I want only scrape events from edge cluster why I need to deploy full controller setup? Same for kagent, I don't need deploy it to consume remote mcp, so I manage my agents in one cluster, same for hooks.

Maybe I got to nuclear about only my needs and ignore the whole picture, we should to talk about it in ds instead of comments here.

[antweiss]

@DmarshalTU can you update on the status? Are you still planning to work on this or have you given up on it after all the conversations we had?