GUACAMOLE-1415: RDP Camera Redirection (RDPECAM) - Server Implementation

[tanonl]

Note: This PR works in conjunction with the corresponding guacamole-client PR for complete camera redirection functionality.

Overview

This PR implements server-side support for RDP Camera Redirection using the MS-RDPECAM (RDP Enhanced Camera) protocol. This enables Apache Guacamole to forward H.264 video streams from web clients to remote Windows sessions via the RDPECAM dynamic virtual channel.

---

Problem Statement

Remote desktop users need the ability to use their local cameras in remote Windows sessions for video conferencing, biometric authentication, or other applications. This server-side implementation provides the protocol handling and stream management necessary to forward camera data from Guacamole clients to RDP sessions using Microsoft's RDPECAM specification.

---

Technical Implementation

1. Core Video Infrastructure (5881e9ee)

Added generic video handler infrastructure to Guacamole's core library, enabling protocols to receive video streams from clients:

Changes:

• New video_handler field in guac_user structure (src/libguac/guacamole/user.h)
• Video instruction handler in user-handlers.c/h
• Video handshake support

Design Philosophy:

• Protocol-agnostic architecture (not specific to RDPECAM)
• Mirrors existing audio handler design pattern
• Reusable by other protocols (VNC, SSH, etc.)

/**
 * Handler for Guacamole video streams.
 */
guac_user_video_handler* video_handler;

2. RDPECAM Channel Implementation (4b59c528)

Implemented the MS-RDPECAM protocol specification with three core components in src/protocols/rdp/channels/rdpecam/:

rdpecam.c/h - Channel Management

• FreeRDP dynamic virtual channel lifecycle
• Virtual camera device state machine
• Device activation/deactivation control
• Stream credit management (flow control)
• Channel cleanup and resource management

rdpecam_caps.c/h - Capability Negotiation

• H.264 format capability parsing
• Media type negotiation
• Frame rate and resolution capability handling
• Device name sanitization (Unicode → ASCII, max 127 chars)
• Capability structure serialization/deserialization

Key Features:

• Automatic format selection from device capabilities
• Safe string handling for device names
• Capability matching between client and server

rdpecam_sink.c/h - Video Frame Queue

• Thread-safe frame buffering (10-frame queue)
• H.264 frame ingestion from Guacamole streams
• Frame forwarding to RDPECAM channel
• Queue overflow handling (drop oldest frames)
• Frame metadata management (timestamps, keyframes)

3. RDPECAM Plugin (4ce6bf0b)

FreeRDP plugin bridge in src/protocols/rdp/plugins/guacrdpecam/:

guacrdpecam.c/h - Plugin Core

The centerpiece of the implementation, handling:

• FreeRDP plugin registration and lifecycle
• Virtual camera device management
• Dequeue thread for video frame processing
• State synchronization between Guacamole and FreeRDP
• Credit-based flow control (RDP protocol requirement)
• Sample submission to RDPECAM channel
• Error handling and recovery

Architecture:

Guacamole Stream → Frame Queue → Dequeue Thread → RDPECAM Channel → RDP Session

Key Components:

• Device state tracking (inactive, active, streaming)
• Thread-safe device map (virtual device ID → device context)
• Credit management for flow control
• Sample formatting (MediaSampleHeader + H.264 data)

rdpecam_proto.c/h (781 lines) - Protocol Messages

Low-level MS-RDPECAM message handling:

Message Types Implemented:

• ACTIVATE_DEVICE_REQUEST/RESPONSE
• DEACTIVATE_DEVICE_REQUEST
• START_STREAMS_REQUEST/RESPONSE
• STOP_STREAMS_REQUEST/RESPONSE
• STREAM_LIST_REQUEST/RESPONSE
• PROPERTY_LIST_REQUEST/RESPONSE
• STREAM_SAMPLE (video frame transmission)
• SUCCESS_RESPONSE / ERROR_RESPONSE

Functions:

• Message builders (serialize structs → byte streams)
• Message parsers (byte streams → structs)
• Endianness handling
• GUID and timestamp formatting

4. RDP Protocol Integration (63cca7bf, cb656338)

Wired RDPECAM into the RDP protocol handler:

rdp.c modifications:

• Load RDPECAM plugin when enabled
• Initialize device mapping structures

settings.c/h modifications:

• New parameter: enable-rdpecam (boolean, default: disabled)
• Configuration parsing and validation

user.c modifications:

• Register video handler for RDP connections
• Route video streams to RDPECAM sink

client.c modifications:

• Initialize RDPECAM device map
• Cleanup on disconnect

Build system (configure.ac, Makefile.am):

• Conditional compilation support
• Link RDPECAM sources into RDP protocol library
• Test suite integration

5. Dynamic Enable/Disable Support (72de7796)

Camera can be toggled during active sessions:

• Start streaming: Client initiates stream, server activates device
• Stop streaming: Server deactivates device, preserves connection
• No reconnection required for camera on/off

6. Thread Efficiency Optimization (a00eb82c)

Replaced polling-based dequeue thread with condition variables:

Before: usleep() polling every 10ms
After: pthread_cond_wait() with broadcast signals

Signals sent on:

• Stream channel assignment changes
• Streaming state transitions (start/stop)
• Active sender changes
• Credit grants from RDP layer
• Device stop requests

7. Comprehensive Unit Tests (3a7a8156)

Added CUnit test suites in src/protocols/rdp/tests/rdpecam/:

rdpecam_sink_test.c (520 lines)

Tests for frame queue operations:

• Frame enqueue/dequeue
• Queue overflow behavior
• Concurrent access
• Frame dropping logic
• Metadata handling

rdpecam_proto_test.c (482 lines)

Tests for protocol message handling:

• Message serialization/deserialization
• All message type builders
• Edge cases (null pointers, invalid data)
• Endianness correctness

rdpecam_caps_test.c (351 lines)

Tests for capability processing:

• Device name sanitization (Unicode, length limits)
• Capability parsing
• Format selection logic
• Error handling

Test Standards:

• ✅ All tests follow test_SUITENAME__TESTNAME() naming convention (required by generate-test-runner.pl)
• ✅ TAP (Test Anything Protocol) output via automake
• ✅ Integrated with make check
• Total: 1,353 lines of test coverage

8. Bug Fixes and Cleanup (8a9dbde6)

Fixed device mapping cleanup:

• Proper resource deallocation on disconnect
• Prevent memory leaks in device map
• Thread-safe cleanup procedures

---

Architecture

Component Hierarchy

┌─────────────────────────────────────────┐
│   Guacamole Core (Video Infrastructure) │
│   - guac_user.video_handler              │
│   - Video instruction handler            │
└────────────────┬────────────────────────┘
                 │
┌────────────────▼────────────────────────┐
│        RDP Protocol Handler              │
│   - Video handler registration           │
│   - RDPECAM plugin loading               │
└────────────────┬────────────────────────┘
                 │
        ┌────────┴────────┐
        │                 │
┌───────▼──────┐  ┌──────▼───────────┐
│ RDPECAM      │  │   RDPECAM        │
│ Channel      │◄─┤   Plugin         │
│              │  │   (guacrdpecam)  │
│ - rdpecam.c  │  │                  │
│ - caps.c     │  │ - Device mgmt    │
│ - sink.c     │  │ - Proto msgs     │
└──────────────┘  │ - Dequeue thread │
                  └──────────────────┘

Data Flow

Browser Client
    │ (video instruction)
    ▼
Guacamole Server (video_handler)
    │
    ▼
rdpecam_sink (frame queue)
    │
    ▼
guacrdpecam dequeue thread
    │ (pthread_cond_wait)
    ▼
rdpecam_proto (message formatting)
    │
    ▼
RDPECAM Channel (FreeRDP)
    │ (RDP virtual channel)
    ▼
Windows RDP Host (Camera DVR Service)

---

Key Features

✅ MS-RDPECAM compliant - Full implementation of Microsoft specification
✅ H.264 video streaming - Industry-standard codec support
✅ Capability negotiation - Automatic format/resolution selection
✅ Flow control - Credit-based backpressure handling
✅ Dynamic activation - Enable/disable during active sessions

---

Known Limitations

1. H.264 only - Current implementation supports H.264 exclusively
2. Windows compatibility - Remote host must support RDPECAM (Win10+, Server 2019+)
3. Single streaming camera - One streaming (In Use) camera device per connection

---

Dependencies

• FreeRDP 2.x OR 3.x

---

Ready for review and merge.

[tanonl]

As you will realize, this PR have been mostly using AI Coding. While exploring agentic AI programming, I wanted to do something actually useful and It came out pretty good.
Special thanks to Oleg for his guidance over LinkedIn. It's actually him who implemented the support of RDPECAM in FreeRDP

[necouchman]

As you will realize, this PR have been mostly using AI Coding. While exploring agentic AI programming, I wanted to do something actually useful and It can out pretty good. Special thanks to Oleg for his guidance over LinkedIn. It's actually him who implemented it the support of RDPECAM in FreeRDP

Similar comments, here, as on the client one - we'll need @mike-jumper's guidance on accepting/using AI-generated code, and the PR title needs to be updated to match the correct format.

[tanonl]

As you will realize, this PR have been mostly using AI Coding. While exploring agentic AI programming, I wanted to do something actually useful and It can out pretty good. Special thanks to Oleg for his guidance over LinkedIn. It's actually him who implemented it the support of RDPECAM in FreeRDP

Similar comments, here, as on the client one - we'll need @mike-jumper's guidance on accepting/using AI-generated code, and the PR title needs to be updated to match the correct format.

Yes, I was not sure what is the policy of Apache about AI assisted work.

[mike-jumper]

It's a tricky question. The primary concerns would be around copyright:

• If the AI happens to include someone else's copyrighted work in its output, then we would wouldn't have permission to include that (unless the original author gives permission).
• If the contents are largely machine-generated, they may not be copyrightable at all.

ASF policy says it's OK to include output from generative tools so long as it's verified to not infringe on someone else's copyright:

https://www.apache.org/legal/generative-tooling.html#include-in-contributions

[tanonl]

It's a tricky question. The primary concerns would be around copyright:

• If the AI happens to include someone else's copyrighted work in its output, then we would wouldn't have permission to include that (unless the original author gives permission).
• If the contents are largely machine-generated, they may not be copyrightable at all.

ASF policy says it's OK to include output from generative tools so long as it's verified to not infringe on someone else's copyright:

https://www.apache.org/legal/generative-tooling.html#include-in-contributions

I do not think there is any copyrighted work in this implementation. I spent a full week going back and forth, reviewing, adjusting and fixing the generated chunk of code produced by the agents. I am quite sure we are OK regarding that matter.

[Shankhadeep1234]

This is a solid and much-needed feature for remote users! The protocol-agnostic video handler infrastructure in the core library makes future camera/video streaming features much more maintainable across other protocols (e.g., VNC, SSH, etc.). Great to see that the pattern mirrors the audio handler work. Out of curiosity, do you foresee any unique challenges if/when adapting the video handler for non-H.264 codecs or non-RDP protocols?

[tanonl]

This is a solid and much-needed feature for remote users! The protocol-agnostic video handler infrastructure in the core library makes future camera/video streaming features much more maintainable across other protocols (e.g., VNC, SSH, etc.). Great to see that the pattern mirrors the audio handler work. Out of curiosity, do you foresee any unique challenges if/when adapting the video handler for non-H.264 codecs or non-RDP protocols

Hi @Shankhadeep1234, thank you for the positive feedback! I'm glad the protocol-agnostic design resonates with you.
For your question, I think the main challenges would be: -- browser codec support - H.264 has universal MediaRecorder support and hardware encoding, while VP8/VP9/AV1 vary significantly across browsers and may struggle with real-time encoding on older devices, -- protocol specs - The current RDPECAM implementation benefits from well-defined capability structures in the MS-RDPECAM spec, VNC has no official camera redirection standard so we'd need a custom RFB extension -- and SSH would likely require tunneling as separate forwarded channels rather than protocol integration. At least, the video_handler infrastructure and CameraRecorder.js are already codec/protocol-agnostic, so the heavy lifting is reusable - each other protocol mainly needs capability negotiation and a transport bridge layer.