Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions server/services/s2s/grok.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
title: "Grok Realtime"
description: "Real-time speech-to-speech service implementation using xAI's Grok Voice Agent API"
---

## Overview

`GrokRealtimeLLMService` provides real-time, multimodal conversation capabilities using xAI's Grok Voice Agent API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management with low-latency response times.

<CardGroup cols={2}>
<Card
title="Grok Realtime API Reference"
icon="code"
href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.grok.realtime.llm.html"
>
Pipecat's API methods for Grok Realtime integration
</Card>
<Card
title="Example Implementation"
icon="play"
href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/51-grok-realtime.py"
>
Complete Grok Realtime conversation example
</Card>
<Card
title="Grok Voice Documentation"
icon="book"
href="https://docs.x.ai/docs/guides/voice"
>
Official xAI Grok Voice Agent API documentation
</Card>
<Card title="xAI Console" icon="external-link" href="https://console.x.ai/">
Access Grok models and manage API keys
</Card>
</CardGroup>

## Installation

To use Grok Realtime services, install the required dependencies:

```bash
pip install "pipecat-ai[grok]"
```

## Prerequisites

### xAI Account Setup

Before using Grok Realtime services, you need:

1. **xAI Account**: Sign up at [xAI Console](https://console.x.ai/)
2. **API Key**: Generate a Grok API key from your account dashboard
3. **Model Access**: Ensure access to Grok Voice Agent models
4. **Usage Limits**: Configure appropriate usage limits and billing

### Required Environment Variables

- `XAI_API_KEY`: Your xAI API key for authentication

### Key Features

- **Real-time Speech-to-Speech**: Direct audio processing with low latency
- **Multilingual Support**: Support for multiple languages
- **Voice Activity Detection**: Server-side VAD for automatic speech detection
- **Function Calling**: Seamless support for external functions and tool integration
- **Multiple Voice Options**: Various voice personalities available
- **WebSocket Support**: Real-time bidirectional audio streaming
71 changes: 71 additions & 0 deletions server/services/s2s/ultravox.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
title: "Ultravox Realtime"
description: "Real-time speech-to-speech service implementation using Ultravox's Realtime API"
---

## Overview

`UltravoxRealtimeLLMService` provides real-time conversational AI capabilities using Ultravox's Realtime API. It supports both text and audio modalities with voice transcription, streaming responses, and tool usage for creating interactive AI experiences.

<CardGroup cols={2}>
<Card
title="Ultravox Realtime API Reference"
icon="code"
href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.ultravox.llm.html"
>
Pipecat's API methods for Ultravox Realtime integration
</Card>
<Card
title="Example Implementation"
icon="play"
href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/50-ultravox-realtime.py"
>
Complete Ultravox Realtime conversation example
</Card>
<Card
title="Ultravox Documentation"
icon="book"
href="https://docs.ultravox.ai/overview"
>
Official Ultravox API documentation
</Card>
<Card
title="Ultravox Console"
icon="external-link"
href="https://app.ultravox.ai/"
>
Access Ultravox models and manage API keys
</Card>
</CardGroup>

## Installation

To use Ultravox Realtime services, install the required dependencies:

```bash
pip install "pipecat-ai[ultravox]"
```

## Prerequisites

### Ultravox Account Setup

Before using Ultravox Realtime services, you need:

1. **Ultravox Account**: Sign up at [Ultravox Console](https://app.ultravox.ai/)
2. **API Key**: Generate an Ultravox API key from your account dashboard
3. **Model Access**: Ensure access to Ultravox Realtime models
4. **Usage Limits**: Configure appropriate usage limits and billing

### Required Environment Variables

- `ULTRAVOX_API_KEY`: Your Ultravox API key for authentication

### Key Features

- **Audio-Native Model**: Ultravox is an audio-native model for natural voice interactions
- **Real-time Streaming**: Low-latency audio processing and streaming responses
- **Multiple Input Modes**: Support for Agent, One-Shot, and Join URL input parameters
- **Voice Transcription**: Built-in transcription with streaming output
- **Function Calling**: Support for tool integration and API calling
- **Configurable Duration**: Set maximum call duration limits
3 changes: 2 additions & 1 deletion server/services/supported-services.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ Speech-to-Text services receive and audio input and output transcriptions.
| [Sarvam](/server/services/stt/sarvam) | `pip install "pipecat-ai[sarvam]"` |
| [Soniox](/server/services/stt/soniox) | `pip install "pipecat-ai[soniox]"` |
| [Speechmatics](/server/services/stt/speechmatics) | `pip install "pipecat-ai[speechmatics]"` |
| [Ultravox](/server/services/stt/ultravox) | `pip install "pipecat-ai[ultravox]"` |
| [Whisper](/server/services/stt/whisper) | `pip install "pipecat-ai[whisper]"` |

## Large Language Models
Expand Down Expand Up @@ -118,7 +117,9 @@ Speech-to-Speech services are multi-modal LLM services that take in audio, video
| [AWS Nova Sonic](/server/services/s2s/aws) | `pip install "pipecat-ai[aws-nova-sonic]"` |
| [Gemini Multimodal Live](/server/services/s2s/gemini-live) | `pip install "pipecat-ai[google]"` |
| [Gemini Live Vertex AI](/server/services/s2s/gemini-live-vertex) | `pip install "pipecat-ai[google]"` |
| [Grok Voice Agent](/server/services/s2s/grok) | `pip install "pipecat-ai[grok]"` |
| [OpenAI Realtime](/server/services/s2s/openai) | `pip install "pipecat-ai[openai]"` |
| [Ultravox](/server/services/stt/ultravox) | `pip install "pipecat-ai[ultravox]"` |

## Image Generation

Expand Down