OpenAIRealTimeWebRTCTransport

Overview

The OpenAI Realtime WebRTC transport implementation enables real-time audio communication directly with the OpenAI Realtime API using WebRTC voice-to-voice service. It handles media device management, audio/video streams, and state management for the connection.

Installation

Add the OpenAI transport package to your project:

.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-openai-realtime.git", from: "0.0.1"),

// Add to your target dependencies
.target(name: "YourApp", dependencies: [
    .product(name: "PipecatClientIOSOpenAIRealtimeWebrtc", package: "pipecat-client-ios-openai-realtime")
],

Usage

Create a client:

let rtviClientOptions = RTVIClientOptions.init(
    enableMic: currentSettings.enableMic,
    enableCam: false,
    params: .init(config: [
        .init(
            service: "llm",
            options: [
                .init(name: "api_key", value: .string(openaiAPIKey)),
                .init(name: "initial_messages", value: .array([
                    .object([
                        "role": .string("user"), // "user" | "system"
                        "content": .string("Start by introducing yourself.")
                    ])
                ])),
                .init(name: "session_config", value: .object([
                    "instructions": .string("You are Chatbot, a friendly and helpful assistant who provides useful information, including weather updates."),
                    "voice": .string("echo"),
                    "input_audio_noise_reduction": .object([
                        "type": .string("near_field")
                    ]),
                    "turn_detection": .object([
                        "type": .string("semantic_vad")
                    ])
                ])),
            ]
        )
    ])
)
self.rtviClientIOS = RTVIClient.init(
    transport: OpenAIRealtimeTransport.init(options: rtviClientOptions),
    options: rtviClientOptions
)
try await rtviClientIOS.start()

Currently, invalid session configurations will result in the OpenAI connection being failed.

API Reference

Demo

Simple Chatbot OpenAI Demo

Source

iOS OpenAI Realtime WebRTC

Pipecat iOS Client Reference

Complete API documentation for the OpenAI transport implementation