Voice Interaction

Voice interaction connects microphone input to the selected device session. After selecting a device, requests use the current agent's model, tools, and collaboration scope, and answers are spoken back.

Before You Start

Before enabling voice, complete these steps:

• Create a Device Agent and define the device model for the target device. Voice requests ultimately resolve to these commands, property fields, and events.
• Connect the target device through the current IoT platform adapter or direct device connection, and make sure it can receive commands, return results, and report state.
• Prepare usable speech recognition and speech synthesis provider credentials.
• Console usage requires browser microphone permission. SDK or device-side integration requires device-side microphone capture and audio playback.

Configure Voice

Enable voice under Settings → Models → Voice model, then choose a speech provider. Current providers include:

Provider	Capability	Credentials	Recognition models	Synthesis models and voices
Volcengine	Speech recognition and speech synthesis	APP ID and Access Token (Chinese)	Recognition resource IDs (Chinese)	Synthesis resource IDs, voice list (Chinese)
Alibaba Cloud	Speech recognition and speech synthesis	API key	Real-time speech recognition	Real-time speech synthesis, CosyVoice voices, Qwen-Audio-TTS voices
AWS	Transcribe for speech recognition and Polly for speech synthesis	Access keys	Transcribe supported languages	Polly voices
ElevenLabs	Speech recognition and speech synthesis	API key	Models	Models, Voice Library

Voices must match the synthesis model. Volcengine synthesis resource IDs determine which voice version is available, and Alibaba Cloud CosyVoice and Qwen-Audio-TTS use separate voice lists. Available models and voices depend on the provider account.

Common settings include:

• Voice enablement.
• Voice bind address and port.
• Voice region, provider credentials, recognition model, synthesis model, and voice.

Configure recognition sample rate and voice TLS through environment variables. See Voice Configuration for configuration and remote access.

Console Usage and SDK Integration

Voice interaction has two usage paths. A common flow is to validate recognition quality and the control path in the console, then integrate the SDK on a real device.

Path	Use case	Requirement
Console usage	Development, demos, and validation that the device model works well with voice control	Enable voice, select a device, then start a request with the browser microphone
SDK/device-side integration	Devices with microphones, speakers, or system audio interfaces	Connect the device side to /ws/voice, send audio frames, and play returned TTS audio

Both paths enter the same Device Agent session and use the same device model.

Console Usage

Open a Device Agent workspace, select a device, then open the voice entry point. Later voice requests are bound to that device.

In the console, you can:

• Start a voice command related to the device model, such as querying state, adjusting a parameter, or triggering an action.
• Check whether the recognized text is correct and whether the Device Agent calls the expected command or reads the expected state.
• Listen to the spoken response or interrupt playback when needed.
• Convert the recognized text back into the input box to troubleshoot recognition or prompting issues.

If camera is also enabled, a voice turn can include the current camera frame. The Device Agent can use vision when the user asks about visible context, such as status lights, screen text, or objects in view.

SDK/Device-Side Integration

When the generated bundle provides voice support, use its connection values and examples. Otherwise, integrate through the WebSocket protocol. Device-side integration follows these steps:

1. Start the device-side MQTT or SDK connection and confirm that the device is online in the console.
2. Obtain the voice connection URL, authentication method, and audio parameters from the deployment. Use bundled values when available, then connect to the voice service at /ws/voice.
3. Send hello after the connection opens, including device ID, session ID, provider, and audio parameters.
4. Send listen when the user starts speaking, then stream 16 kHz mono Int16LE PCM audio frames.
5. Send stop when the user stops speaking. Attach vision frame references if the turn should include camera context.
6. Listen for ASR text, Device Agent replies, and TTS audio frames, then play the audio on the device speaker or system audio output.

Where the connection supports device-credential authentication, devices can use the identity supplied by their platform; the deployment determines the available method. See API Reference for message types, connection parameters, and a hello example.

When a voice request controls a device, the real device still receives commands and returns responses through the current adapter or direct device connection. The device side does not need a separate "voice command" branch that diverges from the device model.

To generate device-side microphone, speaker, or system audio integration logic, use agent-assisted SDK adaptation in SDK Access.

Adjust the voice experience

Maintain recognition hotwords, voice tool scope, and responses and feedback in Configuration Overview. Voice tool scope controls tool availability; A2A controller bindings still determine collaboration targets.

For a state query, set a waiting message such as Checking current device data under Responses and Feedback. The message reports progress; confirm device execution from the final result and latest state.

Platform deployments provide Monitoring for latency and speech usage. Use Logs to inspect an individual request.

Verify Results

After console configuration or SDK/device-side integration, verify these results:

• The voice WebSocket connects and hello returns session information.
• Microphone audio is recognized as text, and the recognized text enters the selected device conversation context.
• State queries read real device state, and control requests call commands defined in the device model.
• The device side receives commands, returns results, and shows corresponding records in the console or logs.
• TTS audio plays in the browser or on the real device, and abort interrupts current playback.
• For remote access or production deployment, clients use the public voice WebSocket URL published by the deployment administrator, including the correct public host and port; an HTTPS console uses wss://.

Notes

For microphone, provider, and connection requirements, see Before You Start and Configure Voice. Voice is optimized for manually started, short spoken exchanges. Long reports, tables, code, wake word, and continuous listening scenarios are better handled outside this voice flow.