Agent Orchestration

Agent orchestration combines multiple published Device Agents to accomplish a goal. The orchestration engine is built into the Device Agent gateway, starts with bun run dev:agent, and calls the selected agents over A2A over MQTT. There are two ways to orchestrate: scenes and tasks.

Prerequisites

• A2A collaboration is enabled and at least two Device Agents are published and online. See A2A Collaboration.
• The web console has VITE_FF_A2A_MARKETPLACE_ENABLE=true so the A2A Marketplace entry is visible.
• The gateway has a usable LLM configured. The engine uses it to generate scene graphs and task plans. See LLM Configuration.

Scenes

A scene is a predefined multi-agent flow: you select agents and a goal, and the orchestration engine uses the LLM to break the goal into a directed acyclic graph (DAG). Each node is one agent invocation; edges between nodes express execution dependencies.

Create a scene

1. Open the A2A Marketplace and click Create Scene in the scenes panel on the right.
2. Enter a scene name and a goal description. A more specific goal produces a more accurate graph.
3. Select the online agents to involve.
4. Click Create Scene.

The scene starts in the "creating" state while the engine analyzes the selected agents' capabilities and generates the graph in the background, then becomes "ready". If a selected agent is offline, the goal cannot be decomposed, or the LLM call fails, the state becomes "failed".

During generation the model reflects and validates: can every piece of information a step needs be obtained from some agent or from user input? If a required parameter is missing (for example, "too hot" without a threshold), the scene enters the "needs input" state and lists questions; answer them on the create page and the model regenerates with your input. If the plan is unsound, the model re-plans and retries.

Important: the model never bakes planning-time facts (today's date, the current time) into a task, because they are stale by run time. Every runtime value a condition needs (current temperature, current time, whether a door is open) must be provided by a selected agent at run time. If the goal needs a runtime observation that none of the selected agents can provide (for example "at night" needs the current time, but no agent can report it), scene generation fails with an explanation of which capability is missing and which kind of agent to add; add such an agent and create the scene again.

View the flow graph

Click a scene card to open its flow graph. Each node is one agent invocation and shows the target agent and the step instruction; edges show execution dependencies. Nodes marked terminal are final output steps whose results are joined into the scene's final reply. A downstream step's instruction may reference an upstream step's output with {<upstream task id>}, which is substituted with the actual result at run time.

Start a scene

Click Start on a scene card or in the flow graph view. This is equivalent to telling the scene go. The engine dispatches agents in dependency order: steps with no dependencies run first; when a step completes, the steps depending on it become ready; if a step fails, the failure propagates downstream. A step that depends on upstream output has those results injected into its prompt (conditionals live in the downstream step's prompt), and progress streams live into the orchestration chat box, including what each step sent, what it received from upstream, and what it returned.

Tasks

A task is an ad-hoc multi-agent instruction that does not need a predefined graph. Describe a goal in natural language in the orchestration chat box; the engine uses the LLM to generate an execution plan (which agents to call and what to say to each), then calls them in sequence and replies like a normal conversation.

Tasks suit one-off, cross-agent needs such as "check the temperature in every room and set the air conditioner to 24 degrees in the rooms that are too warm".

Live status

Both scene and task progress appear in the orchestration chat box on the A2A Marketplace page:

• A progress entry updates when each step starts, completes, or fails.
• The final result returns as a message.
• Failures call out the failing step and the reason.

Progress is pushed to the page over SSE in real time; no manual refresh is needed.

Limitations

• Run state is held in memory. If the gateway restarts, in-flight runs are interrupted; scene definitions persist and can be started again after a restart.
• The engine dispatches to agents that are online at run time. If an agent goes offline mid-run, its step fails.
• Scene graphs and task plans are LLM-generated; quality depends on model capability and how clearly the goal is described.