Desktop and computer use#

A TaskCloud sandbox can run a full Linux desktop. People can watch and control it in the browser, and AI agents can drive it through a screenshot and input API.

The desktop has two paths:

• People open a signed stream URL (KasmVNC) in a browser, or in an <iframe> inside your product.
• Agents take screenshots (GET /sandboxes/{sandbox_id}/screenshot) and send mouse and keyboard actions (POST /sandboxes/{sandbox_id}/input). Agents don't need the video stream, and screenshots cost much less.

Private beta. Latency numbers on this page are targets, not guarantees.

The desktop template#

The template is a RAM snapshot of a desktop that is already running, so a new sandbox starts with the screen ready.

No GPU. WebGL, 3D maps and video playback run slowly. Office apps, web apps and ordinary browsing work well.

import { Client } from "@taskcloud/sdk";

const client = new Client(); // reads TASKCLOUD_API_KEY
const sandbox = await client.sandboxes.create({ template: "desktop", size: "default" });

from taskcloud import Client

client = Client()
sandbox = client.sandboxes.create(template="desktop", size="default")

taskcloud create --template desktop

To add the desktop stack to your own image, set desktop: true when you create a template from an image. See Templates.

Watching in a browser: stream URLs#

POST /sandboxes/{sandbox_id}/desktop returns a signed URL of the form https://<short_id>-desktop.taskhost.cloud/?token=....

curl -X POST https://api.task.cloud/v1/sandboxes/$SANDBOX_ID/desktop \
  -H "Authorization: Bearer $TASKCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"ttl_seconds": 900, "read_only": true}'

{
  "id": "dsk_7zab3kq2mxw5",
  "url": "https://a2b3c4d5e6f7-desktop.taskhost.cloud/?token=dt_...",
  "read_only": true,
  "single_use": false,
  "expires_at": "2026-11-09T12:15:00Z",
  "created_at": "2026-11-09T12:00:00Z"
}

const stream = await sandbox.desktop.stream({ ttl_seconds: 900, read_only: true });
console.log(stream.url);

// Later, before it expires:
await sandbox.desktop.revoke(stream.id);

stream = sandbox.desktop.stream(ttl_seconds=900, read_only=True)
sandbox.desktop.revoke(stream.id)

taskcloud desktop $SANDBOX_ID --ttl 15m --read-only --open

• List active URLs: GET /sandboxes/{sandbox_id}/desktop.
• Revoke a URL: DELETE /sandboxes/{sandbox_id}/desktop/{stream_id}. This invalidates the URL and disconnects anyone using it.

• Embed it in your product:

  <iframe src="https://a2b3c4d5e6f7-desktop.taskhost.cloud/?token=dt_..."
          style="width:100%;aspect-ratio:16/9;border:0" allow="clipboard-read; clipboard-write"></iframe>

• Wake on request: opening the URL of a paused sandbox resumes it. The target is under 2 s from click to desktop.

• Auto-pause: an active viewer counts as activity, so the sandbox doesn't pause while someone is watching. A desktop with no viewers and no input pauses normally after its idle timeout. See Lifecycle.
• The session survives a pause. Pausing snapshots RAM, so windows, open browser tabs and logged-in sessions are all still there after a resume. The viewer reconnects automatically.

Screenshots#

GET /sandboxes/{sandbox_id}/screenshot. Target p95 is under 300 ms.

The response format depends on the Accept header:

• Default: the raw image, with X-Screen-Width and X-Screen-Height headers.
• Accept: application/json: a JSON object { data, media_type, width, height, screen_width, screen_height, taken_at }, where data is base64.

curl -s https://api.task.cloud/v1/sandboxes/$SANDBOX_ID/screenshot \
  -H "Authorization: Bearer $TASKCLOUD_API_KEY" -o shot.png
# or
taskcloud screenshot $SANDBOX_ID -o shot.png

const shot = await sandbox.desktop.screenshot({ format: "png" });
// shot.data is base64; shot.screen_width x shot.screen_height is the coordinate space

Input#

POST /sandboxes/{sandbox_id}/input runs a batch of actions in order. Action names and fields follow the Anthropic computer-use tool, so a model's tool call can be forwarded as { "action": <tool name>, ...input }.

Coordinates are [x, y] in full-screen pixels, with the origin at the top left.

Request options:

• return_screenshot: true returns a screenshot taken after the last action. It is taken after screenshot_delay_ms (default 300) so the UI can settle.
• coordinate_scale divides incoming coordinates. Use it when they come from a screenshot taken with scale < 1.

If an action fails, the actions after it are not run and are reported with status: "skipped". The cursor position is available at GET /sandboxes/{sandbox_id}/cursor.

curl -X POST https://api.task.cloud/v1/sandboxes/$SANDBOX_ID/input \
  -H "Authorization: Bearer $TASKCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      {"action": "left_click", "coordinate": [640, 412]},
      {"action": "type", "text": "hello world"},
      {"action": "key", "text": "Return"}
    ],
    "return_screenshot": true
  }'

{
  "results": [
    {"index": 0, "action": "left_click", "status": "ok", "error": null},
    {"index": 1, "action": "type", "status": "ok", "error": null},
    {"index": 2, "action": "key", "status": "ok", "error": null}
  ],
  "cursor": {"x": 640, "y": 412},
  "screenshot": {"data": "iVBORw0...", "media_type": "image/png", "width": 1920, "height": 1080, "screen_width": 1920, "screen_height": 1080}
}

Example: a computer-use agent loop with Claude#

These examples connect Claude's computer-use toolset (computer_toolset_20260801) to a sandbox. The toolset is generally available on the Claude API with no beta header. The tool entry takes no name and no display size.

How the toolset maps to the sandbox API:

• Calls: each call from Claude is a tool_use block with toolset_name: "computer", and the block's name is the action (screenshot, zoom, left_click, type, key, ...). One turn can contain several blocks.
• Results: return one tool_result per block, all in the next user message, and each one must include toolset_name: "computer". Return results for every block:
  • screenshot and zoom return an image.
  • cursor_position returns text such as X=512, Y=384.
  • Other actions return OK.
  • If an action fails, set is_error: true on its result, and answer every later block in that turn with Not executed: an earlier computer action in this turn failed.
• Mapping to the sandbox:
  • screenshot → sandbox.desktop.screenshot()
  • zoom → a screenshot with region
  • cursor_position → sandbox.desktop.cursor()
  • Everything else → sandbox.desktop.input({ actions: [{ action: block.name, ...block.input }] })
• No scaling needed at 1920x1080. That resolution fits the model's image limits (up to 2576 px on the long edge), so coordinates pass through unchanged. If you downscale screenshots with scale, also send coordinate_scale with the same value on input.
• Optional config: configs: { zoom: { enabled: false } } turns off the zoom action if you don't want it.

TypeScript#

import Anthropic from "@anthropic-ai/sdk";
import { Client } from "@taskcloud/sdk";

const anthropic = new Anthropic();
const client = new Client();

const sandbox = await client.sandboxes.create({ template: "desktop", size: "default" });
console.log("Watch live:", (await sandbox.desktop.stream({ read_only: true, ttl_seconds: 3600 })).url);

type Block = { type: string; id?: string; name?: string; input?: any; toolset_name?: string };

async function runAction(block: Block) {
  const input = block.input ?? {};
  switch (block.name) {
    case "screenshot": {
      const s = await sandbox.desktop.screenshot({ format: "png" });
      return [{ type: "image", source: { type: "base64", media_type: "image/png", data: s.data } }];
    }
    case "zoom": {
      const [x0, y0, x1, y1] = input.region;
      const s = await sandbox.desktop.screenshot({ format: "png", region: `${x0},${y0},${x1},${y1}` });
      return [{ type: "image", source: { type: "base64", media_type: "image/png", data: s.data } }];
    }
    case "cursor_position": {
      const c = await sandbox.desktop.cursor();
      return [{ type: "text", text: `X=${c.x}, Y=${c.y}` }];
    }
    default: {
      const r = await sandbox.desktop.input({ actions: [{ action: block.name, ...input }] });
      const res = r.results[0];
      if (res.status !== "ok") throw new Error(res.error ?? `${block.name} failed`);
      return [{ type: "text", text: "OK" }];
    }
  }
}

const messages: any[] = [
  { role: "user", content: "Open Firefox, go to news.ycombinator.com and tell me the top story title." },
];

try {
  while (true) {
    const response = await anthropic.messages.create({
      model: "claude-opus-5-5",
      max_tokens: 16000,
      tools: [{ type: "computer_toolset_20260801" } as any],
      messages,
    });
    messages.push({ role: "assistant", content: response.content });

    if (response.stop_reason === "refusal") {
      console.error("Model declined the request:", response.stop_details);
      break;
    }
    const calls = (response.content as Block[]).filter((b) => b.type === "tool_use");
    if (calls.length === 0) {
      for (const b of response.content) if (b.type === "text") console.log(b.text);
      break;
    }

    const results: any[] = [];
    let failed = false;
    for (const call of calls) {
      const base = { type: "tool_result", tool_use_id: call.id, toolset_name: "computer" };
      if (failed) {
        results.push({ ...base, is_error: true, content: "Not executed: an earlier computer action in this turn failed." });
        continue;
      }
      try {
        results.push({ ...base, content: await runAction(call) });
      } catch (err) {
        failed = true;
        results.push({ ...base, is_error: true, content: String(err) });
      }
    }
    messages.push({ role: "user", content: results });
  }
} finally {
  await sandbox.pause(); // $0 compute while paused; the desktop session is kept
}

Python#

import anthropic
from taskcloud import Client

claude = anthropic.Anthropic()
client = Client()

sandbox = client.sandboxes.create(template="desktop", size="default")
print("Watch live:", sandbox.desktop.stream(read_only=True, ttl_seconds=3600).url)

SKIPPED = "Not executed: an earlier computer action in this turn failed."


def image(data: str) -> list:
    return [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": data}}]


def run_action(block) -> list:
    name, inp = block.name, dict(block.input or {})
    if name == "screenshot":
        return image(sandbox.desktop.screenshot(format="png").data)
    if name == "zoom":
        x0, y0, x1, y1 = inp["region"]
        return image(sandbox.desktop.screenshot(format="png", region=f"{x0},{y0},{x1},{y1}").data)
    if name == "cursor_position":
        c = sandbox.desktop.cursor()
        return [{"type": "text", "text": f"X={c.x}, Y={c.y}"}]
    result = sandbox.desktop.input(actions=[{"action": name, **inp}]).results[0]
    if result.status != "ok":
        raise RuntimeError(result.error or f"{name} failed")
    return [{"type": "text", "text": "OK"}]


messages = [{"role": "user", "content": "Open Firefox, go to news.ycombinator.com and tell me the top story title."}]

try:
    while True:
        response = claude.messages.create(
            model="claude-opus-5-5",
            max_tokens=16000,
            tools=[{"type": "computer_toolset_20260801"}],
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason == "refusal":
            print("Model declined the request:", response.stop_details)
            break
        calls = [b for b in response.content if b.type == "tool_use"]
        if not calls:
            print("".join(b.text for b in response.content if b.type == "text"))
            break

        results, failed = [], False
        for call in calls:
            base = {"type": "tool_result", "tool_use_id": call.id, "toolset_name": "computer"}
            if failed:
                results.append({**base, "is_error": True, "content": SKIPPED})
                continue
            try:
                results.append({**base, "content": run_action(call)})
            except Exception as err:
                failed = True
                results.append({**base, "is_error": True, "content": str(err)})
        messages.append({"role": "user", "content": results})
finally:
    sandbox.pause()  # $0 compute while paused; the desktop session is kept

Tips#

• Batch the non-visual actions into one input call when you control the sequence, and set return_screenshot: true. That saves a round trip.
• Give a person a link to watch the agent. Use a read_only stream URL with a short TTL.
• Fork a prepared desktop to parallelize work. For example, log in once and then fork with memory. Each fork has its own browser session. See Snapshots and fork.

Security#

• Use a sandbox for untrusted browsing. Each sandbox is a separate microVM with its own kernel. Don't put long-lived credentials on a desktop that an agent drives through untrusted websites.
• Prompt injection is a risk. Web pages can contain text that tries to redirect the agent. Restrict what the agent may do: use an API key limited to that sandbox (restrict.sandbox_ids or restrict.labels), keep a human in the loop for sensitive actions, and review the agent's actions in a recorded or read-only stream.
• Hand out viewer links carefully. Use read_only: true, a short ttl_seconds and single_use: true for links you give to end users, and revoke them when you're done. The VNC port is never exposed without a signed token.

See Security and Limits.

Roadmap#

• WebRTC streaming at 60 fps, audio, clipboard sync and file upload into the desktop are on the roadmap. They are not part of the MVP.
• No GPU instances are planned for the MVP.