BorgVR Guide

Agent control

Model Context Protocol

Trusted AI agents can inspect datasets, control the visualization, and evaluate rendered images without driving the user interface remotely.

Mandelbulb volume controlled through BorgVR
Edit states and filesThe Data and Edit States guide explains storage, automatic edit persistence, variants, shared SharePlay state, and export. MCP uses the same restrictions as the UI; in particular, states cannot be switched during a running session.

What BorgVR MCP provides

All three native renderers contain the same optional MCP server. It is independent of the dataset and WebGPU servers and uses the same central command layer as the macOS scripting language. An agent can therefore act reproducibly instead of operating buttons through mouse or gaze control.

  • Read the dataset catalog with complete metadata and known origins, then open or close datasets
  • Read and change rendering mode, isovalue, transform, clipping, and lighting
  • Manage objects, sphere and stroke markers, mesh instances, and length, area, and volume measurements
  • Hide objects or the complete measurement visualization without deleting data
  • Wait for a completed rendered frame, retrieve the current view as a PNG image, and export isosurfaces as OBJ on macOS

Enabling MCP

  1. Open BorgVR Settings and select the MCP section.
  2. Enable Enable MCP server and choose another port if required. The default is 8765.
  3. Add one of the displayed endpoints to the configuration of your MCP client.
This device onlyWithout network access enabled, BorgVR listens only at http://127.0.0.1:8765/mcp. Local connections require neither TLS nor an access token.

Access from the network

When Allow connections from the network is enabled, BorgVR binds the MCP server to all available network interfaces and displays an HTTPS endpoint for every reachable address. Depending on router, firewall, and port forwarding, the service may also be reachable from outside the local network.

  • Generate an access token or enter your own. The client sends it as Authorization: Bearer ….
  • The generation button copies the new token directly to the clipboard.
  • Network connections use HTTPS. Without an imported PKCS#12 identity, BorgVR creates a temporary self-signed certificate whenever the server starts.
Trust boundaryA connected MCP client can open datasets, change the visualization, edit annotations and measurements, erase and restore local volume data, discard all local edits, export files, and read screenshots. Share the endpoint and token only with trusted systems.

HTTP requests are limited to 4 MiB including at most 16 KiB of headers. Headers must arrive within 10 seconds and the complete request within 60 seconds; receive pauses longer than 10 seconds close the connection. Response transmission has a separate 30-second deadline. Command execution does not count as receive time. Network requests are authenticated as soon as their headers arrive, before the complete body is loaded.

Typical agent workflow

A complete catalog query is limited to 60 seconds, 2000 datasets, and 16 MiB of serialized catalog data. Dataset-server text replies are limited to 4 MiB and lists to 10000 entries; MCP metadata replies to 32 MiB per dataset. Timeouts and task cancellation close active catalog connections. Exceeded limits are reported as errors, not as a successful, silently truncated catalog.

  1. The agent uses list_datasets to inspect metadata and origins, then opens the selected dataset with open_dataset.
  2. It inspects get_application_state, changes the view and rendering parameters, and hides objects or measurements when required.
  3. After each change it uses wait_until_rendered to wait for a new completed frame, then calls capture_screenshot.
  4. On macOS it can use export_isosurface to save the current dataset as an OBJ file at the current or an explicitly supplied normalized isovalue.
  5. By comparing multiple views, rendering modes, isovalues, or clipping ranges, the agent can inspect visible structures systematically.

The JSON schemas returned by tools/list are the authoritative reference for argument names and value ranges.

Transfer functions and picture configurations

get_transfer_function and set_transfer_function read and set a zero-based channel's TF. Each entry is an array of four RGBA bytes from 0 through 255. Compact color ramps are linearly interpolated to the current table size. Other channels remain unchanged; RGB/RGBA volumes have no TFs. load_transfer_function and save_transfer_function use native .tf1d files.

get_background and set_background offer system, solid, and gradient modes on macOS and iOS. primary is the solid/top color and secondary the bottom color; both are RGBA arrays from 0 through 1. visionOS retains passthrough. For capture_screenshot, include_alpha:true (default) preserves the transparent render image. false composites the background and produces a PNG without alpha; visionOS uses black instead of passthrough. output_path optionally also saves the PNG to a file.

save_visualization_state and load_visualization_state store all channel TFs, transform, render mode/isovalue, clipping, lighting, background, and screen projection in a freely named JSON file. The same dataset must be open. Like the UI, visionOS stores positioning relative to the head rather than stale room coordinates. Objects and measurements are not part of this picture configuration.

{"name":"set_transfer_function","arguments":{"channel":0,"samples":[[0,0,0,0],[255,80,0,100],[255,255,255,255]]}}
{"name":"set_background","arguments":{"mode":"solid","primary":[0,0,0,1]}}
{"name":"save_visualization_state","arguments":{"output_path":"/path/on/app-device/picture.json"}}
{"name":"capture_screenshot","arguments":{"include_alpha":false,"output_path":"/path/on/app-device/picture.png"}}

Back up and restore settings

save_settings, reset_settings, and load_settings let agents back up registered settings, work with defaults, then restore the previous configuration. The .borgvrsettings JSON format is compatible with the UI; missing entries remain unchanged and unknown entries are ignored. Credentials are exported only with include_credentials:true. Reset requires confirm:true. Import and reset always preserve the running MCP connection's settings (enable, port, token, and TLS), so the agent stays connected.

All file paths are absolute paths on the app device and must be accessible to its sandbox. Existing output files are replaced only with overwrite:true. After an app update, the MCP client must refresh its tool list.

{"name":"save_settings","arguments":{"output_path":"/path/on/app-device/backup.borgvrsettings","include_credentials":true}}
{"name":"reset_settings","arguments":{"confirm":true}}
{"name":"load_settings","arguments":{"input_path":"/path/on/app-device/backup.borgvrsettings"}}

Brick diagnostics and optimizations

set_bounding_box uses a boolean visible to show or hide the complete dataset frame. This local setting persists and is recorded as GSC; renderingOptions.datasetBoundingBoxVisible reports its current value.

set_brick_visualization accepts optional boolean fields enabled, resident, empty, and missing. Type filters are local and also apply to subblocks: red = resident, green = skipped as empty, blue = missing or coarser substitute. Hiding a type changes neither traversal nor brick requests.

set_render_optimizations accepts boolean subbrick_skipping and opacity_adaptive_sampling, plus alpha_epsilon from 0 through 1. These settings persist locally. Alpha epsilon affects only TF empty-space classification, not the transfer function itself or iso rendering.

For both tools, omitted fields remain unchanged; at least one field is required. get_application_state reports all current values in renderingOptions. While GSC recording is active, UI and MCP changes are recorded, including the initial values of these options.

{"name":"set_brick_visualization","arguments":{"enabled":true,"resident":false,"empty":true,"missing":true}}
{"name":"set_render_optimizations","arguments":{"subbrick_skipping":true,"opacity_adaptive_sampling":true,"alpha_epsilon":0}}

macOS window control

get_window_state returns content/frame dimensions in logical points and backing pixels, display scale, and fullscreen/transition status. resize_window accepts width and height (1 through 16384), optional units: points|pixels and area: content|frame. Defaults are points/content; frame includes window decoration. Window size limits still apply, and neither backing dimension may exceed 16384 pixels. Only the main window changes, not the active panel or script log.

set_fullscreen explicitly sets native fullscreen with enabled:true or false. An already matching state is unchanged. Transitions wait for AppKit's completion notification; resize exits fullscreen first when necessary. Conflicting simultaneous transitions or open sheets blocking fullscreen report an error. On iOS and visionOS these three tools report that window control is unsupported. Camera projection stays unchanged.

{"name":"get_window_state","arguments":{}}
{"name":"resize_window","arguments":{"width":2880,"height":1800,"units":"pixels","area":"frame"}}
{"name":"set_fullscreen","arguments":{"enabled":true}}

Screen projection

set_screen_projection sets the local camera on macOS and iOS. Optional arguments are field_of_view_degrees (10 through 120 degrees) and parallel (boolean); at least one is required. Omitted values stay unchanged. The field of view also controls equivalent framing in parallel projection. Settings persist per device and are not synchronized through SharePlay. Changes are recorded while GSC recording is active.

get_application_state.renderingOptions includes screenFieldOfViewDegrees and parallelProjection on macOS and iOS. On visionOS these fields are absent, and the setter reports that the operation is unsupported.

{"name":"set_screen_projection","arguments":{"field_of_view_degrees":45,"parallel":false}}

Volume editing and export

configure_volume_brush also accepts channels: "all" or an array of one-based channel numbers, e.g. [1,3]. At least one valid channel is required. get_volume_edit_state reports componentCount and channels as a list of selected, one-based channel numbers. Opening selects all channels. Selection is local; SharePlay carries it with each operation. discard_volume_edits always resets every channel.

Dataset and isosurface exports are disabled during SharePlay. Edits and edit-state changes are rejected while an export is running. Starting SharePlay cancels a running export.

The following tools are available on macOS, iOS, and visionOS. Edits affect the active edit state and are shared during SharePlay. The original file is never modified.

  • get_volume_edit_state returns availability, dataset ID, brush settings, edit status, pending brick count, and the latest export job with UUID, status, progress, and any error. The same state is included in get_application_state.volumeEditing.
  • get_volume_edit_state includes savedStates, activeStateID, and SharePlay status. select_edit_state selects a UUID or original; create_edit_state creates an empty state or a copy with duplicate=true. rename_edit_state changes its name. delete_edit_state removes an inactive state with confirm=true. State management is locked during SharePlay.
  • configure_volume_brush accepts optional mode (erase/restore), radius_fraction (0.002–0.15 of the largest extent), project_onto_volume, and activate. The latter activates hand/mouse/touch editing and defaults to false; Muse/PSVR modes stay independent.
  • edit_volume requires mode and samples, an array of objects with center_meters: [x,y,z] and positive radius_meters. Coordinates are physical meters from the volume minimum corner, not screen, hand, or world coordinates. Interactive radius and projection do not affect these explicit samples; the channel selection from configure_volume_brush applies here too. At most 65536 samples per call within the HTTP request limit; no interpolation. Erase sets selected channels to 0; restore recovers only their original values.
  • edit_volume returns after journaling. Use wait_volume_edits with optional timeout_seconds (default 30, maximum 86400) to wait for background jobs. Unreachable remote bricks may cause a timeout.
  • prepare_volume_edits rebuilds pending edited bricks with foreground priority and waits for persistence. Uses Metal when available; cancellation preserves recorded edits.
  • discard_volume_edits requires explicit confirm: true and clears only the active edit state. During SharePlay, it clears the shared state; other states are preserved.
  • export_edited_dataset starts an export job with absolute output_path (.data), plus optional name and description. Paths refer to the app device and must be accessible to its sandbox. Existing destinations are refused. The consistent export includes all previously recorded edits and preserves source compression; waiting for brick jobs first is unnecessary.
  • Poll get_volume_edit_state until export.status is completed, cancelled, or failed; progress ranges from 0 to 1. cancel_volume_export with the job UUID in id cancels only that job. Incomplete export files are removed; recorded edits remain.
{"name":"edit_volume","arguments":{"mode":"erase","samples":[{"center_meters":[0.5,0.5,0.5],"radius_meters":0.05}]}}
{"name":"export_edited_dataset","arguments":{"output_path":"/path/on/app-device/edited.data","name":"Edited volume"}}

Troubleshooting

Empty response or no connection
Verify that scheme and endpoint match: local access uses http, network access uses https.
Unauthorized
For network access, the Authorization header must contain exactly the currently displayed bearer token. Local access on the same device does not check a token.
Certificate error
Explicitly trust the self-signed certificate or import your own PKCS#12 identity. Some externally hosted agents cannot reach private or local IP addresses at all.