Interact — Control the Browser

The interact tool gives the AI control over the browser — navigate, click, type, read elements, run JavaScript, upload files, record sessions, manage state, and display narration. Requires AI Web Pilot to be enabled in the extension popup.

Need one runnable call + response shape + failure fix for every action? See Interact Executable Examples.

Synchronous Mode

Tools now block until the extension returns a result (up to 15s). Set background: true to return immediately with a correlation_id.

Quick Reference

interact({what: "navigate", url: "https://example.com"})
interact({what: "click", selector: "text=Submit"})
interact({what: "type", selector: "label=Email", text: "user@example.com"})
interact({what: "list_interactive"})
interact({what: "screenshot"})
interact({what: "execute_js", script: "document.title"})
interact({what: "batch", steps: [{what: "click", selector: "text=Menu"}, ...]})
interact({what: "set_cookie", name: "theme", value: "dark", domain: "example.com"})

Common Parameters

These parameters can be added to any action:

Parameter	Type	Description
subtitle	string	Narration text displayed at bottom of viewport. Empty string clears.
reason	string	Why this action is being performed — shown in the action toast
correlation_id	string	Link this action to a specific error or investigation
analyze	boolean	Enable performance profiling (returns perf_diff in result)
tab_id	number	Target a specific tab (omit for active tab)
element_id	string	Stable element handle from list_interactive (preferred for deterministic follow-up actions)
index	number	Element index from list_interactive results (legacy alternative to selector/element_id)
index_generation	string	Generation token from list_interactive to ensure index resolves against the same element snapshot
scope_selector	string	Container CSS selector to constrain DOM actions to a specific region
include_screenshot	boolean	Capture a screenshot after the action completes and return it inline
evidence	string	Visual evidence capture mode: off (default), on_mutation, always
observe_mutations	boolean	Track element-level DOM mutations during action execution
wait_for_stable	boolean	Wait for DOM stability (no mutations) before returning. Composable with navigate and click.
frame	string	Target iframe: CSS selector, 0-based index, or "all"

Subtitles

Subtitles are persistent narration text displayed at the bottom of the viewport, like closed captions. They stay visible until replaced or cleared.

// Standalone
interact({what: "subtitle", text: "Creating a new project"})

// Composable — narration + action in one call
interact({what: "click", selector: "text=Create",
          subtitle: "One click to start a new project"})

// Clear
interact({what: "subtitle", text: ""})

Action Toasts

When enabled, action toasts show brief notifications at the top of the viewport for each action. The reason parameter customizes the toast label:

// Without reason: toast shows "Click: #submit-btn"
interact({what: "click", selector: "#submit-btn"})

// With reason: toast shows "Submit the registration form"
interact({what: "click", selector: "#submit-btn", reason: "Submit the registration form"})

Toasts can be toggled off in the extension popup — useful during demos.

---

Semantic Selectors

All actions that accept a selector parameter support semantic selectors in addition to CSS:

Syntax	Example	How it works
text=	text=Submit	Finds element whose text content contains “Submit”
role=	role=button	Finds [role="button"]
placeholder=	placeholder=Email	Finds [placeholder="Email"]
label=	label=Username	Finds label containing text, follows for attribute to target
aria-label=	aria-label=Close	Finds [aria-label="Close"] with starts-with matching
CSS	#submit-btn	Standard document.querySelector()

Semantic selectors are resilient to UI changes — they target meaning, not structure. text=Submit still works after a CSS redesign, a framework migration, or a component library upgrade.

The aria-label= selector uses starts-with matching, so aria-label=Send matches Gmail’s "Send ‪(⌘Enter)‬".

Iframe Targeting

Use the frame parameter to target elements inside iframes:

interact({what: "click", selector: "text=Submit", frame: "#payment-iframe"})
interact({what: "type", selector: "label=Card", text: "4242...", frame: 0})

Value	Behavior
CSS selector	Target iframe matching the selector
Number	Target iframe by 0-based index
"all"	Search all iframes

---

Navigation

navigate

interact({what: "navigate", url: "https://example.com"})

Navigates the active tab to the specified URL. Automatically includes perf_diff in the async result (before/after performance comparison).

Parameter	Type	Description
url	string (required)	URL to navigate to
include_content	boolean	Return page content with response (url, title, text_content, vitals)
new_tab	boolean	Open URL in a background tab instead of replacing current tab
auto_dismiss	boolean	After navigation completes, automatically dismiss cookie consent banners and overlays

refresh

interact({what: "refresh"})

Refreshes the current page. Includes perf_diff automatically.

back / forward

interact({what: "back"})
interact({what: "forward"})

Browser history navigation.

new_tab

interact({what: "new_tab", url: "https://example.com"})

Opens a new tab with the specified URL.

Parameter	Type	Description
url	string (required)	URL to open in the new tab

switch_tab

Switch to a different browser tab.

interact({what: "switch_tab", tab_id: 123})
interact({what: "switch_tab", tab_index: 2})

Parameter	Type	Description
tab_id	number	Tab ID to switch to
tab_index	number	Tab index in current window ordering (alternative to tab_id)
set_tracked	boolean	Whether to update the tracked tab to the newly activated tab (default: true). Set to false to switch focus without changing which tab the server targets.

close_tab

Close a browser tab.

interact({what: "close_tab", tab_id: 123})

Parameter	Type	Description
tab_id	number	Tab ID to close (omit for active tab)

activate_tab

Activate (bring to front) a specific browser tab.

interact({what: "activate_tab", tab_id: 123})

Parameter	Type	Description
tab_id	number	Tab ID to activate

---

DOM Primitives

click

interact({what: "click", selector: "text=Submit"})
interact({what: "click", selector: "#confirm-btn", reason: "Confirm the order"})

Parameter	Type	Description
selector	string	CSS or semantic selector

type

interact({what: "type", selector: "label=Email", text: "user@example.com"})
interact({what: "type", selector: "placeholder=Search", text: "wireless headphones", clear: true})

Parameter	Type	Description
selector	string	CSS or semantic selector
text	string	Text to type
clear	boolean	Clear existing value before typing

select

interact({what: "select", selector: "#country", value: "US"})

Parameter	Type	Description
selector	string	CSS or semantic selector for <select> element
value	string	Option value to select

check

interact({what: "check", selector: "text=I agree to the Terms"})
interact({what: "check", selector: "#newsletter", checked: false})

Parameter	Type	Default	Description
selector	string	—	CSS or semantic selector
checked	boolean	true	Whether to check or uncheck

key_press

interact({what: "key_press", selector: "label=Search", text: "Enter"})
interact({what: "key_press", text: "Escape"})

Parameter	Type	Description
selector	string	Element to target (optional — omit for page-level key press)
text	string	Key name: Enter, Tab, Escape, Backspace, ArrowDown, ArrowUp, Space

focus

interact({what: "focus", selector: "label=Email"})

hover

Hover over an element.

interact({what: "hover", selector: "text=Products"})
interact({what: "hover", selector: ".dropdown-trigger"})

Parameter	Type	Description
selector	string	CSS or semantic selector

hardware_click

Click using OS-level hardware simulation instead of DOM events. Useful for elements that don’t respond to synthetic clicks (e.g., custom canvas, embedded iframes with strict event handling).

interact({what: "hardware_click", selector: "#canvas-element"})

Parameter	Type	Description
selector	string	CSS or semantic selector

scroll_to

interact({what: "scroll_to", selector: "#pricing-section"})

wait_for

Wait for an element to appear on the page. Useful for async content.

interact({what: "wait_for", selector: "text=Dashboard", timeout_ms: 10000})

Parameter	Type	Default	Description
selector	string	—	CSS or semantic selector
timeout_ms	number	5000	Maximum wait time in milliseconds

wait_for_stable

Wait for the DOM to stop changing (no mutations) before returning. Useful after dynamic content loads.

interact({what: "wait_for_stable"})
interact({what: "wait_for_stable", stability_ms: 1000})

Parameter	Type	Default	Description
stability_ms	number	500	Milliseconds of DOM quiet time required

---

Element Reading

get_text

interact({what: "get_text", selector: ".order-total"})

Returns the text content of the matched element.

get_value

interact({what: "get_value", selector: "input[name='email']"})

Returns the current value of a form element.

get_attribute

interact({what: "get_attribute", selector: "#submit", name: "disabled"})

Parameter	Type	Description
name	string	Attribute name to read

query

Run selector queries without mutating page state.

interact({what: "query", selector: "text=Submit", query_type: "exists"})
interact({what: "query", selector: ".error", query_type: "count"})
interact({what: "query", selector: "#status", query_type: "text"})

Parameter	Type	Description
query_type	string	exists, count, text, text_all, or attributes
attribute_names	array	Attribute names when query_type is attributes

set_attribute

interact({what: "set_attribute", selector: "#submit", name: "disabled", value: "false"})

Parameter	Type	Description
name	string	Attribute name
value	string	Value to set

list_interactive

Discovers all interactive elements on the page — buttons, links, inputs, selects, etc. Returns up to 100 elements with suggested selectors.

interact({what: "list_interactive"})
interact({what: "list_interactive", limit: 20, visible_only: true})

Parameter	Type	Description
limit	number	Max elements to return (default: all)
visible_only	boolean	Only return visible elements

Returns an array of elements, each with:

Field	Description
tag	Element tag name (button, input, a, etc.)
type	Input type (email, password, text) — only for <input>
selector	Suggested selector (semantic preferred: text=, aria-label=, placeholder=)
label	Human-readable label (aria-label > title > placeholder > text content)
role	ARIA role
visible	Whether the element is visible on screen

This is the best way to discover what’s clickable on an unfamiliar page. The AI can read the list and choose the right selector.

---

JavaScript Execution

execute_js

Run arbitrary JavaScript in the page context. See How Gasoline Executes Page Scripts for the full deep dive.

interact({what: "execute_js", script: "document.title"})
interact({what: "execute_js", script: "window.location.href"})
interact({what: "execute_js",
          script: "document.querySelectorAll('li').length",
          world: "main"})

Parameter	Type	Default	Description
script	string	—	JavaScript code to execute
world	string	auto	auto (try main, fallback to isolated), main (page globals), isolated (CSP-proof)
timeout_ms	number	5000	Maximum execution time

---

Visual

highlight

Highlights an element on the page with a visual overlay. Useful for demos and debugging.

interact({what: "highlight", selector: "#error-message"})
interact({what: "highlight", selector: "text=Submit", duration_ms: 3000})

Parameter	Type	Default	Description
selector	string	—	Element to highlight
duration_ms	number	5000	How long to show the highlight

subtitle

Display persistent narration text at the bottom of the viewport.

interact({what: "subtitle", text: "Welcome to the product tour"})
interact({what: "subtitle", text: ""})  // Clear

Parameter	Type	Description
text	string (required)	Narration text. Empty string clears.

---

State Management

Aliases

state_save, state_load, state_list, and state_delete are aliases for save_state, load_state, list_states, and delete_state respectively.

save_state

Save the current browser state (URL, title, tab) as a named checkpoint.

interact({what: "save_state", snapshot_name: "logged-in"})

load_state

Restore a saved state checkpoint.

interact({what: "load_state", snapshot_name: "logged-in"})
interact({what: "load_state", snapshot_name: "logged-in", include_url: true})

Parameter	Type	Default	Description
snapshot_name	string	—	Name of the saved state
include_url	boolean	false	Navigate to the saved URL when restoring

list_states

interact({what: "list_states"})

Returns all saved states with metadata (name, URL, title, saved_at).

delete_state

interact({what: "delete_state", snapshot_name: "old-checkpoint"})

---

Browser Storage

set_storage

Set a key in localStorage or sessionStorage.

interact({what: "set_storage", storage_type: "localStorage", key: "theme", value: "dark"})
interact({what: "set_storage", storage_type: "sessionStorage", key: "debug", value: "true"})

Parameter	Type	Description
storage_type	string	localStorage or sessionStorage
key	string	Storage key
value	string	Value to set

delete_storage

Remove a key from localStorage or sessionStorage.

interact({what: "delete_storage", storage_type: "localStorage", key: "theme"})

Parameter	Type	Description
storage_type	string	localStorage or sessionStorage
key	string	Storage key to delete

clear_storage

Clear all keys from localStorage or sessionStorage.

interact({what: "clear_storage", storage_type: "localStorage"})

Parameter	Type	Description
storage_type	string	localStorage or sessionStorage

set_cookie

Set a browser cookie.

interact({what: "set_cookie", name: "session_id", value: "abc123", domain: "example.com"})

Parameter	Type	Description
name	string	Cookie name
value	string	Cookie value
domain	string	Cookie domain
path	string	Cookie path (default: /)

delete_cookie

Delete a browser cookie.

interact({what: "delete_cookie", name: "session_id", domain: "example.com"})

Parameter	Type	Description
name	string	Cookie name
domain	string	Cookie domain
path	string	Cookie path (default: /)

---

Dialog Handling

confirm_top_dialog

Accept (confirm/OK) the topmost browser dialog (alert, confirm, prompt).

interact({what: "confirm_top_dialog"})

dismiss_top_overlay

Dismiss the topmost overlay, modal, or popup on the page.

interact({what: "dismiss_top_overlay"})

auto_dismiss_overlays

Automatically detect and dismiss cookie consent banners, newsletter popups, and other overlays.

interact({what: "auto_dismiss_overlays"})

---

File Upload

upload

Upload a file via native file dialogs. Works with <input type="file"> elements and drag-and-drop zones.

interact({what: "upload", selector: "input[type='file']", file_path: "/path/to/report.pdf"})

Parameter	Type	Description
selector	string	File input element selector
file_path	string	Absolute path to the file to upload
name	string	Filename (alternative to file_path)
api_endpoint	string	API endpoint for direct upload mode
submit	boolean	Submit form after upload

---

Draw Mode

draw_mode_start

Activate the visual annotation overlay. Users draw rectangles on the page and type feedback. Press ESC to finish. Retrieve annotations with analyze({what: "annotations"}).

interact({what: "draw_mode_start"})
interact({what: "draw_mode_start", annot_session: "checkout-review", wait: true, timeout_ms: 300000})

Parameter	Type	Default	Description
annot_session	string	—	Named session for multi-page annotation review
wait	boolean	—	Block until the user finishes drawing (press ESC)
timeout_ms	number	300000	Max wait time (only with wait: true, max 600000)

---

Recording

screen_recording_start

Start video recording of the current tab.

interact({what: "screen_recording_start"})
interact({what: "screen_recording_start", name: "checkout-flow", audio: "tab", fps: 30})

Parameter	Type	Default	Description
name	string	Auto-generated	Recording name
audio	string	—	Audio source: tab, mic, or both. Omit for video-only.
fps	number	15	Frames per second (5-60)

screen_recording_stop

Stop the active video recording.

interact({what: "screen_recording_stop"})
interact({what: "screen_recording_stop", name: "checkout-flow"})

Parameter	Type	Description
name	string	Recording name (if multiple active)

screenshot

Capture a screenshot from the interact tool. Prefer observe({what: "screenshot"}) for passive captures — this version is useful when composing with other interact actions via include_screenshot.

interact({what: "screenshot"})
interact({what: "screenshot", selector: ".hero-section"})

Parameter	Type	Default	Description
selector	string	—	Capture a specific element by CSS selector
full_page	boolean	false	Capture the full scrollable page
format	string	png	Image format: png or jpeg
quality	number	—	JPEG quality 1-100 (only for jpeg)
wait_for_stable	boolean	false	Wait for layout to stabilize before capture

---

Clipboard

clipboard_read

Read the current clipboard text.

interact({what: "clipboard_read"})

clipboard_write

Write text to the clipboard.

interact({what: "clipboard_write", text: "Deploy status: green"})

paste

Paste text at the currently focused element.

interact({what: "paste", text: "Hello, world!"})

Parameter	Type	Description
text	string	Text to paste

---

Composer

open_composer

Open the page’s native compose/new-message UI (e.g., Gmail’s compose window).

interact({what: "open_composer"})

submit_active_composer

Submit the currently active compose form.

interact({what: "submit_active_composer"})

---

Content Extraction

get_readable

Extract the page content as readable text (similar to reader mode).

interact({what: "get_readable"})

get_markdown

Extract the page content as markdown.

interact({what: "get_markdown"})

---

Compound Actions

These actions combine multiple steps into a single call, reducing round-trips.

navigate_and_wait_for

Navigate to a URL and wait for a specific element to appear before returning.

interact({what: "navigate_and_wait_for", url: "https://example.com/dashboard", wait_for: ".dashboard-loaded"})

Parameter	Type	Description
url	string (required)	URL to navigate to
wait_for	string	CSS selector to wait for after navigation
timeout_ms	number	Max wait time for the selector

navigate_and_document

Navigate and return a structured page snapshot in one call.

interact({what: "navigate_and_document", url: "https://example.com/docs"})
interact({what: "navigate_and_document", url: "https://example.com/docs", include_screenshot: true})

fill_form

Fill multiple form fields without submitting. Use when you need to fill fields and then perform additional actions before submission.

interact({what: "fill_form",
          fields: [
            {selector: "label=Email", value: "user@example.com"},
            {selector: "label=Name", value: "Jane Doe"}
          ]})

Parameter	Type	Description
fields	array	Form fields: [{selector, value, index?}]

fill_form_and_submit

Fill multiple form fields and submit the form in a single action.

interact({what: "fill_form_and_submit",
          fields: [
            {selector: "label=Email", value: "user@example.com"},
            {selector: "label=Password", value: "password123"}
          ],
          submit_selector: "text=Sign In"})

Parameter	Type	Description
fields	array	Form fields: [{selector, value, index?}]
submit_selector	string	CSS or semantic selector for submit button
submit_index	number	Submit button index from list_interactive (alternative to selector)

run_a11y_and_export_sarif

Run an accessibility audit and export results as SARIF in one step.

interact({what: "run_a11y_and_export_sarif", save_to: "/tmp/a11y.sarif"})

Parameter	Type	Description
save_to	string	File path to save the SARIF report

explore_page

Autonomous page exploration — discovers interactive elements, follows links, and maps the page structure.

interact({what: "explore_page"})

batch

Execute multiple interact actions sequentially in a single call. Reduces round-trips for multi-step flows.

interact({what: "batch",
          steps: [
            {what: "navigate", url: "https://example.com/login"},
            {what: "type", selector: "label=Email", text: "user@example.com"},
            {what: "type", selector: "label=Password", text: "password123"},
            {what: "click", selector: "text=Sign In"}
          ]})

Parameter	Type	Default	Description
steps	array (required)	—	Ordered list of interact action objects
step_timeout_ms	number	10000	Timeout per step
continue_on_error	boolean	true	Continue executing remaining steps after a failure
stop_after_step	number	—	Stop execution after this many steps

---

Performance Profiling

Add analyze: true to any DOM action to get a perf_diff in the result — before/after timing comparison with Web Vitals ratings and a verdict.

interact({what: "click", selector: "text=Load Dashboard", analyze: true})

The navigate and refresh actions include perf_diff automatically.