Connection check
local install
screenhand
Give AI eyes and hands on your desktop. Open-source MCP server for desktop automation — screenshots, UI control, browser automation, OCR. Works with Claude, Cursor, and any MCP client. macOS + Windows.
Tools
85
GitHub stars
14
Installs / wk
33
Licence
AGPL-3.0
Transport
stdio
Last checked
never
Tools & capabilities
85 toolsRead from the published package source — this server runs locally, so there is no endpoint to query. Names are taken from the code, not observed at runtime, and descriptions are often absent.
app_focus
from source
Bring a running application to the foreground. Bring a running application to the foreground.
app_idle
from source
app_launch
from source
Launch a macOS/Windows application by bundle ID (e.g., Launch a macOS/Windows application by bundle ID (e.g.,
applescript
from source
Run an AppleScript command. For controlling Finder, Safari, Mail, Notes, etc. (macOS only). WARNING: Executes arbitrary AppleScript — can perform destructive actions (delete files,… Run an AppleScript command. For controlling Finder, Safari, Mail, Notes, etc. (macOS only). WARNING: Executes arbitrary AppleScript — can perform destructive actions (delete files, send emails). All executions are audit-logged.
app_list
from source
List all running applications with their bundle IDs, names, and PIDs. List all running applications with their bundle IDs, names, and PIDs.
apps
from source
List all running applications with bundle IDs and PIDs List all running applications with bundle IDs and PIDs
browser_click
from source
Click an element in Chrome/Electron by CSS selector. Uses CDP Input.dispatchMouseEvent for realistic mouse events. Click an element in Chrome/Electron by CSS selector. Uses CDP Input.dispatchMouseEvent for realistic mouse events.
browser_dom
from source
Query the DOM of a Chrome/Electron page. Returns matching elements Query the DOM of a Chrome/Electron page. Returns matching elements
browser_fill_form
from source
Fill a form field with human-like typing (anti-detection). Uses real keyboard events via CDP Input domain. Fill a form field with human-like typing (anti-detection). Uses real keyboard events via CDP Input domain.
browser_human_click
from source
Alias for browser_click — both use realistic mouseMoved → mousePressed → mouseReleased events. Prefer browser_click directly. Alias for browser_click — both use realistic mouseMoved → mousePressed → mouseReleased events. Prefer browser_click directly.
browser_js
from source
Execute JavaScript in a Chrome/Electron tab. Returns the result. WARNING: This runs arbitrary JS in the browser context — avoid on sensitive pages (banking, email). All executions… Execute JavaScript in a Chrome/Electron tab. Returns the result. WARNING: This runs arbitrary JS in the browser context — avoid on sensitive pages (banking, email). All executions are audit-logged.
browser_navigate
from source
Navigate the active Chrome/Electron tab to a URL Navigate the active Chrome/Electron tab to a URL
browser_open
from source
Open a URL in Chrome/Electron (creates new tab) Open a URL in Chrome/Electron (creates new tab)
browser_page_info
from source
Get current page title, URL, and text content summary Get current page title, URL, and text content summary
browser_stealth
from source
Inject anti-detection patches into Chrome/Electron page. Call once after navigating to a protected site. Hides webdriver flag, patches plugins/languages/permissions. Inject anti-detection patches into Chrome/Electron page. Call once after navigating to a protected site. Hides webdriver flag, patches plugins/languages/permissions.
browser_tabs
from source
List all open Chrome/Electron tabs. Use cdpPort to connect to a specific app (e.g. 9333 for Codex Desktop). List all open Chrome/Electron tabs. Use cdpPort to connect to a specific app (e.g. 9333 for Codex Desktop).
browser_type
from source
Type into an input field in Chrome/Electron. Uses CDP Input.dispatchKeyEvent for real keyboard events (works with React/Angular). Type into an input field in Chrome/Electron. Uses CDP Input.dispatchKeyEvent for real keyboard events (works with React/Angular).
browser_wait
from source
Wait for a condition on a Chrome/Electron page Wait for a condition on a Chrome/Electron page
click
from source
Click at screen coordinates Click at screen coordinates
click_text
from source
SLOW fallback: Find text on screen via OCR and click it. Use ui_press instead when possible — it SLOW fallback: Find text on screen via OCR and click it. Use ui_press instead when possible — it
click_with_fallback
from source
Click a target by text using the canonical fallback chain: AX → CDP → OCR. Automatically retries and falls through methods. Click a target by text using the canonical fallback chain: AX → CDP → OCR. Automatically retries and falls through methods.
coverage_report
from source
Check what ScreenHand knows about an app: shortcuts, selectors, flows, playbooks, error patterns, and stability %. Useful before complex workflows to decide strategy: learn first (… Check what ScreenHand knows about an app: shortcuts, selectors, flows, playbooks, error patterns, and stability %. Useful before complex workflows to decide strategy: learn first (if empty), go fast (if high coverage), or use fallback tools (if error patterns exist). Optional for quick actions.
discover_features
from source
Extract features from an app Extract features from an app
drag
from source
Drag from one point to another Drag from one point to another
element_tree
from source
Get the accessibility element tree of the current app. Useful for understanding the UI structure and finding elements to interact with. Get the accessibility element tree of the current app. Useful for understanding the UI structure and finding elements to interact with.
execution_plan
from source
Show the execution plan for an action type. Returns the ordered fallback chain based on available infrastructure. Show the execution plan for an action type. Returns the ordered fallback chain based on available infrastructure.
export_playbook
from source
Generate a playbook JSON from your session. Extracts URLs, selectors, errors+solutions from memory. Share the output with ScreenHand to help others automate this platform. Generate a playbook JSON from your session. Extracts URLs, selectors, errors+solutions from memory. Share the output with ScreenHand to help others automate this platform.
extract
from source
Extract data from a UI element. Returns text content, table data, or structured JSON from the element. Extract data from a UI element. Returns text content, table data, or structured JSON from the element.
focus
from source
Focus/activate an application (or a specific window by windowId) Focus/activate an application (or a specific window by windowId)
ingest_documentation
from source
Parse a documentation page (HTML, markdown, or text) and extract shortcuts, workflows, and tips. Merges extracted knowledge into the app Parse a documentation page (HTML, markdown, or text) and extract shortcuts, workflows, and tips. Merges extracted knowledge into the app
ingest_tutorial
from source
Extract structured playbook steps from a video transcript (e.g. YouTube captions). Converts tutorial narration into actionable automation steps with tool mappings. Extract structured playbook steps from a video transcript (e.g. YouTube captions). Converts tutorial narration into actionable automation steps with tool mappings.
key
from source
Press a key combination Press a key combination
key_combo
from source
Send a keyboard shortcut. Keys: Send a keyboard shortcut. Keys:
launch
from source
Launch an application. Chrome/Chromium browsers are launched with CDP enabled (port 9222) for browser_* tools. Launch an application. Chrome/Chromium browsers are launched with CDP enabled (port 9222) for browser_* tools.
locate_with_fallback
from source
Find an element Find an element
map_app
from source
Visually map an app Visually map an app
menu_click
from source
Click a menu item in an app Click a menu item in an app
navigate
from source
Navigate a browser to a URL, or open an app via Navigate a browser to a URL, or open an app via
observer_ocr_roi
from source
Submit a targeted ROI OCR command to the running observer daemon. The daemon captures the window region, runs OCR, and stores the result. Non-blocking — returns a command ID you ca… Submit a targeted ROI OCR command to the running observer daemon. The daemon captures the window region, runs OCR, and stores the result. Non-blocking — returns a command ID you can poll with a second call.
observer_start
from source
Start the observer daemon to continuously watch an app window. Captures frames via CGWindowListCreateImage, runs OCR only when pixels change, detects popups. Zero overhead on engin… Start the observer daemon to continuously watch an app window. Captures frames via CGWindowListCreateImage, runs OCR only when pixels change, detects popups. Zero overhead on engine — reads a JSON file.
observer_status
from source
Get observer daemon status — frames captured, OCR text, popup detection. Get observer daemon status — frames captured, OCR text, popup detection.
observer_stop
from source
Stop the observer daemon. Stop the observer daemon.
ocr
from source
OCR a window with element positions. SLOW — prefer ui_tree for structured element discovery. Use OCR only for reading visual/canvas content. OCR a window with element positions. SLOW — prefer ui_tree for structured element discovery. Use OCR only for reading visual/canvas content.
orchestrator_start
from source
Start the multi-agent orchestrator daemon. Manages parallel worker slots: web tasks (CDP) run in parallel, native tasks (AX/keyboard) are serialized per-app. Survives restarts. Start the multi-agent orchestrator daemon. Manages parallel worker slots: web tasks (CDP) run in parallel, native tasks (AX/keyboard) are serialized per-app. Survives restarts.
orchestrator_status
from source
Get orchestrator status — worker slots, task queue, active/completed tasks. Get orchestrator status — worker slots, task queue, active/completed tasks.
orchestrator_stop
from source
Stop the orchestrator daemon. Running tasks finish before exit. Stop the orchestrator daemon. Running tasks finish before exit.
orchestrator_submit
from source
Submit a task to the orchestrator. Web tasks (CDP) run in parallel, native tasks queue per-app. Returns immediately — task is processed asynchronously. Submit a task to the orchestrator. Web tasks (CDP) run in parallel, native tasks queue per-app. Returns immediately — task is processed asynchronously.
platform_explore
from source
Autonomously explore an app or website. Maps all interactive elements, tries each one, records working selectors and broken paths. Outputs a reference JSON. Autonomously explore an app or website. Maps all interactive elements, tries each one, records working selectors and broken paths. Outputs a reference JSON.
platform_guide
from source
Get automation guide for a platform (selectors, URLs, flows, error solutions). Reads from references/ (curated knowledge). Zero cost — only loads when called. Get automation guide for a platform (selectors, URLs, flows, error solutions). Reads from references/ (curated knowledge). Zero cost — only loads when called.
platform_learn
from source
Scrape official docs, help center, keyboard shortcuts for a platform. Crawls pages via Chrome and extracts structured data into a reference JSON. Scrape official docs, help center, keyboard shortcuts for a platform. Crawls pages via Chrome and extracts structured data into a reference JSON.
playbook_list
from source
List all available playbooks with their IDs, names, platforms, and success rates. List all available playbooks with their IDs, names, platforms, and success rates.
playbook_preflight
from source
Quick feasibility check before automating a platform. Scans the page for known blockers (captchas, WebGL, iframes), checks against playbook errors, tests selector availability. Ret… Quick feasibility check before automating a platform. Scans the page for known blockers (captchas, WebGL, iframes), checks against playbook errors, tests selector availability. Returns go/yellow/red.
playbook_record
from source
Macro recorder: start/stop/trim/clean recorded playbooks. Use Macro recorder: start/stop/trim/clean recorded playbooks. Use
playbook_run
from source
Execute a saved playbook by ID or auto-match by task description. Playbooks run deterministically without AI calls. If a step fails, AI automatically recovers and patches the playb… Execute a saved playbook by ID or auto-match by task description. Playbooks run deterministically without AI calls. If a step fails, AI automatically recovers and patches the playbook for next time.
press
from source
Click/press a UI element. Finds the element by text, role, selector, or coordinates, then clicks it. Click/press a UI element. Finds the element by text, role, selector, or coordinates, then clicks it.
read_with_fallback
from source
Read text content from the screen or a specific element using the canonical fallback chain: AX → CDP → OCR. Returns the text found. Read text content from the screen or a specific element using the canonical fallback chain: AX → CDP → OCR. Returns the text found.
recording_cancel
from source
Cancel the current recording without saving. Cancel the current recording without saving.
recording_start
from source
Start recording user actions to auto-generate a playbook. Do the task manually while recording, then call recording_stop to save. Start recording user actions to auto-generate a playbook. Do the task manually while recording, then call recording_stop to save.
recording_status
from source
Check if recording is active and how many events captured so far. Check if recording is active and how many events captured so far.
recording_stop
from source
Stop recording and save the captured actions as a new playbook. Stop recording and save the captured actions as a new playbook.
scan_menu_bar
from source
Scan an app Scan an app
screenshot
from source
Take a screenshot and OCR it. Returns all visible text. NOTE: For finding/clicking UI elements, ui_tree + ui_press is 10x faster. Take a screenshot and OCR it. Returns all visible text. NOTE: For finding/clicking UI elements, ui_tree + ui_press is 10x faster.
screenshot_file
from source
Take a screenshot and return the file path (for viewing the actual image) Take a screenshot and return the file path (for viewing the actual image)
scroll
from source
Scroll at a position Scroll at a position
scroll_with_fallback
from source
Scroll within an element or the active window using the canonical fallback chain: AX → CDP → coordinates. Scrolls until target text is visible, or by a fixed amount. Scroll within an element or the active window using the canonical fallback chain: AX → CDP → coordinates. Scrolls until target text is visible, or by a fixed amount.
select_with_fallback
from source
Select an option from a dropdown/menu using the canonical fallback chain: AX → CDP. Finds the control, opens it, and picks the specified option. Select an option from a dropdown/menu using the canonical fallback chain: AX → CDP. Finds the control, opens it, and picks the specified option.
session_start
from source
Start a new automation session. Returns a sessionId needed by all other tools. Automatically attaches to the frontmost app. Start a new automation session. Returns a sessionId needed by all other tools. Automatically attaches to the frontmost app.
task_run
from source
Run a complete task autonomously. Starts an observe→decide→act loop that uses the accessibility tree (not screenshots) to see the UI and Claude to decide each action. The loop cont… Run a complete task autonomously. Starts an observe→decide→act loop that uses the accessibility tree (not screenshots) to see the UI and Claude to decide each action. The loop continues until the task is fully done or max steps reached. Returns a summary of all actions taken.
type_into
from source
Type text into a UI element (text field, search box, etc). Locates the field, optionally clears it, then types. Type text into a UI element (text field, search box, etc). Locates the field, optionally clears it, then types.
type_text
from source
Type text using the keyboard. Auto-detects Electron apps and routes through CDP for reliable editor input. Type text using the keyboard. Auto-detects Electron apps and routes through CDP for reliable editor input.
type_with_fallback
from source
Type text into a target field using the canonical fallback chain: AX → CDP → coordinates. Finds the field by label/placeholder, focuses it, then types. Type text into a target field using the canonical fallback chain: AX → CDP → coordinates. Finds the field by label/placeholder, focuses it, then types.
ui_find
from source
Find a specific UI element by text, title, or value. Falls back to value search if title match fails (e.g. finds Safari URL bar by URL). Find a specific UI element by text, title, or value. Falls back to value search if title match fails (e.g. finds Safari URL bar by URL).
ui_press
from source
PREFERRED: Find and press/click a UI element by its title via Accessibility. Faster and more reliable than click_text — no screenshot needed. PREFERRED: Find and press/click a UI element by its title via Accessibility. Faster and more reliable than click_text — no screenshot needed.
ui_set_value
from source
Set the value of a UI element (text field, slider, etc.). Searches by title first, falls back to value match. Set the value of a UI element (text field, slider, etc.). Searches by title first, falls back to value match.
ui_tree
from source
PREFERRED: Get the full UI element tree of an app via Accessibility. ~50ms, no screenshot/OCR. Use this FIRST to find elements — returns titles, roles, and bounds. Then use ui_pres… PREFERRED: Get the full UI element tree of an app via Accessibility. ~50ms, no screenshot/OCR. Use this FIRST to find elements — returns titles, roles, and bounds. Then use ui_press/ui_find to interact.
wait_for
from source
Wait for a condition: element appears/disappears, text appears, URL changes, window title matches, etc. Wait for a condition: element appears/disappears, text appears, URL changes, window title matches, etc.
wait_for_state
from source
Wait until a condition is met on screen: text appears, text disappears, or element becomes available. Polls at intervals using the fallback chain. Wait until a condition is met on screen: text appears, text disappears, or element becomes available. Polls at intervals using the fallback chain.
watch_dialog
from source
Register a dialog watch rule: when a dialog matching the pattern appears, auto-execute an action. Register a dialog watch rule: when a dialog matching the pattern appears, auto-execute an action.
watch_register
from source
Register a watch rule: when element with matching title appears, execute an action. Use for automated responses to known UI states. Register a watch rule: when element with matching title appears, execute an action. Use for automated responses to known UI states.
watch_start
from source
Start the state watcher polling loop. Evaluates registered watch rules every 2s against the world model. Start the state watcher polling loop. Evaluates registered watch rules every 2s against the world model.
watch_status
from source
Get all registered watch rules and their fire counts. Get all registered watch rules and their fire counts.
watch_stop
from source
Stop the state watcher polling loop. Stop the state watcher polling loop.
watch_unregister
from source
Remove a watch rule by ID. Remove a watch rule by ID.
window_list
from source
List all visible windows with their titles, positions, and sizes. List all visible windows with their titles, positions, and sizes.
windows
from source