Substrate

<div align="center">

# Substrate

**An open-source autonomous desktop agent powered by local and cloud LLMs.**

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Platform: Windows](https://img.shields.io/badge/Platform-Windows%2010%2F11-0078D6.svg)](#system-requirements)
[![Version](https://img.shields.io/badge/Version-1.1.0-green.svg)](package.json)

</div>

---

Substrate is a fully autonomous desktop agent that gives an LLM complete control over your computer — shell commands, file operations, browser automation, mouse/keyboard, screen capture, voice I/O, MIDI/hardware device control, memory, scheduling, and more. It runs locally with Ollama or connects to cloud providers (OpenAI, Anthropic, Google, Perplexity), and ships with both an Electron desktop app and a PWA-capable WebUI.

Think of it as an always-on AI co-pilot that lives on your machine, learns from your interactions, runs tasks on a schedule, and can operate autonomously in the background — all while keeping your data local.

<!-- TODO: Add screenshot here -->
<!-- ![Substrate Desktop UI](website/img/screenshot-desktop.png) -->

---

## Features

### Desktop Control
- **Shell execution** — Run any command with background support, streaming output, and process management
- **File operations** — Read, write, edit, grep, list, and inspect files with smart diffing
- **Process & window management** — List, kill, focus, resize, and send keystrokes to any window
- **Mouse & keyboard** — Pixel-level click, drag, scroll, hotkeys, and text typing
- **UI automation** — Windows-native element inspection and interaction via pywinauto
- **Screen capture & recording** — Screenshot any window or region, record screen sessions

### Multi-Model LLM Support
- **Local models** — Ollama (Llama, DeepSeek, Qwen, Dolphin, Mistral, Falcon, and more)
- **Cloud providers** — OpenAI (GPT-4), Anthropic (Claude), Google (Gemini), Perplexity
- **Any OpenAI-compatible endpoint** — LM Studio, vLLM, text-generation-webui, etc.
- **Hot-swap models** — Switch models mid-conversation from the UI dropdown
- **Vision support** — Send images to multimodal models for analysis

### Tool System
- **20+ built-in tools** — File ops, exec, browser, desktop, mouse, screen, memory, search, media, obsidian, and more
- **On-demand loading** — Tools activate based on conversation context (no wasted tokens)
- **MCP server support** — Connect external Model Context Protocol tool servers
- **Skill system** — Record workflows, save as reusable skills, promote from emergent to finalized
- **Plugin architecture** — Hook-based plugin system for extending functionality

### Browser Automation
- **CDP-based control** — Full Chrome DevTools Protocol automation of Chrome/Edge
- **Tab management** — Open, close, switch, and navigate tabs
- **Element interaction** — Click, type, submit forms, read content, extract elements
- **JavaScript evaluation** — Execute arbitrary JS in page context
- **Page screenshots** — Capture rendered pages for visual analysis

### Voice
- **Kokoro-82M** — Local text-to-speech with zero cloud dependency
- **ElevenLabs** — High-quality cloud TTS with voice cloning support
- **Speech recognition** — Voice input via microphone
- **Auto-speak** — Optionally speak all agent responses aloud

### Memory
- **Unified SQLite database** — Single source of truth for all memory
- **FTS5 full-text search** — Fast BM25 keyword matching
- **Vector embeddings** — Semantic similarity search with cached embeddings
- **Hybrid search** — Combined keyword + semantic retrieval with configurable weights
- **Session memory** — Cross-session context persistence
- **Deduplication** — Hash-based prevention of duplicate entries

### Circuits & Scheduling
- **CIRCUITS.md** — File-driven task scheduling (the agent reads and executes tasks from a markdown file)
- **Cron jobs** — Standard cron-style scheduling for recurring tasks
- **Background daemon** — System tray service that runs circuits even when the UI is closed
- **PRIME.md** — Startup tasks that run once on each cold boot

### Autonomous Mode
- **Background agent loops** — Periodic autonomous execution with configurable intervals
- **Sub-agent spawning** — Spawn parallel background agents for concurrent work
- **Followup queues** — Priority queue system for chaining autonomous actions
- **Screenshot observation** — Periodically observe and react to screen content

### Interfaces
- **Electron desktop app** — Animated avatar, retro-styled chat, radial config panel
- **WebUI** — PWA-capable browser UI accessible from any device on your network
- **Remote bridge** — HTTPS + QR code pairing for phone/tablet access
- **System tray** — Background daemon with quick-action menu

### And More
- **Profile system** — Multiple user profiles with separate configs, avatars, and knowledge
- **Canvas** — Shared scratchpad pages served via local HTTP
- **Webhooks** — Inbound webhook triggers for external integrations
- **Obsidian integr

... (truncated)