Deep dive
Under the hood
A quick tour of how ScreenMind turns a screen into memory — enough to understand it, without the internals you'd need to rebuild it.
┌ Capture ───┐ ┌ Analyze · one GPU ───┐ ┌ Store ───┐
│ dedup │──▶│ OCR → Gemma 4 → embed │──▶│ SQLite │──▶ Search · Chat
│ a11y+redact│ └───────────────────────┘ │ + FTS5 │ Agents · MCP
└────────────┘ ▲ └──────────┘
└── Chat · Voice · Meetings share the same modelCapture → memory
1
Smart capture
A worker watches the screen and only saves when it meaningfully changes (perceptual hashing), auto-pausing on games/editors and skipping blocked apps.
2
Read the screen
Accessibility APIs and OCR pull the on-screen text, handed to the model as context so it spends its budget understanding, not reading.
3
Redact secrets
A filter strips cards, SSNs, API keys and passwords before anything is stored or sent to the model.
4
Understand
Gemma 4 turns the frame + text into structured meaning — app, activity, summary, mood, layout.
5
Index
A MiniLM embedding (on CPU) plus a full-text index land in local SQLite, powering hybrid semantic + keyword search.
One model, one GPU
Everything — screen vision, meeting audio, chat and summaries — runs on a single local Gemma 4 model. The interesting part is sharing one GPU gracefully:
- Chat comes first. A chat message cancels any in-flight analysis, frees the GPU in under a second, then answers — the dropped frame is re-queued.
- Don't repeat work. A per-app cache skips frames that barely changed, so most captures never touch the model.
- Three modes. Fast (~12s), Balanced (~40s) or Accurate (~76s) on a 4 GB GTX 1650 — and far quicker on bigger GPUs.
Footprint
- ~3–4 GB VRAM, ~1.5–2 GB RAM, ~80–150 MB disk per active day.
- 100% local — the only network call is the one-time model download.
- Windows, macOS and Linux (X11 + Wayland) via an OS-abstraction layer.
Fun fact: “fast” mode pre-fills an empty <think></think> block so the model skips its own reasoning and answers immediately — that's most of the jump from ~76s down to ~12s per frame.
↯ back to the brain