Build a Private Local AI Assistant

Choose the privacy claim first

🟨 Gotcha: a local MCP bridge to Figma does not make a Figma file local. Figma describes itself as cloud-based and says offline edits sync when connectivity returns. Keep confidential designs in a tool whose storage policy meets your requirement, or treat Figma access as a separately approved cloud boundary.

A small architecture that can grow

User → local UI / CLI → task host → local model runtime
                         │             (loopback only)
                         ├→ file broker → approved workspace
                         ├→ build broker → bounded build command
                         └→ optional plugin/MCP broker → approved app
                         │
                         └→ local state, redacted logs, cancel control

Text alternative: the task host owns state and permissions. It sends selected context to a local model, validates proposed tool calls, and dispatches only approved operations to isolated brokers. Optional app integrations are separate privacy boundaries.

Start with one model, one agent, one disposable project, and one read-only tool. Add a write tool only after read access and denial behavior work. Add subagents only when independent tasks benefit from them. Use model selection and benchmarks to choose a candidate; memory and quality claims need a test on the actual machine.

Local inference starter

Ollama's current quickstart documents a local model and API at 127.0.0.1:11434. Install the runtime from its official site, then use a model that fits your available memory. This example uses the quickstart's gemma4:e2b for a harmless smoke test; it is not a claim that this is the best coding model. The initial model download requires network access.

ollama pull gemma4:e2b
ollama run gemma4:e2b

After leaving the interactive chat, test the local API with synthetic input:

curl http://127.0.0.1:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma4:e2b","messages":[{"role":"user","content":"Say hello without reading files."}],"stream":false}'

For Ollama-only local mode, merge "disable_ollama_cloud": true into the existing ~/.ollama/server.json and restart the runtime, or launch its server with OLLAMA_NO_CLOUD=1. Its FAQ says this disables Ollama cloud models and web search and that the API binds to loopback by default. Check the actual listener and log message after restart. This setting governs Ollama features; it does not block network access by your agent host, plugins, design app, or operating system.

Add a host and narrow tools

The host needs a task record, model adapter, context selector, tool registry, permission broker, cancellation token, and artifact store. Keep the runtime adapter replaceable so a later model comparison does not rewrite tool policy. A useful first tool is read_project_file(relative_path), restricted to a chosen disposable workspace. Resolve the final path before reading, reject paths outside the workspace, and deny symlink escapes. For stronger isolation, run the tool process with OS-level filesystem and network restrictions; path checks alone do not address races or compromised dependencies.

Introduce a run_build tool only with a fixed, reviewed command and a timeout. Capture exit code and bounded output, redact secrets from logs, and stop the process on cancellation. Avoid a general shell tool in the first prototype. If the model suggests git push, reads a private home directory, or sends a message, the broker should deny it regardless of the model's explanation. For actual product behavior, use the host's supported permission model and test the denial cases.

// Illustrative policy shape; the host must enforce it outside the model.
type Capability = {
  name: "read_project_file" | "run_build";
  workspaceId: string;
  effect: "read" | "local-write";
  timeoutMs: number;
};
​
type TaskRequest = {
  taskId: string;
  objective: string;
  allowedCapabilities: Capability[];
  maxSteps: number;
  deadlineMs: number;
};

Validate parsed inputs at runtime. workspaceId must map to a server-held approved root; never accept a caller-supplied absolute root as authorization. Keep credentials in an OS secret store or process environment outside the model context, source control, tool output, and logs. Never send an entire repository or home folder as default context.

Add plugins and design integration deliberately

For each plugin, document its purpose, publisher, version, exact commands, read/write scopes, network destinations, logs, and approval points. Test it in a disposable project with synthetic names. A design plugin might expose read_selected_frame first, then create_draft_frame after explicit write authorization. It should return an object ID and read-back properties so the host can verify that the intended editable object exists. Screenshots are evidence, but they do not prove the underlying design tree is editable.

When a design tool is cloud-backed, label that boundary in the UI and task record. A “local plugin” describes where the bridge runs, not where the document is stored. If strict offline work is required, use a truly local design artifact and turn off cloud connectors for that task. Review plugin updates before re-enabling them, because capability and network behavior can change.

Protect confidential data in practice

1. Classify before loading. Mark secrets, client files, personal notes, and regulated data as out of scope by default. Select a disposable workspace for the first trial.
2. Allowlist at the broker. Restrict readable paths, writable paths, commands, network destinations, and app documents. Deny broad home-directory, keychain, mail, and browser-profile access.
3. Minimize context. Retrieve only relevant snippets. Replace personal identifiers with synthetic values in examples and evaluation cases. Redaction is a second control, not a substitute for access limits.
4. Keep logs useful and safe. Store task IDs, tool names, timing, decisions, and test results locally. Avoid full prompts, file contents, tokens, or screenshots in default logs. Set retention and deletion rules.
5. Gate side effects. File edits need a diff and verification. Publishing, sending, purchasing, pushing, and cloud escalation need an explicit user authorization path appropriate to the action.
6. Verify with canaries. Put a synthetic marker in a denied fixture, request an out-of-scope read, and check that access is refused and the marker never appears in model context, logs, or outbound traffic. Inspect actual network connections during the run.

🟥 Never test exfiltration with a real password or personal document. A canary proves only the paths and configuration tested; repeat after tool, model, or plugin changes.

Milestones and evidence

This sequence makes the privacy claim testable. It also keeps model quality separate from tool security: a stronger model cannot repair a permissive broker.

Practice: find the leak

Prompt: The model runs on localhost. An agent plugin reads a Figma frame, writes verbose logs containing frame text, and sends diagnostics to its vendor. Can the product claim that all design data stays local?

Answer

No. The plugin's diagnostics and Figma's cloud storage cross the local boundary. Disable or replace the diagnostic path, verify network behavior, minimize/redact logs, and use a local design source if strict offline storage is required. Then retest with synthetic data before making a narrower, evidence-backed claim.