How to Pull Gigabytes of Message Histories from the Cloud and Build a Local Search Engine with DuckDB and SQLite

Think about how many times you've tried to find an old email with an attachment, a bug discussion in Slack from two years ago, or an important idea from a Discord DM. Usually it all ends with opening ten different browser tabs, digging through clunky corporate messenger searches, and frustration that Slack's free archive has once again cut off your message history.

I recently discovered msgvault. It's a Go command-line application that pulls all your conversations from a dozen sources onto your local disk, packages them into SQLite and Parquet, and on top of that provides a handy web interface, a terminal TUI, and an MCP server for neural networks.

Why Pull Message Histories onto Your Own Disk

The idea of pulling data out of third-party services isn't new. However, more often than not, backups turn into giant folders with JSON dumps or .eml that gather dust on your hard drive. Searching through them without a dedicated database is a separate ordeal.

msgvault replaces the traditional archive download approach. The utility downloads emails, chats, calendars, and meeting transcripts, then deploys a full-featured search analytics layer over them. The data stays with you. No paid subscriptions for keeping old threads or third-party cloud processors.

Here's what the utility can gather into a unified database:

• Gmail mailboxes (via OAuth or Service Account) and any standard IMAP server.
• Google Calendar (events, participants, and organizers are indexed alongside emails).
• Microsoft Teams and Slack corporate chats, including reactions, files, and nested threads.
• Discord servers via a built-in bot agent.
• Messengers from Beeper Desktop (gives access to iMessage, WhatsApp, Telegram, Signal, and SMS).
• Meeting recordings and transcripts from Granola and Circleback.
• Local MBOX dumps, Apple Mail archives (.emlx), PST files, and SMS backups from Android.

How Archiving Works Under the Hood

The project architecture was designed with large data volumes in mind. If you've accumulated half a million emails with images and documents over ten years, a regular hard drive will start lagging just from the number of files in a single directory.

The authors solved this with a combination of SQLite, DuckDB, and content matching by hashes.

Core text search runs through FTS5 in SQLite. Complex analytics and filtering across a massive message archive run on DuckDB on top of the columnar Parquet format. This lets heavy aggregation queries in the TUI execute in milliseconds.

Attachments aren't stored scattered around. msgvault deduplicates them by SHA-256 and packs them into isolated binary packs. This approach saves inodes on Linux and doesn't slow down the filesystem on Windows or NAS. If needed, these packs can always be unpacked back with maintenance commands.

Web Interface, Terminal, and Vector Search

After pulling in the data, you can work with it in three ways.

First option: the built-in web interface. The daemon spins up an HTTP server with an application that doesn't need external runtimes like Node.js. The interface is optimized for fast keyboard navigation. It's convenient for grouping emails by domain, tracking contacts, and viewing files.

Second path: TUI right in the console. It outputs detailed analytics across your entire message history. You can filter conversations from a specific year by a particular recipient in just a couple of keystrokes.

The third option is for those who want to search messages not just by exact words but by meaning. msgvault can connect to a local embedding server like Ollama, LM Studio, or llama.cpp.

After generating vectors, hybrid search kicks in. It combines classic BM25 full-text search and vector search via Reciprocal Rank Fusion. You enter a query like "where did we discuss changing database architecture," and the utility finds the relevant fragments even if the word "architecture" didn't appear in the text.

Connecting Neural Networks via MCP

For those using Claude Desktop or console AI agents, msgvault includes a built-in MCP server (Model Context Protocol).

Instead of manually copying long email threads into a chat window with a neural network, you give the agent access to the msgvault tool. After that, Claude can search for context in your past conversations, pull up old agreements, and cross-reference data from your calendar and Slack. All search queries execute locally on your machine.

Quick Start

The utility ships as a ready-made binary. On macOS and Linux, installation takes one command:

curl -fsSL https://msgvault.io/install.sh | bash

Homebrew users can install the package directly:

brew install msgvault

To get started, initialize the database and link a test Gmail account:

msgvault init-db
msgvault add-account [email protected]
msgvault sync-full [email protected] --limit 100
msgvault serve

The add-account command opens a browser for OAuth authorization. After downloading one hundred messages, msgvault serve starts a local web server with search.

If you need to run a background process for continuous scheduled synchronization, the utility can operate in daemon mode:

msgvault daemon start
msgvault daemon status

Things to Keep in Mind

The project is in active development (alpha version). The developers warn that storage formats and CLI flags may change between releases, so it's best to back up the database itself regularly.

The second nuance concerns initial setup. To connect Gmail or Google Calendar, you'll need to spend about five minutes creating your own OAuth client in Google Cloud Console. However, the project documentation has a detailed step-by-step guide.

Who Will Find This Project Useful

msgvault is useful for developers, privacy enthusiasts, and anyone whose work life is spread across dozens of cloud services. The tool solves the dependency problem on other platforms and turns scattered message archives into a unified, fast, local knowledge base.