GrooveSeek

Semantic search over a Markdown knowledge base, served over MCP.

View the Project on GitHub alphabet-h/grooveseek

Architecture

Source-level structure and data flow of groove, for contributors extending or modifying the codebase.

日本語版: ARCHITECTURE.ja.md

Source layout

File Responsibility
grooveseek/src/lib.rs (v0.7.1+) Library crate root. Re-exports the modules below as grooveseek::* so benches under benches/ and integration tests under tests/ can drive internal APIs without subprocess. The library surface is intentionally unstable and not intended for external consumers.
grooveseek/src/main.rs Binary entry point. The clap CLI dispatches every subcommand groove has; each is documented in usage.md, and the service verbs under its service section. Consumes the lib via use grooveseek::*;. Loads groove.toml and merges with CLI args. JSON / text output formatting.
grooveseek/src/config.rs 4-tier groove.toml discovery (--config flag → CWD → .git ancestor (CWD + up to 19 ancestors) → binary-side legacy). Config::discover() returns a ConfigSource enum that main.rs logs at startup. Resolves CLI > config > default precedence. Injects FASTEMBED_CACHE_DIR env when the config sets it and the env is unset.
grooveseek/src/server.rs rmcp::ServerHandler impl. Dispatches six MCP tools. search routes to db.search_hybrid and wraps the result in a SearchResponse with low_confidence / match_spans / filter_applied (BREAKING in v0.3.0; see CHANGELOG). State lives in KbCore; each tool is a thin async wrapper that runs its body there via spawn_blocking, so a long call occupies a blocking thread instead of an async worker.
grooveseek/src/server/search.rs (v1.0.0+) The search half: the search tool body, the pipeline it runs, and the limits a request is held to before either of them sees it. Split out of server.rs the way db.rs was split before it — bodies byte-identical and in the order they were already in, mod tests left in the parent. The only thing that changed was visibility: three private items became pub(super) because the parent still calls or names them. What stayed behind is the tool surface itself, the #[tool_router] / #[tool_handler] impls and the parameter and response types.
grooveseek/src/server/documents.rs (v1.0.0+) Reading a document out of the knowledge base, and deciding whether it may be read at all: the get_document and get_best_practice bodies, the staged path check they go through, and the size limits they are held to. Split from server.rs on the same terms as server/search.rs; every item that gained pub(super) did so because cargo check named it after a move made with no visibility changes at all. build_document_response parses with parse_bytes_for_read, not parse_bytes, so a read keeps the built-in decompression budget whatever [index] sets (feature-61).
grooveseek/src/server/kb_uri.rs (v1.0.0+) The corpus side of the kb:// resource surface: which documents this server will hand over, the URIs it advertises for them, and what a resources/read gets back. resources.rs below is the URI side — it builds kb:// strings and takes them apart, and knows nothing about the corpus; this is the half that knows it. Split from server.rs on the same terms as the other two.
grooveseek/src/prompts.rs (v0.22.0+) The MCP prompts surface: summarize_topic, deep_dive, whats_new, find_gaps, declared with rmcp’s #[prompt] / #[prompt_router] and dispatched by the #[prompt_handler] on server.rs’s ServerHandler. Each handler only calls the free function beneath it, so the tests exercise the production text without constructing a KbServer (which owns a Database and an Embedder) — the same split, for the same reason, as watcher::should_process_parts. Text-only messages: a prompt that embedded a resource would oblige the server to implement the resources capability as well. Fixed at compile time rather than configurable, because prompt text reaches the model and groove.toml is discovered, which would put a [prompts] section in the same privileged category as kb_path under restrict_untrusted. The router is generated with vis = "pub" so the handler in server.rs can reach it across the module boundary.
grooveseek/src/resources.rs (v0.22.0+) The MCP resources surface: the kb:// URI codec and the grouping behind resources/list, as pure functions with no database in sight so the encoding rules are testable directly. kb://topic/<prefix> names a topic group — the first one or two path segments, the same derivation the indexer uses for category / topic, so a group and the database agree without a second query to keep in step — and kb://doc/<path> names one document. Groups are built from KbCore::servable_document_paths() — the indexed paths minus those ServableRules will not hand over — and a read is checked against that same query, so a URI the listing offers cannot be refused when it is read back (ADR-0004). ServableRules (in server.rs) is the single predicate behind that query and the uri on a search hit: an extension the active parser registry can no longer open, or a recorded size past the cap a read would apply (ADR-0005). Separators stay / and everything else is percent-encoded; the traversal check runs after decoding, because %2e%2e%2f is ../ and a check that ran first would not see it. Tested against paths with non-ASCII characters, spaces and % even though the corpus this was built against contains none — an encoder that never fires is an encoder nobody has checked. doc_is_addressable is the one predicate for a name the index can hold: the index walk, the watcher, groove doctor, get_document’s lexical check and the URI side all ask it (ADR-0023).
grooveseek/src/schema_compat.rs (v0.14.0+) Normalises the JSON Schema advertised for those tools before it leaves the server. schemars derives Option<T> as a union type ({"type": ["string", "null"]}) and stamps Rust integer widths as format: uint32; both are legal JSON Schema 2020-12, but strict tool-calling runtimes reject the union and do not know the format. Unrelated to schema.rs, which validates document frontmatter — the names are close, the subjects are not.
grooveseek/src/service/ (v0.8.0+) Cross-platform OS user service installer. mod.rs (= ServiceBackend trait + InstallContext + ServiceState), install.rs / uninstall.rs / status.rs (= orchestration), linux.rs / macos.rs / windows.rs (= per-OS backends, cfg-gated), plus two modules deliberately kept outside the cfg gates so they compile and are tested on every OS leg: render.rs (v0.14.0+, the unit / plist templates and their escaping — a plist bug used to be detectable only on the macOS runner) and powershell.rs (v0.14.0+, the UTF-8 output prelude and the strict / diagnostic decoders for what powershell.exe writes back). Phase 1 = user-level only (= no admin/sudo, Linux systemd-user / macOS LaunchAgent / Windows Task Scheduler AT_LOGON). groove service install self-registers using Rust crates only (= no NSSM / WiX / 3rd-party tooling). The Windows backend (v0.8.3+) invokes PowerShell’s Register-ScheduledTask -Action -Trigger -Settings cmdlet via Command::new("powershell") — schtasks /Create /XML was abandoned across v0.8.0 → v0.8.3 due to layered locale / elevation / Principal issues documented in .dev/knowledge/windows-task-scheduler-pitfalls.md.
grooveseek/src/indexer.rs walkdir-based file scan using Registry::extensions(). All read paths (initial scan, index_single_disk_entry, reindex_single_file, rename_single_file) go through links::read_checked (v0.20.0+; bytes, not read_to_string), which opens the file once and decides from that handle whether its bytes may be used before reading them — and hash the raw bytes with SHA-256 for content-hash diff detection; for existing UTF-8 KBs this is a no-op vs. the old string hash. Parses via the Parser trait (parse_bytes), embeds, stores. Per-file skip isolation: a read failure, a size-cap breach, or a parse_bytes error skips just that file (logged as a warning) instead of aborting the whole run, and the skipped path is retained in the index rather than pruned as deleted. A refusal (SingleResult::Refused) is kept distinct from those: it means the object opened was not the file that was collected, and rename_single_file maps it to its own outcomes rather than reporting a successful rename. Incremental APIs (reindex_single_file / deindex_single_file / rename_single_file) shared with the file watcher.
grooveseek/src/indexer/progress.rs (v0.7.8+) ProgressReporter + ProgressMode enum. Drives per-file output for groove index: Verbose (default) / Quiet (--quiet) / Auto (--progress, TTY = indicatif::ProgressBar, non-TTY = periodic Progress: N/M (P%) lines). MCP server rebuild_index tool wires Quiet directly. Bar lifetime is closed inside rebuild_index (lazy init via start_indexing(total)) so Backfilled / Found lines stay plain eprintln!. (v1.13.0+) ProgressReporter::with_callback takes a boxed Fn(ProgressEvent<'_>) + Send (ProgressCallback) instead, so an application embedding this crate receives Started / Indexed / Unchanged / Renamed / Deleted / Finished as values rather than lines. What that silences is the reporter: rebuild_index writes its own diagnostics — the scan-time Skipping ... warnings, the Found N source files line, the backfill line and the summary lines — to stderr whatever reporter it was handed, so an embedding application whose stderr must stay quiet has to capture or redirect it. No CLI flag selects it. (v1.14.0+) A CancelToken (a cloneable Arc<AtomicBool> flag) goes on a reporter of any mode through ProgressReporter::with_cancel. rebuild_index looks at it after each scanned file, before applying renames, before each document (except a rename’s forced re-parse, which it processes first) and once before the deletion sweep; on a set token it stops there, emits Cancelled { done, total } in place of Finished, and returns Ok with IndexResult::cancelled. Cancelled.done counts documents reported, never scanned files. The index it leaves is consistent per document: rows of files deleted from disk, the frontmatter policy and the declared-field set wait for the next completed run (field filters are refused until then), and the run leaves no declared-field pass of its own open. After a stopped force run the index holds only what was committed, and the next incremental run adds the rest; with no schema, that run also records the empty declared-field set [], so no second --force is needed. A cancel that arrives once the deletion sweep has begun is too late, and the run completes. The scan also reports Scanning { done, total } once per file, counting the files it declines, so done reaches total; --progress on a terminal shows it as the bar’s scanning N/M message.
grooveseek/src/parser/ Parser trait + Registry. mod.rs (Frontmatter / Chunk / ParsedDocument / TableRowSpan (feature-66: where each .docx / .pptx table row sits in a chunk’s content, in bytes; table_rows_fit is the one check that spans can be sliced), plus the parse_bytes(bytes, path_hint, exclude_headings) -> Result<ParsedDocument> entry point that every call site — indexer and server — now goes through. parse_bytes lives on the ParserExt extension trait with a blanket impl<T: Parser + ?Sized>, so no parser can define it: it always delegates to Parser::parse_bytes_inner (same signature) inside a catch_unwind, and a panic anywhere in a parser or its dependencies becomes a per-file Err instead of aborting the whole index run. A default method would not do — it can be overridden, and an override silently bypasses the guard. parse_bytes_inner’s default impl validates UTF-8 then delegates to parse, so md/txt need no override, while binary-format parsers override it directly. is_binary() (default false) flags binary parsers for get_document’s size-cap classification and for the quality filter’s QualityProfile::Binary. MAX_RAW_BINARY_BYTES = 50 MiB and MAX_RAW_TEXT_BYTES = 50 MiB (text, since v0.17.0) are the defaults of the indexer’s raw-byte caps, and DEFAULT_MAX_DECOMPRESSED_BYTES = 50 MiB the default decompression budget; FileSizeLimit / FileSizeLimits carry the values [index] sets (a byte count, a size with a unit, or "unlimited"), so no format is read into memory unbounded unless a trusted config says so. The read side does not share them: get_document and resources/read use GET_DOCUMENT_BINARY_MAX_BYTES in server/documents.rs, and parse through ParserExt::parse_bytes_for_read (the same panic guard, delegating to Parser::parse_bytes_for_read_inner), which the four binary parsers override to pass DEFAULT_MAX_DECOMPRESSED_BYTES instead of the budget they were built with), markdown.rs, txt.rs, pdf.rs (v0.10.0+, see below), ooxml.rs / xlsx.rs / docx.rs / pptx.rs (v0.11.0+, see below), panic_guard.rs (see below), registry.rs (extension lookup, binary_extensions(), and limits(), the [index] size caps the registry was built with: the binary parsers are constructed with with_budget(limits.decompressed) and the indexer reads the raw caps from it).
grooveseek/src/parser/code/ (v1.2.0+) Source code parsed into one chunk per definition, opt-in via [parsers].enabled = ["md", "rs"]. mod.rs holds the language-neutral half: LoadedGrammar (a grammar plus the tags query validated against it), CodeParser (one instance per extension, is_binary() == false, since that flag decides get_document’s size cap; the quality filter exempts a definition through QualityProfile::Definition instead (v1.4.0+, ADR-0015)), and the chunker. Definitions come from the grammar’s own tags.scm; a definition’s range extends backwards over an unbroken run of comment nodes above it, so a doc comment cannot land in both its definition’s chunk and the gap chunk above; an oversized definition recurses into nested definitions and splits by lines when it has none (the common path, since methods hold no nested definitions); every byte no definition covers becomes a heading-less gap chunk, with fragments under the quality filter’s short-content threshold dropped unless the file would otherwise produce nothing. Scope names come from walking Node::parent() for a name / type / trait field rather than from the definition tree, because Rust’s tags query captures impl blocks as references. symbol_kind carries the tags vocabulary verbatim (class covers struct / enum / union) rather than a language keyword, which is what keeps a grammar to data. MAX_RAW_CODE_BYTES = 1 MiB, compared with > so a file of exactly the cap parses as well as it reads — tree-sitter’s allocator aborts on OOM instead of unwinding, so panic_guard cannot catch it and the file has to be refused before the parser sees it. static_rust.rs holds the compiled-in Rust grammar behind the default-on grammar-rust feature. plugin.rs (v1.3.0+) is the other arrival path: it dlopens a library the user placed and checks it — exports present, groove’s own ABI version, the grammar’s tree-sitter ABI against the range this runtime speaks, the tags query compiling, and a single valid extension — before handing back the same LoadedGrammar. It opens a file named from a fixed id-to-library table rather than enumerating the directory, because opening a library runs its initialisers before any symbol can be inspected; an accepted library is then deliberately leaked, since the parse table and the tags query live inside it. See ADR-0012 and ADR-0013.
grooveseek/src/parser/panic_guard.rs The panic-isolation machinery behind ParserExt::parse_bytes: catch_parser_panic runs the parser under catch_unwind and turns a panic into Err("<path>: <id> parser panicked: <payload>"), keeping the payload so the indexer’s skip line still says what happened. A wrapper panic hook is installed once (never swapped) and consults a thread-local flag set by an RAII guard, so only the parsing thread’s own backtrace is suppressed while panics on other threads keep reporting — a hook that is swapped per call races when two threads parse at once. This started as PDF-only (v0.10.0) and moved here so docx/xlsx/pptx are covered too; without it a single crafted spreadsheet aborts groove index outright (calamine’s get_dimension subtracts unchecked, so ref="B2:A1" panics in any build with debug assertions).
grooveseek/src/parser/pdf.rs (v0.10.0+) PdfParser, is_binary() == true, opt-in via [parsers].enabled = ["md", "pdf"]. Extracts text page-by-page with oxidize-pdf (PdfReader + PdfDocument::extract_text), one non-empty page per chunk (heading p.N, level: None). Title / CreationDate PDF metadata become frontmatter, falling back to a filename-derived title when Title is absent. A malformed PDF that panics inside the crate’s parser internals degrades to a per-file Err (indexer skip-and-warn) instead of aborting the whole index run — that catch_unwind used to live here, and now sits in Parser::parse_bytes / parser/panic_guard.rs where every parser gets it. Extracted pages pass two gates, in this order (reject_unindexable_pages). First, mojibake detection (v0.15.1+), via two complementary signals: text whose C1 control codes (U+0080–U+009F) reach 1% of all characters is rejected — reading UTF-16BE one byte at a time lands the high bytes of U+8000–U+9FFF (and the low bytes of voiced kana) in C1, which correctly decoded text never contains, measured 0.00% across six correctly-extracted samples versus 3.61–15.59% across four mis-decoded ones — and text carrying the byte-pair signature of the same mis-decoding (≥30% of characters) is rejected, which covers the one shape that emits no C1 at all: unvoiced-kana-only text, whose high bytes are all 0x30, comes out as pure ASCII (あいうえお… → 0B0D0F…, 0.00% C1 at 407 chars/page, measured on oxidize-pdf 4.1.1 — the pin at the time; the pin since v0.15.2 is 4.3.0, which extracts this form correctly, leaving the gates as defense-in-depth). Long runs are judged per run (the leading parity near-constant, the other varied — the mirror orientation, alternating identifiers like 1A2A3A, is not produced by bytewise decoding and is not flagged); runs of two to seven characters — too short for the per-run judgment to be able to fire; a label sheet or word list splits into 4-char tokens (measured 148 chars/page) — are aggregated document-wide and judged as a pool. The mojibake gate runs first because mis-decoding turns one character into two, so mojibake clears the density gate below (1052 chars/page measured) while correctly extracted sparse text (29 chars/page) does not. Second, a PDF yielding under 50 chars/page on average is rejected. That heuristic was written for scanned/image-only PDFs (no text layer, no OCR support), but it is not specific to them: a PDF that decodes perfectly but genuinely carries little text per page — a cover sheet, a label, a figure-heavy deck — lands here too. The threshold is not lowered, because a scan carrying only digitally-added page numbers and a “CONFIDENTIAL” stamp measures 39 chars/page: a character count cannot separate worthless boilerplate from sparse real content. The diagnostic reports the measurement and offers common causes as an open list, rather than asserting “scanned” or enumerating a closed set. Japanese PDFs embedding a TrueType subset with /ToUnicode (Word / LibreOffice / Google Docs output) extract correctly; the historical mojibake case was a CID-keyed font with a predefined CMap and no /ToUnicode, whose root cause was upstream — oxidize-pdf read /DescendantFonts only when the CIDFont was an indirect reference, not when it was a direct dictionary. Reported and fixed by this project (bzsanti/oxidizePdf#469, shipped in oxidize-pdf 4.3.0, the pin since v0.15.2), so that form now extracts correctly too; the mojibake gates stay as defense-in-depth against decode failures from other causes. encrypted PDFs surface as an Err from PdfReader::new / extract_text (oxidize-pdf’s ParseResult-based error design, no password support). Post-processing joins conservative line-end hyphenation (-\n only when both neighbors are ASCII lowercase) and normalizes common ligatures (fi/fl/ff/ffi/ffl).
grooveseek/src/parser/ooxml.rs (v0.11.0+) Shared OOXML zip/XML helper consumed by xlsx.rs / docx.rs / pptx.rs (no parser struct of its own). row_line (feature-64 / feature-65) writes one table row as one line of a chunk body for docx.rs and pptx.rs, so the two formats agree on what a row looks like. push_row_line (feature-66) adds that line to a body and records where it went as a RawRowSpan, under the table’s number as read, and chunk_table_rows turns a chunk’s raw spans into the TableRowSpans of Chunk::table_rows, numbering the tables by first appearance in the chunk. read_zip_part reads one zip part as raw bytes against the parser’s decompression budget and names the part and [index].max_decompressed_size on stderr when one part alone is over it (the fixed-cap read_zip_entry and read_zip_entry_capped exist for tests only); read_optional_zip_part (feature-62) reads a part the document can do without, word/styles.xml, through the same body, but leaves it out with a warning instead of failing the document when it would take the document past the budget, and inflates it only up to what is left; core_xml_frontmatter / parse_core_xml map docProps/core.xml (Dublin Core: dc:title / dcterms:created or modified / cp:keywords) to Frontmatter, falling back to a filename-derived title when the part is missing or title is empty; local_name_pub strips a namespace prefix from a QName (cp:title → title) so element matching is prefix-agnostic; resolve_general_ref resolves quick-xml’s Event::GeneralRef (entity references such as &amp; arrive as a separate event, not folded into Event::Text; verified against the pinned 0.41) — factored out here because docx.rs and pptx.rs both need the same char-ref/named-entity handling.
grooveseek/src/parser/xlsx.rs (v0.11.0+) XlsxParser (.xlsx), is_binary() == true. XlsParser (.xls) still exists here but is not reachable from the registry since v0.14.0 (AU-06): calamine builds one dense cell grid per sheet while Xls::new runs, and BIFF bounds a sheet (65,536 x 256 = 512 MB of Data) but not a workbook, so a small file declaring many maximal sheets exhausts memory before groove regains control — and an allocation failure aborts the process instead of skipping the file. Bounding it would mean reading BOUNDSHEET / DIMENSIONS out of the CFB container ourselves, before Xls::new; deferred until someone asks for .xls. Both share one parse_workbook_bytes entry point that dispatches per format. Each opens the reader matching the registered extension (calamine::Xlsx / calamine::Xls) rather than probing formats, so a payload whose real container disagrees with its extension is rejected instead of parsed. .xlsx first runs a decompression preflight: every entry is checked against the parser’s decompression budget ([index].max_decompressed_size, default 50 MiB, independent of the raw-byte cap) both by its declared uncompressed size and by decompressing it for real (output discarded, bounded at the remaining budget + 1 byte), so the invariant needs no reference to naming — an archive may decompress to at most the cap, in total. The second layer is what stops a zip bomb, since the ZIP format does not enforce the declared size and zip 8.6 does not bound deflate output by it (measured: a 101 KB crafted workbook declaring 10 bytes expanded to 100 MB). Checking every entry rather than a suffix list is deliberate: calamine resolves parts through relationship Targets and ignores filenames, so xl/worksheets/payload is read like any worksheet — suffix-based selection was bypassed three times (fixed paths, then missing .rels, then the suffix assumption itself). The trade-off is that xl/media/ images count too, so a workbook near the raw cap that is mostly pictures can be skipped. Then one chunk per non-empty sheet (heading Sheet: <name>, tab-joined cell text per row), truncated at SHEET_MAX_BYTES (1 MiB) with row-aligned truncation semantics (the row that pushes the running total past the cap is still emitted whole, then extraction for that sheet stops — never cuts mid-row). Frontmatter comes from docProps/core.xml via ooxml::core_xml_frontmatter when the bytes open as a zip (true for .xlsx).
grooveseek/src/parser/docx.rs (v0.11.0+) DocxParser, is_binary() == true. Reads word/document.xml paragraph-by-paragraph (<w:p>), treating a heading paragraph as a section boundary — the same heading-hierarchy chunking rule markdown.rs uses for Markdown headings, including exclude_headings support (body under an excluded heading is dropped until the next non-excluded heading). Whether a paragraph is a heading is decided by a StyleTable read from word/styles.xml after docProps/core.xml (feature-62): the paragraph’s <w:pStyle w:val> is looked up by style ID, with entities resolved and compared byte for byte, and the style’s name (heading N), its w:basedOn chain (at most 16 styles) and an outline level of 9 set before the heading ancestor decide. heading_digits is the one reading of heading N (heading_number is the same with 0 left out), for a style name and for a style ID’s spelling, which decides alone for an ID the table does not define and for every paragraph of a document whose styles part is missing, unusable or over the budget (read through ooxml::read_optional_zip_part, so a document near the budget is still indexed). A table (<w:tbl>) is written one row per line: the text of each cell, its paragraphs and line breaks joined with a space, is joined to the next cell’s with a tab, an empty cell keeps its place as an empty field, and a row whose cells are all empty adds nothing. Outside <w:tc> the ordinary <w:p> boundary handling is unchanged, a heading paragraph inside a cell still opens a section (the row is cut where it stands), and a nested table flushes the outer row before its own rows, so the text stays in document order. A section whose body is empty once trimmed is not a chunk: fold_empty_heading_sections folds its heading’s text, one line per heading, into the start of the next section that has a body, drops headings left over at the end, and keeps a chunk per heading when no section has a body (feature-63). Frontmatter via ooxml::core_xml_frontmatter.
grooveseek/src/parser/pptx.rs (v0.11.0+) PptxParser, is_binary() == true. Collects ppt/slides/slideN.xml entries and sorts by the numeric slide index (not zip iteration order), emitting one chunk per slide (heading Slide N: <title> when a ctrTitle/title placeholder shape has text, else bare Slide N). A table (<a:tbl>) is written one row per line: the paragraphs and line breaks of a cell are joined with a space, the cells of a row with a tab through ooxml::row_line, an empty cell keeps its place as an empty field and a row whose cells are all empty adds nothing; a cell a merge covers from the left (hMerge) adds no field and one covered from above (vMerge) is an empty field. A line break (<a:br>) outside a table is a newline, and a space in the title the heading takes. The chunk body is trimmed of every whitespace but the tab, so a table row at either end of the slide keeps its empty fields. Speaker notes are appended as a trailing [notes] section, resolved by reading the slide’s ppt/slides/_rels/slideN.xml.rels for a notesSlide relationship Target — deliberately not a same-numbered-file heuristic (slideN.xml ↔ notesSlideN.xml), which a dry-run (plan Task 3.7) showed can misattribute notes to the wrong slide when slide/notes numbering diverges after edits. Frontmatter via ooxml::core_xml_frontmatter.
grooveseek/src/doctor.rs (v0.23.0+) The checks behind groove doctor. Three groups. Integrity: whether chunks, vec_chunks and fts_chunks still agree about every chunk — a disagreement raises no error, it silently removes results from one half of the hybrid search, which is why backfill_fts exists at all. Servability: which indexed documents the resource surface withholds, computed by calling paths_with_unregistered_extension and ServableRules rather than reimplementing them, so the report cannot drift from what the server actually offers. What the chunker gave up on: which source files were chunked by lines rather than at their definitions, read back from the tags the code parser writes when it declines — whole and searchable, but carrying no symbol kind, heading or scope. Provenance comes from chunks.start_line, not from the tags, since those are frontmatter a note can declare; and an index written before the policy that produces them says so instead of answering, because a file whose content has not changed is never re-chunked. It reports and never repairs — the contract paths_with_unregistered_extension already states — and each finding carries what fixes it, which for that third group is a change to the file, since an index run reaches the same bound again. The SQL lives in db/meta.rs because Database::conn is private to that module, which also puts the queries next to backfill_fts, their nearest relative.
grooveseek/src/exclusion.rs (v0.21.0+) The one place that decides whether a knowledge-base path is out of scope, for all three surfaces that walk or watch it: the full index walk, the validate walk (which lives in the binary target and reaches this through the library’s public API) and the live watcher. ExclusionRules::load combines the built-in .git / .svn / node_modules fail-safe, exclude_dirs, and <kb_path>/.grooveignore — root only, no .gitignore, case_insensitive(true) set before any pattern is added because it does not apply retroactively. is_excluded(rel, is_dir) tests every ancestor as a directory first and stops at the first exclusion, which is what makes it agree with a walk that prunes: git’s rule is that a file under an ignored directory cannot be re-included with !. Only the matcher from the ignore crate is used and walkdir stays, because WalkBuilder defaults hidden() to true (on Windows that means dot-prefixed or FILE_ATTRIBUTE_HIDDEN), resolves add_ignore against the process cwd rather than the walk root, and makes .gitignore handling depend on require_git. matched_path_or_any_parents is deliberately not used: measured, it answers Whitelist for a file a walk would never reach, which would have put the drift back at the level of which API was called. The file itself is read through links::read_checked, capped at 64 KiB and 1000 patterns, with a leading BOM stripped; anything unreadable warns and is left out rather than stopping the run. The boundary is the index, not access — see validate_get_document_path.
grooveseek/src/links.rs (v0.19.0+) Hard-link detection for the three places a file can enter the index or leave it as content (the full index, the watcher, get_document), which already refuse symlinks. hard_link_count reads nlink from symlink_metadata on Unix and GetFileInformationByHandle on Windows — MetadataExt::number_of_links is still unstable there, and walkdir’s WIN32_FIND_DATAW metadata carries no link count — and is_multiply_linked fails open, since the check also gates deindexing and a deleted file has no link count. Checked after the extension filter at every call site: the Windows path needs the file opened, and on Linux every directory has a link count of at least two. (v0.20.0+) read_checked is the second entry point and the one the content actually passes through: it opens the file once and takes the link count, the file type and the size cap from that single fstat before reading the bytes from the same descriptor, so a hard link renamed over a collected path after the walk checked it is refused rather than read. On Unix the open adds O_NOFOLLOW and O_NONBLOCK (a symlink swapped in is refused; a FIFO cannot block the run); Windows adds neither, because symlink creation there needs administrator privilege and refusing reparse points would refuse OneDrive placeholders. The module doc states what the check does not do (link-then-unlink, intermediate-directory symlinks, and filesystems that always report a count of 1) and why the knowledge base directory is not a security boundary.
grooveseek/src/poison.rs (v0.19.0+) Recovering from a poisoned mutex instead of inheriting the panic. recover / recover_try take the LockResult rather than the mutex, so the literal .lock() stays at the call site where server’s meta-test scans for it; recover_db additionally rolls back a transaction a failed Drop-time ROLLBACK left open, since rusqlite swallows that error and every later &self write would join a transaction nobody commits. The first recovery in a process warns and names the mutex, later ones are debug — poisoning is sticky, so a per-recovery warning repeats for every request.
grooveseek/src/markdown.rs Thin shim over crate::parser::markdown::MarkdownParser, retained for legacy parse() / parse_with_excludes() callers.
grooveseek/src/watcher.rs notify-debouncer-full bridged to a tokio channel. Filters by extension and path — the path half through exclusion::ExclusionRules, the same object the index walk uses — then dispatches to indexer::{reindex,deindex,rename}_single_file. Runs alongside the MCP server via tokio::spawn. (v0.21.0+) A batch touching <kb_path>/.grooveignore rebuilds those rules before anything is classified, since the file has no registered extension and would otherwise be dropped by the same filter it is meant to change; run_watch_loop owns the state, so the reload needs &mut and no lock.
grooveseek/src/transport/ MCP transport abstraction. mod.rs (Transport enum + CLI/config resolution), stdio.rs (stdio), http.rs (rmcp StreamableHttpService + axum, mounts /mcp and /healthz; v0.8.0+ also mounts an admin sub-router with /ui + /api/admin/status, wrapped in admin_security_headers (CSP + nosniff, outermost so refusals carry them too). One dns_rebinding_gate serves every route that validates (ADR-0009): peer, then Host, then Origin, each group handed its own DnsRebindingGate state — /mcp the effective host and origin lists, the admin routes allowed_admin_hosts plus the same origin list plus the loopback-peer requirement, and /healthz the host list alone, and only when healthz_public = false (the default mounts it with no gate). rmcp’s own checks are given empty lists on purpose, so /mcp is validated here too — outside the session gate, i.e. before admission. /api/search was removed in v0.27.0; /ui searches through /mcp now). systemd_fd.rs (Linux only — macOS answers ENOPROTOOPT to getsockopt(SO_ACCEPTCONN), so the listening check could never pass there, and ADR-0021 records why the target narrowed rather than the check: the listening socket a service manager passed, checked against sd_listen_fds(3)’s rules before it is adopted — LISTEN_PID against our own pid first, since a variable inherited from a parent otherwise hands over somebody else’s descriptor; LISTEN_FDS at exactly one; SO_TYPE before SO_ACCEPTCONN, so a datagram socket is reported for its type rather than for not listening; FD_CLOEXEC and O_NONBLOCK on what is kept. Adopted once per process, which is what replaces unsetting the variables — std::env::remove_var is unsafe in edition 2024). KbServerShared is Arc-shared through a session factory so each connection gets a lightweight handle.
grooveseek/src/transport/webui_index.html (v0.8.0+) The operator view served at /ui, embedded via include_str! in transport/http.rs::ui_index. Raw HTML + JS, no CSS framework, no external requests, XSS-safe via textContent / createElement only (= no innerHTML). A status band over a search box; the search goes through /mcp, so the page is also the smallest example of an MCP client over Streamable HTTP. Browsing a knowledge base is expected to move to MCP clients during 1.x — see stability.md.
crates/groove-tray/ (v0.9.0+) Windows-only system tray binary (groove-tray.exe, GUI subsystem) for daemon monitoring + lifecycle control. Polls /api/admin/status every 5s and renders a 4-state status dot (green / yellow indexing / red 1min+ down / gray polling-pending), right-click menu with 6 items (Status / Open Web UI / Start / Stop / Restart / Quit Tray). Start goes through PowerShell Start-ScheduledTask (= same path as grooveseek/src/service/windows.rs); stop does not — since v0.14.0 the tray reads the daemon’s pid from /api/admin/status and terminates that process through the Win32 API (OpenProcess once, then image-name check and TerminateProcess on the same handle, so a recycled pid cannot be hit). Stop-ScheduledTask still runs in both branches, but never as the deciding step: after a pid stop it is best-effort cleanup of the task instance a pre-v0.9.1 install would have (failures logged and ignored), and when the probe yields no usable pid — including an unreachable status endpoint — it is the fallback attempt whose error is deferred rather than returned. Either way success is decided by binding the daemon’s configured address, never by the mechanism’s own return value. Dual event loop: tao on the main thread, tokio runtime on a dedicated thread, bridged via EventLoopProxy::send_event. Panic hook + tracing-appender::rolling::daily write logs to %LOCALAPPDATA%\groove\logs\tray.YYYY-MM-DD. Library API (install::install_autostart / uninstall_autostart) is invoked by groove service install --with-tray / service uninstall / service tray-install / service tray-uninstall to manage the shell:startup .lnk shortcut via PowerShell WScript.Shell COM. cargo-dist publishes groove-tray.exe only for x86_64-pc-windows-msvc.
crates/groove-svc/ (v0.9.1+) Windows-only launcher binary (groove-svc.exe) that exists solely to start groove.exe serve without a console window. groove.exe is a console-subsystem binary, and Windows allocates a conhost window for it before the process runs, so hiding it in-process only removes it after a ~1 s flash (unfixed platform behavior, microsoft/terminal#249). This crate is windows_subsystem = "windows", so it has no console to pass on — but that alone is not enough: a console-subsystem child with no inheritable console calls AllocConsole() itself and gets a fresh visible window. The spawn therefore must pass CREATE_NO_WINDOW (0x0800_0000), which tells CreateProcess to skip console allocation for the child; that flag plus the windows-subsystem parent is what yields a 0-flash launch. It detach-spawns groove.exe with stdio nulled and exits. Since v0.9.1 the Task Scheduler Action points here instead of at groove.exe. child_args prepends serve unconditionally, which is why resolve_action_target in grooveseek/src/service/windows.rs leaves the Action’s -Argument clause empty — both halves of that invariant are unit-tested because a mismatch only surfaces at the next logon. Non-Windows builds compile to a fail-fast stub so the workspace builds everywhere; cargo-dist ships the binary only for x86_64-pc-windows-msvc. Consequence for the tray: the scheduled task’s own process exits immediately, which is why stopping the daemon cannot go through Stop-ScheduledTask (see the tray row).
crates/groove-grammar-abi/ (v1.2.0+) The contract between groove and a tree-sitter grammar. GrammarDescriptor carries the language name, the single file extension it claims, the parse table and the tags query; ABI_VERSION is groove’s own contract number, distinct from the grammar’s tree-sitter ABI, and is what a plugin declares so the loader can refuse a mismatch. The compiled-in Rust grammar builds a descriptor directly; a grammar shipped as a separate dynamic library builds the same one and exports it across a C ABI (the loader arrives in v1.3.0). Deliberately holds no logic — a grammar contributes data, and everything done with that data lives on groove’s side, which is what keeps adding a language from needing per-language code. #![forbid(unsafe_code)], publish = false, not shipped as a binary. See ADR-0013.
crates/groove-grammar-python/ (v1.3.0+) The Python grammar as a loadable library — the first grammar groove does not compile in. src/lib.rs is one invocation of groove_grammar_abi::groove_grammar_plugin!, naming the language (python), the extension it claims (py), and the parse table and tags query it takes from tree-sitter-python; what the C ABI needs lives in the macro, so a second grammar is a manifest and one macro call rather than an FFI surface to review again. Built as a cdylib, so cargo produces groove_grammar_python with the platform’s library decoration — the name parser/code/plugin.rs builds when it looks the file up. cargo-dist distributes it as its own app: dist = true alone is not enough for a package with no [[bin]], so the manifest also carries package-libraries = ["cdylib"] and cdylibs = [...], and it deliberately omits targets so the workspace’s four platforms are inherited rather than narrowed the way the two Windows-only binaries narrow theirs. The archive is named after the package, which is why the package name and the library name have to stay one name in cargo’s two spellings.
crates/groove-grammar-php/ (v1.5.0+) The PHP grammar as a loadable library, built the way the Python one is — src/lib.rs is one groove_grammar_abi::groove_grammar_plugin! invocation, and the manifest carries the same cdylib and cargo-dist keys for the same reasons. What differs is the two choices the macro is handed. The language name and the extension are one word here (php) where Python’s are two (python and py), so a reader who learned the pair from that crate does not find it repeated. And tree-sitter-php publishes two grammars: LANGUAGE_PHP, taken here, reads a file that may open in HTML and switch at <?php, which is the one upstream declares file-types: php on; LANGUAGE_PHP_ONLY reads a file that is code from its first byte. The tags query that travels with it captures namespaces, classes, interfaces, traits (as interface), functions, methods and properties — and no constants, so a const is filled in by line, the same shape Rust’s query has.
grooveseek/src/schema.rs Frontmatter schema validation. Reads groove-schema.toml under kb_path; [fields.<name>] takes any name (v1.9.0+) and enforces required / type / pattern / enum / min_length / max_length / allow_empty (pattern and enum apply to a string’s value or to each element of an array). The five keys the parser stores in their own fields are read from there; every other key comes from Frontmatter.extra, where it is a string, a list of strings, or an opaque shape that satisfies required alone (a key written with no value counts as absent instead). [options].allow_unknown_fields = false (or groove validate --strict) makes a key the schema does not name an undeclared_field violation. A document whose YAML the parser refused is one frontmatter_unparsed violation and is not held against the schema. Invoked by the groove validate CLI which reports in text / JSON / GitHub-annotation formats.
grooveseek/src/embedder.rs Provider-neutral embedding boundary (Embedder over the private EmbeddingProvider trait) with FastEmbed as its default implementation and an opt-in OpenAI-compatible HTTP implementation. EmbeddingSettings holds the resolved provider settings and the index identity (see ADR-0022). ModelChoice selects BGE-small-en-v1.5 / BGE-M3, while RerankerChoice + Reranker provide optional local cross-encoder reranking.
grooveseek/src/db.rs rusqlite + sqlite-vec + FTS5 (trigram). Manages the chunks / vec_chunks / fts_chunks / document_fields (feature-58: the declared keys’ values, one row per value) schemas and CRUD. Exposes search_hybrid (Reciprocal Rank Fusion; the constant and the bm25 column weights are configurable via [search.fusion] since v0.13.0, defaults k = 60 and 2.0 / 1.0 / 1.0) and the v0.7.0 unbounded variants for the MMR / parent retriever pipeline. SearchFilters struct unifies filter args (path globs / tags / date range / min_quality); MatchSpan carries byte-offset citations (added in v0.3.0). chunks.level (added v0.7.0) distinguishes h2 / h3 headings.
grooveseek/src/db/schema.rs (v0.15.0+) Schema creation and forward migrations. Everything here runs from Database::init, which every constructor calls, so opening a database is what upgrades it.
grooveseek/src/db/search.rs (v0.15.0+) Retrieval: vector KNN, FTS5 candidates, and the RRF fusion that merges them. The half of the module whose behavior is observable as a number. (feature-66) Both queries read chunks.table_rows and pass it through table_rows_for (in db.rs), so a value that does not fit the chunk’s content reaches SearchResult::table_rows as None, never as an error.
grooveseek/src/db/fts_query.rs (v0.16.0+) Compiles a query string into the FTS5 MATCH expression db/search.rs sends. parse_query keeps "..." regions verbatim (FTS5’s own doubled-quote convention), cuts the rest at separators and then at script boundaries (kanji / hiragana / katakana / other word characters), merges runs under the 3-character trigram floor into their neighbours within the same group, and joins the resulting phrases with ` OR ` (deduplicated, capped at 32). Every emitted phrase is a contiguous substring of the input — a trigram tokenizer matches nothing else, so a phrase that is not one could never match by construction. A query yielding no phrase falls back to the pre-v0.16.0 form (the whole trimmed query as one phrase) at 3 characters or more, and to vector-only below that. (v1.1.0+) A whitespace-delimited group led by - is tokenized the same way but excluded rather than searched for, capped separately at 32; ParsedQuery::match_expr joins the two sides as (positives) NOT (negatives), and ParsedQuery::positive_text is the raw query with the excluded groups cut out, for the embedder, the reranker, and match_spans (see ADR-0011). Query-side only: schema and tokenizer are unchanged, so no re-index is needed.
grooveseek/src/db/storage.rs (v0.15.0+) Documents and chunks. Writing a document is a multi-table operation — documents, chunks, fts_chunks, vec_chunks, document_fields all have to agree — so several methods here open a transaction only when the caller has not (is_autocommit()). (feature-66) insert_chunk_with_table_rows writes chunks.table_rows (the JSON of Chunk::table_rows, NULL for not computed) in the same insert as the chunk, update_chunk_table_rows rewrites that column alone, and chunk_table_rows_for_path reads it back in chunk order.
grooveseek/src/db/meta.rs (v0.15.0+) Index-level metadata, statistics and whole-index maintenance: index_meta (embedding model / dim / context mode / declared_fields), document and chunk counts, path-to-hash maps for rename detection, and (AU-71) corpus_snapshot, which reads counts and a digest of the indexed chunks inside one transaction.
grooveseek/src/mmr.rs (v0.7.0+) Maximal Marginal Relevance greedy re-rank with a similarity cache. mmr_select operates on the post-rerank candidate pool and is gated by [search.mmr] config or the mmr per-call param.
grooveseek/src/parent.rs (v0.7.0+) Display-time parent retriever. apply_parent_retriever expands hit chunks via expand_adjacent (level-aware sibling merge) or expand_whole_document (full-doc fallback for chunks under whole_doc_threshold_tokens). Score / rank / match_spans stay on the original hit; only content and the new expanded_from field change.
grooveseek/src/quality.rs Per-chunk quality scoring (length / boilerplate / structure signals). QualityProfile (Text / Binary / Definition) decides which of them apply: the two length-based signals are exempted for a binary-format chunk and, since v1.4.0, for a source-code definition.
grooveseek/src/graph.rs Connection graph BFS over the vector index, for the get_connection_graph MCP tool and the groove graph CLI.
grooveseek/src/graph_render.rs Draws a finished connection graph: Graphviz DOT, and a standalone SVG laid out here rather than by a dependency (the walk produces a tree, so depth is the column and sibling order the row). CLI-only; the MCP tool stays JSON.
grooveseek/src/eval.rs Optional retrieval-quality evaluation for the groove eval CLI. Parses a golden YAML, runs each query through db.search_hybrid, and computes recall@k / MRR / nDCG@k. Loads / saves <kb_path>/.groove-eval-history.json for diff display. ConfigFingerprint (v0.7.0+) carries optional mmr / parent_retriever / fusion (v0.13.0+) so eval runs with different settings produce distinguishable history entries; each is recorded only when it differs from the built-in default, keeping older baselines comparable. Also scans the indexed corpus (v0.24.0+) for documents quoting two or more golden queries verbatim and reports them on stderr / in findings, without touching the exit code. Opt-in; does not affect serve / search / index.
grooveseek/src/tune.rs (v0.13.0+) Optional measurement tool for the groove tune CLI. Sweeps a fixed grid of RRF constants and FTS5 bm25 column weights over the golden query set, guards the result with nested leave-one-query-out CV (paired SE, selection stability, secondary-metric non-degradation — a sign test is also computed and reported, but decide does not gate on it), and prints either a paste-ready [search.fusion] snippet or a “keep the defaults” conclusion. Applies nothing automatically and never runs a reranker. Reuses eval’s GoldenSet / compute_query_metrics and db::fuse_rrf_ids.
grooveseek/src/tune/grid.rs (v0.15.0+) The parameter space groove tune sweeps and the per-query state carried through a sweep.
grooveseek/src/tune/stats.rs (v0.15.0+) The statistics behind an adopt / decline decision — mean, sample SD, paired SE, sign test, and the adoption thresholds ADOPT_MIN_MEAN_DELTA / ADOPT_SE_MULTIPLIER / STABILITY_MIN.
grooveseek/src/tune/report.rs (v0.15.0+) Rendering a sweep result for stdout (text via print!, plus the JSON form).
grooveseek/src/test_support.rs #[cfg(test)] only — the scratch directories the unit tests under src/** share, so that every one of them creates them the same way. A name is the process id, a nanosecond clock and an atomic counter: the first two collide between parallel threads of a single process, which was measured rather than assumed. A Drop guard removes the tree. The tempfile crate is deliberately not used. The integration tests under tests/ cannot reach this and carry their own copy in tests/common/temp.rs.

Data flow

.md / .txt / .pdf / .docx / .xlsx / .pptx / .rs files (.py, .php need a plugin)
(filtered by Registry::extensions(); only .md is on by default)
     │
     ▼ walkdir
indexer.rs: SHA-256 content-hash diff vs the chunks.hash column
     │
     ▼ changed files only
parser/: dispatch by extension → extract frontmatter + title + chunk
     │
     ▼
embedder.rs: embedding via FastEmbed by default
              (BGE-small-en-v1.5 → 384 dim, BGE-M3 → 1024 dim)
             or an opt-in OpenAI-compatible HTTP endpoint
              (dimension declared in trusted configuration)
     │
     ▼
db.rs: UPSERT into chunks (metadata)
       + vec_chunks (embedding)
       + fts_chunks (FTS5 trigram)
       + document_fields (declared frontmatter keys)

At query time the search tool runs a hybrid:

The full v0.7.0 pipeline is RRF → reranker → MMR → parent retriever → match_spans. Each stage is a no-op when its config is off, so the pipeline collapses to pre-v0.7.0 behavior by default. See retrieval-pipeline.md for the narrative.

Contextual Retrieval (v0.12.0+)

Static Contextual Retrieval (feature-46) prepends a document-structure breadcrumb to each chunk before it reaches the embedder / FTS index / reranker, entirely at index time and with no LLM call. Gated by [contextual].enabled (default off as of v0.12.0 — a false-by-default judgment gate result, see “Contextual Retrieval” in docs/usage.md for the A/B numbers that drove it).

Embedding cache resolution

embedder.rs::resolve_cache_dir() picks in order:

  1. FASTEMBED_CACHE_DIR env var (highest priority; must be absolute; a non-empty relative value stops groove, an empty value counts as unset)
  2. OS-standard cache directory joined with fastembed:
    • Linux: ~/.cache/fastembed
    • macOS: ~/Library/Caches/fastembed
    • Windows: %LOCALAPPDATA%\fastembed

If neither 1 nor 2 names an absolute directory, groove stops instead of loading a model relative to the CWD.

First run downloads the chosen ONNX model to a HuggingFace-hub-compatible cache layout (BGE-small: ~130 MB, BGE-M3: ~2.3 GB, BGE-reranker-v2-m3: ~2.3 GB). Subsequent runs reuse the cache without re-downloading.

If fastembed-rs’s native TLS to HuggingFace fails (corporate proxies / TLS inspection), see “Working around HuggingFace TLS failures” in docs/clients.md for a huggingface_hub CLI workaround.

CLI output convention

The groove CLI follows a stdout = data, stderr = progress convention:

When writing subprocess tests, grep grooveseek/src/main.rs for the corresponding Commands::* block to confirm which channel each subcommand uses before asserting on the captured output — and remember the arm may delegate its printing to a helper (print_search_results / print_graph / print_validate_report / print_doctor_report). Read the two lists above rather than counting them: the split is settled in ADR-0010 and frozen by docs/stability.md, and a number written beside a list goes stale on its own. serve is a special case: it writes nothing as CLI output, but over the default stdio transport the MCP protocol itself occupies stdout, which is why a subprocess harness must keep draining it. Grep for print! as well as println! — the text branches of eval and tune use it.

Key dependencies