Semantic search over a Markdown knowledge base, served over MCP.
How GrooveSeek behaves once it runs: what is indexed, where it is stored, which files are refused, and how search combines its two indexes.
日本語版: behavior.ja.md
FASTEMBED_CACHE_DIR environment variable, if set.fastembed (Linux: ~/.cache/fastembed, macOS: ~/Library/Caches/fastembed, Windows: %LOCALAPPDATA%\fastembed)..fastembed_cache under the current working directory (final fallback)..groove.db in the parent directory of the --kb-path (i.e. the repository root when --kb-path points to knowledge-base/).[parsers].enabled are indexed. The section defaults to ["md"] (the default behavior); ["md", "txt"] opts into .txt where the title is derived from the filename, ["md", "pdf"] (v0.10.0+) opts into .pdf (see the PDF indexing note below), ["md", "docx", "xlsx", "pptx"] (v0.11.0+) opts into Office documents (see the Office document indexing note below), and ["md", "rs"] (v1.2.0+) opts into source code (see the source code indexing note below) — "py" (v1.3.0+) is source code too, but needs its grammar plugin placed first. Unknown ids (e.g. "rst" / "adoc") are rejected at startup; an empty array is also rejected to avoid silent “nothing is indexed” failures. The section is honoured only from a config groove trusts: one it merely discovered beside a knowledge base has [parsers] reset to the default with a warning, since the key decides which parsers run and whether a grammar plugin is loaded — see Trusted and untrusted config locations.[parsers].enabled = ["md", "pdf"]. Text is extracted page-by-page with oxidize-pdf (pure Rust); each non-empty page becomes one chunk with heading p.N. Title / CreationDate PDF metadata become frontmatter when present, falling back to a filename-derived title when the PDF has no Title. Encrypted PDFs are skipped with a warning (no password support). Like other binary formats, .pdf files are subject to the 50 MiB raw-byte size cap — larger files are skipped with a warning instead of aborting the run. Known limitations:
/ToUnicode (what ReportLab emits). Earlier versions decoded that form to mojibake — the cause was upstream in oxidize-pdf, which read /DescendantFonts only when the CIDFont was written as an indirect reference; this project reported and fixed it (bzsanti/oxidizePdf#469, fix shipped in oxidize-pdf 4.3.0, picked up in v0.15.2). Japanese PDFs embedding a TrueType subset with a /ToUnicode CMap — what Word, LibreOffice and Google Docs export — extracted correctly all along (measured 2026-08-10: 569 chars/page on a dense Japanese report).あ comes out as 0B) — and refuses to index text no query could match, naming the decode failure rather than blaming the page density.Title metadata is not filtered: the filename fallback only triggers when the PDF’s Title field is empty. A non-empty but meaningless auto-generated title (e.g. left over from an export pipeline) is used as-is.-\n is joined only when the character before - and the character after \n are both ASCII lowercase letters (to avoid corrupting hyphenated model numbers, dates, or CJK-adjacent hyphens). This occasionally leaves a genuine word break unjoined, or joins a coincidental lowercase-lowercase pair that wasn’t actually a hyphenation.groove also works around a oxidize-pdf quirk found while dogfooding a real Japanese PDF (2026-07-19): when a PDF’s /Title uses the PDF spec’s UTF-16BE string form (common for non-ASCII titles), the dependency doesn’t detect the byte-order-mark and mis-decodes the string one byte at a time, producing mojibake. groove detects this specific mis-decode pattern and reverses it to recover the original title; if recovery isn’t possible (or the recovered text still looks like garbage) the filename fallback kicks in instead of surfacing mojibake. Extracted page content was never affected by this — only the title field.
Office document indexing (v0.11.0+): opt-in via [parsers].enabled = [..., "docx", "xlsx", "pptx"]. Each format is parsed in-tree (no LibreOffice / MS Office dependency):
| Extension | Library | Chunk granularity | Frontmatter source |
|---|---|---|---|
.docx |
zip + quick-xml | Heading-hierarchy sections, same rule as Markdown (Heading1–Heading6 paragraph styles are section boundaries) |
docProps/core.xml (Dublin Core: title / created / keywords) |
.xlsx |
calamine | 1 chunk per non-empty sheet (heading Sheet: <name>), truncated at 1 MiB per sheet (row-aligned — a row that pushes the sheet past the cap is kept whole, then extraction stops) |
docProps/core.xml |
.pptx |
zip + quick-xml | 1 chunk per slide (heading Slide N: <title>, or Slide N when the slide has no title placeholder), with speaker notes appended as a trailing [notes] section resolved via the slide’s .rels relationship (no same-numbered-file guessing, to avoid misattributing notes to the wrong slide) |
docProps/core.xml |
Known limitations:
No legacy binary formats: pre-2007 .doc (Word), .ppt (PowerPoint) and .xls (Excel) are not supported — only the OOXML forms (.docx / .pptx / .xlsx) above.
.xls was indexed between v0.11.0 and v0.13.1 and was withdrawn in v0.14.0: calamine materialises the whole workbook densely while opening it, BIFF bounds a sheet but not a workbook, and an allocation failure aborts the process rather than skipping the file. Listing "xls" in [parsers].enabled is now rejected at startup with that explanation — convert the workbook to .xlsx, which is read as a stream, and keep the original: the conversion carries over cell text but is not lossless in general (VBA macros need .xlsm, and other legacy-only features may be dropped). Full rationale: ADR-0001.
.odt / .ods / .odp are not supported.index run — there is no password support..docx and .pptx table cells are read as ordinary text runs (no row/column structure preserved in the chunk); .xlsx rows are tab-joined per line. Downstream retrieval sees prose, not a grid.Like .pdf, all four formats share the 50 MiB raw-byte size cap (MAX_RAW_BINARY_BYTES) with the indexer’s size-skip guard and get_document.
[parsers].enabled = [..., "rs"]. The unit is a definition, taken from the grammar’s own tags query, not a heading: a function, a struct, a method each become one chunk. A definition’s chunk starts at the doc comment written directly above it (an intervening blank line ends the run — a comment separated from the definition is commentary on the file), and carries the enclosing scope as context, which is how two methods of the same name in one file stay apart. Everything no definition covers — imports, top-level statements, the braces framing an impl block, and any region the parser could not understand — is filled in by line, so a file with a syntax error contributes the definitions around the break rather than collapsing into one chunk; a fragment shorter than the quality filter’s short-content threshold is dropped unless the file would otherwise produce nothing. A definition over [parsers.code].max_chunk_chars (default 3500 non-whitespace characters) is split into its nested definitions, or by lines when it has none — the usual case, since a method holds no nested definitions; each piece keeps the heading and kind of the definition it came from. Hits carry start_line / end_line / symbol_kind, absent on anything not from a source file; the line range describes the chunk, so opening the file there shows what was returned. symbol_kind is the grammar’s word rather than the language’s keyword — Rust’s tags query calls a struct, an enum and a union all class. Code and prose mix in results by default; separate them with tags_any: ["code"] or a path_globs entry beginning with !. Known limitations: one-line declarations (pub mod x;, unit structs) score below the quality cutoff and are filtered out, carrying no information beyond a name that is indexed elsewhere; a source file over 1 MiB is skipped with a warning, because tree-sitter’s allocator aborts the process on out-of-memory rather than unwinding, so the file has to be refused before the parser sees it; a file holding a definition nested under more than 64 syntax-tree ancestors is chunked by lines instead of by definition and tagged parse:too-deep, because resolving each definition’s scope costs more the deeper it sits and one 10 KB file nested a thousand levels took 64 seconds to index before that bound existed ; target/ is one of the default exclude_dirs, so enabling rs on a repository root does not index build output — but setting that key replaces the whole list rather than adding to it, so a custom exclude_dirs has to name target again. Rust is compiled in; other languages arrive as separate libraries you place, Python being the first (v1.3.0+) — see Placing a grammar plugin. See ADR-0012 and, for the nesting bound, ADR-0014.groove serve spawns a notify-based watcher by default ([watch].enabled = true, 500 ms debounce). Manual saves, git pull, and external scripts are re-indexed incrementally on the same Mutex-guarded resources used by MCP tools, so concurrent triggers are serialized. Disable with --no-watch or [watch].enabled = false.--transport http --port 3100 serves MCP over rmcp’s Streamable HTTP at /mcp, with /healthz for probes and a Mutex-serialized pipeline inside. Default bind is 127.0.0.1:3100 — 0.0.0.0 is opt-in, and GrooveSeek has no authentication by design (stability.md): the boundary belongs to a container, a reverse proxy, or the application in front. Anything that can reach the port can read the whole knowledge base.--model. BGE-small-en-v1.5 = 384, BGE-M3 = 1024. The chosen dim is declared on the vec_chunks virtual table and recorded in the index_meta table; a mismatch at runtime is detected and rejected.index runs (unless --force is passed). Moving / renaming a file without modifying its content is detected via hash match and handled as a documents.path UPDATE — the existing chunks, embeddings, and FTS rows are reused instead of being rebuilt. The rebuild summary reports the number of renames as renamed next to updated / deleted.index run — a stray binary file mixed into --kb-path no longer breaks indexing for the rest of the knowledge base.stat before the file is read, for text (MAX_RAW_TEXT_BYTES, v0.17.0+) as well as binary formats (MAX_RAW_BINARY_BYTES). Text used to be uncapped, which let a single oversized .md pull its whole content into memory — reachable from any client, since rebuild_index is an MCP tool. Files over the cap are skipped with a warning naming which limit applied.search tool combines SQLite FTS5 full-text search (trigram tokenizer, works for Japanese/CJK too; three columns since v0.12.0 — heading, context, content — with heading weighted 2× in bm25 by default) with the vector search via Reciprocal Rank Fusion (k = 60 by default). Both the weights and k are configurable under [search.fusion] since v0.13.0, and groove tune measures whether moving them helps on your KB. The returned score is the RRF score (higher = better), not a distance. Since v0.16.0 the query is compiled into per-token phrases joined with OR rather than searched verbatim (see “One-shot search from the command line” in docs/usage.md). A query that yields no usable phrase falls back to vector-only — but a query whose fragments are all short falls back to one whole-query verbatim phrase first, so vector-only happens exactly when the query is under 3 characters after trimming (below the trigram minimum). Since v1.1.0 a leading - on a whitespace-delimited group excludes it from both halves of the search instead of searching for it ("-foo" searches for a literal hyphen); see ADR-0011.--reranker <model> the top candidates are re-scored by a cross-encoder before being returned. When rerank is applied, score is the cross-encoder raw score instead of the RRF value. Reranking is index-independent — you can toggle it at server start without re-indexing.Connection graph: get_connection_graph / groove graph do BFS over the vector index starting from a document. No extra index is built; each node that gets expanded runs one fresh sqlite-vec KNN, and there is no ANN index, so one KNN scans every vector in the knowledge base.
Two bounds keep a request finite, and both report themselves when they bite:
| bound | default | ceiling | what it bounds |
|---|---|---|---|
max_seed_chunks |
32 | 1000 | chunks of the start document used as seeds. Applied as a SQL LIMIT, so rows past the cap are not read — except one probe row, which is what detects truncation without a second query. |
max_nodes |
100 | 2000 | nodes in the result. Each node is queued once and expands at most once, so knn_queries <= total_nodes <= max_nodes — this one bound caps the response size and the query count. |
depth (max 3) and fan_out (max 20) shape the walk but do not bound it: before these caps existed, the walk seeded from every chunk of the start document, and that count was uncapped. Measured on a 650-document knowledge base (9,419 chunks, BGE-M3) against its largest document, 160 chunks, with the release binary:
depth |
before: KNN / nodes / wall | after (defaults): KNN / nodes / wall |
|---|---|---|
| 1 | 160 / 767 / ~19 s | 14 / 100 / ~1.1 s |
| 2 (default) | 767 / 1997 / ~87 s | 14 / 100 / ~1.1 s |
| 3 (max) | 1997 / 3682 / ~200 s | 14 / 100 / ~1.1 s |
Raising both bounds to their ceilings (--max-seed-chunks 1000 --max-nodes 2000) reproduces the depth-1 and depth-2 rows exactly, with truncated: false — the bounds limit the walk, they do not change what it finds. Depth 3 is the exception: its 3,682 nodes are above the 2,000 ceiling, so the ceiling run returns 2,000 nodes in ~59 s with truncated: true. Results larger than max_nodes’ ceiling are no longer reachable by anyone.
What the caller sees when a bound bites: truncated: true at the root of the response, plus a truncation array whose entries carry reason (seed_chunks / node_budget), the limit that fired, and the remedy for that specific reason. truncated means something was lost, not a counter reached its cap: a walk that exhausts the graph while exactly filling the budget reports false. stats.seeds_used reports how many chunks actually seeded the walk. The CLI text renderer prints the same information on its stats line and one ! line per reason.
Because BFS is breadth-first, the budget is spent on the shallowest layer first — on a long document the default budget fills during the depth-1 expansion, so raising depth alone changes nothing. To spend the budget on depth instead of breadth, use --seed-strategy centroid (one seed node instead of many, so all but that one node goes to connections: the same document returns a depth-2 graph of 24 nodes in ~0.4 s) or lower --max-seed-chunks / --fan-out. Note that max_seed_chunks bounds the read, so centroid averages the same capped prefix — it frees the node budget, it does not recover chunks the seed cap dropped.
The database lock is held throughout, so a graph request still delays concurrent searches for as long as it runs. The bounds make that duration finite and predictable, but they are expressed in nodes, not seconds: one KNN costs about 72 ms on the knowledge base measured above, and scales with its chunk count and embedding dimension. exclude_paths — like search’s path_globs / tags_any / tags_all — is limited to 64 entries of at most 1 KiB each.
Scores are cosine similarity approximated from L2 distance (1 - d²/2, clamped to [0,1]) assuming unit-normalized embeddings (BGE-small / BGE-M3 are normalized internally).
exclude_headings are dropped during chunking. The default is an empty list (keep every section); populate exclude_headings in groove.toml to opt in. Matching is substring-based (heading.contains(pattern)), so short patterns catch suffixed variants ("参考リンク" would also match "## 参考リンク (旧)").walkdir skips any directory whose basename matches an entry in exclude_dirs. Matching is on the whole name and case-insensitive: Unicode lowercase mapping with Greek final sigma folded, so ["résumé"] matches RÉSUMÉ and ["οσ"] matches ΟΣ. It is not full Unicode case folding (straße and STRASSE stay distinct) and names are not normalized (a combining-mark spelling still differs from a precomposed one). exclude_dirs = ["build"] therefore also skips a directory created as Build — on Windows and macOS those are one directory, and an exact match would let the exclusion be bypassed by however it happens to be capitalised on disk. The same rule applies to the full index walk, groove validate, and the live watcher. The default list is [".obsidian", ".git", "node_modules", "target", ".vscode", ".idea"]. A user-specified list replaces the default entirely (no merging); exclude_dirs = [] walks everything except .git / .svn / node_modules, which stay excluded as a fail-safe..grooveignore (v0.21.0+): a file named .grooveignore in the root of the knowledge base excludes paths using gitignore syntax — * / ? / [a-z], ** in its three positions, a trailing / for directories only, a leading / to anchor to the root, and ! to re-include something an earlier line excluded. Where exclude_dirs can only name whole directories, this can name files and globs: drafts/, *.tmp.md, archive/**, notes/*.md with !notes/keep.md.
Only that one file is read. Subdirectory ignore files are not consulted, nothing above kb_path is, and .gitignore is not — a knowledge base kept in git often ignores exactly the large files you want indexed, so what gets indexed does not change because the repository’s ignore rules did. Copy the patterns across if you want them.
The three layers are a union: the built-in .git / .svn / node_modules fail-safe, then exclude_dirs, then this file. Any one of them excluding a path is enough, so ! can only undo an earlier line of .grooveignore itself — it cannot bring back something exclude_dirs or the fail-safe removed. Matching is case-insensitive, like exclude_dirs, so the same file behaves the same way on Linux as on Windows and macOS. As in git, a file under an excluded directory cannot be re-included by a later ! line, because the walk never descends into it.
The same rules apply to the full index walk, groove validate and the live watcher. Editing the file while the server is running takes effect for subsequent file events; documents already in the index stay there until the next groove index (or MCP rebuild_index), which re-reads the file and drops what it now excludes. A file that exists but cannot be read — a hard link, a symlink, a directory, over 64 KiB, or past 1000 patterns — is reported as a warning and the run continues without it (or without the excess patterns) rather than refusing to start.
This bounds indexing, not access. An excluded file is never indexed, so it can never appear in search or get_connection_graph, which read from the database and never touch the filesystem. It remains readable through get_document by a caller that knows its path, exactly as a file under exclude_dirs always has been. That is deliberate rather than an oversight: whoever can write into the knowledge base can also delete .grooveignore, so a rule living inside the tree cannot be the thing that guards it. Keep anything that must not be readable outside kb_path.
get_document, because whoever can write into the knowledge base should not be able to make groove read a file they cannot read and hand it back through search. A hard link does the same thing while looking like an ordinary file — it is a second name for one file, it needs no read access to create, and on Windows no privilege either — so a file with more than one name is refused the same way, in all three places. The refusal names the file and says why. This is deliberately blunt: nothing portable can tell whether the other name is inside the knowledge base or outside it, so a legitimately hard-linked note (deduplicated, or shared between two knowledge bases) is skipped too. Replace it with a copy if it belongs in the index. A file whose link count cannot be read at all — it was just deleted, say — is allowed through, so deletions still reach the index. Since v0.20.0 the decision and the bytes come from the same open handle: link count, file type and the size limit are all read off the handle the content is then read from, so a hard link renamed over a collected path after it was checked is refused rather than read. On Unix that open also refuses to follow a symlink put there instead, and refuses to block on a named pipe; on Windows it does neither, because creating a symlink there needs administrator privilege, and refusing reparse points would refuse every OneDrive placeholder. This still raises the bar rather than drawing a boundary. Whoever can link a file in and delete its original name leaves the knowledge base path as the only name, indistinguishable from a file that was always there — a link count is the state of a file now, not where it came from. An intermediate directory replaced by a symlink is out of reach on every platform. And the count is only ever what the filesystem reports: FAT32, exFAT and most network shares answer 1 whatever the truth is, so a knowledge base on a USB stick or a share gets nothing from this guard. Keep anything that must not be readable by groove outside kb_path, on a path groove’s user cannot read.get_best_practice path templates: the tool is opt-in and requires [best_practice].path_templates in groove.toml. Each template may use {target} as a placeholder (e.g. "best-practices/{target}/PERFECT.md" or "docs/{target}.md"). The server tries templates in order and returns the first existing file under kb_path (path-traversal attempts are rejected). Omitting the section — or writing path_templates = [] — leaves the tool registered but makes it return a “not configured” error, so accidental calls fail loudly instead of silently retrieving an unrelated file.0.3): each indexed chunk gets a quality_score computed from three signals — length (< 30 chars → -0.6), boilerplate-only content (TBD / TODO / 詳細は後述 / etc. → -0.5), poor structure (single line < 80 chars → -0.3). Chunks scoring below the threshold are hidden from search, groove search, and get_connection_graph. Seed chunks of get_connection_graph are exempt. Disable the filter with [quality_filter] enabled = false in groove.toml, or opt out per-query with --include-low-quality (CLI) / include_low_quality: true (MCP). Override the threshold with --min-quality 0.5 / min_quality: 0.5. Upgrading an existing index: the next groove index run transparently adds the quality_score column (ALTER TABLE) and backfills scores once (idempotent).docs/configuration.md — the keys that steer this behaviordocs/retrieval-pipeline.md — the search pipeline in fullREADME.md — install and quick start