QMS Assistant
Problem
Quality-management knowledge is spread across CMMI procedures, specifications and test plans, and the answer to a single question is often split between several of them. Sending those documents to a cloud model was not an option: to stay aligned with KVKK, Turkish data-protection law, everything had to run locally.
Approach
Hybrid retrieval in Qdrant merges dense and sparse BGE-M3 vectors with Reciprocal Rank Fusion, so exact terms and meaning both count. Documents are split into parent and child chunks, and late chunking pools each child's vector from a single pass over its parent, so small chunks keep their surrounding context. A local Ollama model answers at temperature 0, cites its sources, and falls back to an explicit "not in the documents" notice when retrieval is weak. Because BGE-M3 is multilingual, an English question retrieves from the Turkish corpus and is answered in English. Alongside the assistant sit a compliance checker, which runs deterministic structure and traceability rules before an LLM-as-judge content tier, and a lifecycle map extracted from the documents, where any claim that cannot be cited is dropped.
Challenges
The assistant occasionally answered from the wrong document. Logging the exact context the model received showed why: long documents whose relevance was spread across several chunks were ranking just below the top-three cutoff and never reached the model. A per-document scoring blend, the best chunk plus half the rest, let them surface without burying short, densely relevant ones.
Validation
A labelled compliance answer key deliberately includes compliant and reworded documents, so false positives are measured rather than ignored; the deterministic tiers score 1.00 precision and recall across 81 labels. Retrieval is tested with a 20-question bilingual set grouped by capability, including out-of-scope questions that must trigger the fallback. An early automated model comparison was discarded after review showed the LLM grader had never been shown the documents it was judging groundedness against.
Outcome
A one-command local system, with FastAPI serving both the API and a no-build React interface, backed by a pytest suite and running on a 16 GB laptop with no document or question leaving the machine. The repository ships with a synthetic corpus and an ingestion script for pointing it at real documents.