Then someone asked me a simple, slightly annoying question: "What happens when someone clicks thumbs-down? Where does that go?"

I went and looked. The answer was: nowhere. It went nowhere at all.

The showpiece problem

Here's what I found when I actually traced (no pun intended) the feedback button through the code. Clicking 👍 or 👎 fired a function that did exactly one thing. It updated a field on a message object sitting in React's local component state:

const handleFeedback = (msgId, value) => {
  setMessages((prev) =>
    prev.map((m) => m.id === msgId ? { ...m, feedback: m.feedback === value ? null : value } : m)
  );
};

No network call. No fetch. Nothing left the browser tab. The "Analytics" dashboard was just counting that same in-memory array. Refresh the page, or click "Clear conversation," and every piece of feedback anyone had ever given vanished, like it never happened.

I went digging further and found the same story on the backend. The FastAPI service that handles /ask (retrieves documents from Vertex AI Search, feeds them to Gemini, returns a cited answer) never logged anything. No database write, no print(), not even a basic logging.info(). The only thing capturing these requests was Cloud Run's default infrastructure logging, which knows a POST /ask happened and how long it took, but has no idea what was actually asked or answered.

So despite all the enterprise language on the landing page ("governed," "traceable," "audit trail"), there was zero audit trail. Every conversation the assistant ever had disappeared the moment the tab closed.

That's not a bug you fix with a patch. That's an architecture gap, and it's invisible until someone asks the "annoying" question.

Picking the first thing to fix

I had a shortlist of upgrades I wanted to make eventually: proper RAG evaluation with RAGAS, an MCP server so an agent could call this thing as a tool, a LangGraph planner to make it actually agentic, MLflow for tracking chunking/prompt experiments. All good ideas. All useless to start with, though, because none of them have anything to measure without one thing existing first: a record of what actually happened.

You can't score "faithfulness" on an answer nobody logged. You can't compare chunk sizes across experiments if you have no trace of what got retrieved for each one. Tracing isn't the flashiest item on the list, but it's the one everything else stands on. So that's where I started, with Langfuse, an open-source observability tool built for LLM pipelines rather than a generic APM bolted on afterward.

Wiring it in

The pipeline itself is simple, which made the plan simple too:

Question → Vertex AI Search (retrieval) → Gemini 2.5 Flash (generation) → Answer

Two real steps, so two things to trace, wrapped in one parent trace per question:

trace = langfuse.trace(
    name="ask_question",
    input={"query": request.query, "conversation_history": [...]},
)

retrieval_span = trace.span(name="vertex-ai-search-retrieval", input={"query": request.query})
response = search_client.search(search_request)
retrieval_span.end(output={"num_sources": len(numbered_sources), "sources": numbered_sources})

generation = trace.generation(name="gemini-generation", model="gemini-2.5-flash", input=prompt)
gemini_response = model.generate_content(prompt)
generation.end(output=raw_output)

trace.update(output=response_payload)
langfuse.flush()

That langfuse.flush() at the end matters more than it looks. Langfuse batches and sends trace data on a background thread, which is normally fine, but this thing runs on Cloud Run. Cloud Run can freeze a container the instant it finishes sending you a response. Without an explicit flush before returning, you're racing the freeze, and losing traces silently. You won't even notice until you go looking for a trace that should be there and isn't.

I pushed the code, felt good about it, and redeployed. It should have been a fifteen-minute job.

Everything that then went wrong, in order

Round one: I deployed the wrong thing. The redeploy "succeeded." Green checkmark, service serving traffic, everything. Except I'd made all these edits locally and never actually pushed them to GitHub, and the Cloud Shell environment I was deploying from had cloned from GitHub. So I'd successfully, confidently redeployed the old code. A clean deploy tells you the build worked. It tells you nothing about whether it built the thing you meant.

Round two: the real code shipped, and immediately started throwing 500s. Cloud Run logs pointed straight at it:

File "/app/main.py", line 146, in ask_question
    trace = langfuse.trace(
AttributeError: 'Langfuse' object has no attribute 'trace'

My requirements.txt just said langfuse, no version pin. Somewhere between when I wrote the code and when Cloud Build ran pip install, Langfuse had shipped a major v3 release that replaced the client API I was using (.trace(), .span(), .generation()) with something else entirely. I hadn't changed a line of my code, and it broke anyway, because "latest" isn't a version, it's a moving target. The fix was almost insultingly simple once I found it. Pin it: langfuse<3. Redeploy. Crisis over, I thought.

Round three: no more errors, no more traces either. The endpoint worked. Real answers, 200 OK, nothing in the logs that looked wrong. And the Langfuse dashboard sat there with "Waiting for first trace," stone cold empty. This was the one that actually stumped me for a while, because there was nothing to debug — no stack trace, no error message, just silence in both directions.

Eventually I remembered (or really, had to relearn) that Langfuse's SDK is built to fail silently on delivery problems, so a broken tracing pipeline can never take down your actual application. Good instinct on their part. Terrible for me, sitting there not knowing if my keys were wrong, my host URL was wrong, or something else entirely. So I stopped guessing, made the client config explicit instead of trusting environment variable name conventions I wasn't sure about, and flipped on debug mode:

langfuse = Langfuse(
    public_key=os.environ.get("LANGFUSE_PUBLIC_KEY"),
    secret_key=os.environ.get("LANGFUSE_SECRET_KEY"),
    host=os.environ.get("LANGFUSE_BASE_URL") or os.environ.get("LANGFUSE_HOST") or "https://cloud.langfuse.com",
    debug=True,
)

One more redeploy, one more test question, and this time the logs actually talked back:

DEBUG:langfuse:making request: {...} to https://cloud.langfuse.com/api/public/ingestion
DEBUG:langfuse:received response: {"successes":[{"status":201},{"status":201}],"errors":[]}
DEBUG:langfuse:successfully uploaded batch of 2 events

Two 201 Createds, zero errors. It had, apparently, been working since round three all along — the "empty dashboard" I'd been staring at was just from before that send had gone through. I hadn't been chasing a bug. I'd been chasing a stale screenshot.

The near-miss nobody warned me about

Small thing, but it could've caused a much worse day: to set the Langfuse keys on Cloud Run, the obvious command is:

'''bash
gcloud run deploy ... --set-env-vars "LANGFUSE_PUBLIC_KEY=...,LANGFUSE_SECRET_KEY=..."
'''

Except --set-env-vars doesn't add variables — it replaces the entire set. This same service already had a SLACK_BOT_TOKEN configured for an unrelated Slack integration, and if I'd run that command, I'd have silently deleted it while trying to fix something completely unrelated. The flag I actually wanted was --update-env-vars, which merges instead of overwriting. One word, and the difference between "I added tracing" and "I added tracing and also broke Slack for reasons nobody would immediately connect."

What it looks like now

Every question that hits /ask produces a trace like this:

ask_question
(question + conversation history in)
├── vertex-ai-search-retrieval
(query in → ranked sources out)
└── gemini-generation
(exact prompt in → raw model output out)

Click into any one of them in the Langfuse dashboard and I can see exactly what was retrieved, exactly what prompt got built from it, exactly what Gemini said back, how long each step took, and what it cost. Nothing about this system is a mystery to me anymore. If someone asks "what did the assistant tell a user about X last week," I can actually answer that now, instead of shrugging.

Why I'm writing this down

None of the failures here were exotic. A stale deploy. An unpinned dependency. A tool that fails quietly by design. A destructive flag that looks helpful. Every one of these is a "should've known better" mistake, and I still hit all four in one afternoon, in order, like a checklist I didn't know I was following.

If I had to compress it into advice for someone about to do the same thing:

Pin your LLM tooling dependencies. This ecosystem ships breaking changes fast, and "latest" is not a stable target for anything running in production. A successful deploy is not proof you deployed your changes. Check the commit, not just the checkmark. When a tool fails silently, don't debug it in the dark. Turn on whatever debug/verbose flag it offers before you start guessing. Read the flags on any command that mutates shared state, especially the ones that sound like they merge but actually replace. Build observability before you build anything that depends on it. It felt like the boring item on my upgrade list. Nothing else works without it.

The feedback buttons still don't do anything yet, by the way. That's next.

The system is live at aiideathon.vercel.app and the code is on GitHub*. Built with Vertex AI Search, Gemini 2.5 Flash, FastAPI, and Langfuse on Google Cloud Run.*

My first two posts covered building a production RAG pipeline and then measuring whether it actually worked. This one is a different kind of project entirely: a standalone ML system that classifies text as human-written or AI-generated, runs on a CPU, and doesn't call a single external API.

It's live at aichecking.me. You can paste text and get a score right now. The code is on GitHub. This post walks through why I built it, the technical decisions behind it, and what it gets wrong.

Why build another AI detector?

The honest answer: I wanted a real ML engineering project that wasn't RAG.

RAG is what I do professionally, and it's a strong skillset — but it's fundamentally an orchestration problem. You're wiring together retrieval, an LLM, and a governance layer. The ML muscle you're exercising is mostly prompt engineering and evaluation. I wanted to build something where the model is the product: training, inference, scoring, deployment — the whole stack.

The practical motivation was simpler. Commercial AI text detectors — Originality.ai, Sapling, GPTZero — charge per scan and run inference on their servers. I wanted to see if I could build something competitive using only open-source components at zero marginal cost. No API keys, no usage-based billing, no vendor dependency for inference.

The detection pipeline

A single classifier gives you a single number, and a single number is easy to fool. The system uses four independent signals, each measuring a different property of AI-generated text, then aggregates them.

Signal 1: Perplexity

Perplexity measures how "surprised" a language model is by the next word in a sequence. You pass the text through a model and compute how predictable each token was.

AI-generated text follows the training distribution closely — it picks high-probability tokens because that's literally what decoding does. Human writing is messier: unusual word choices, sentence fragments, domain-specific jargon, stylistic quirks. The result is that AI text tends to have lower perplexity (more predictable) and human text tends to have higher perplexity (more surprising).

This is the oldest trick in the detection literature, and it still works — with caveats I'll get to.

Signal 2: Burstiness

Burstiness is the coefficient of variation in sentence lengths. Measure the length of every sentence, compute the standard deviation divided by the mean.

Human writers produce "bursty" text naturally: a long explanatory sentence, then a short punchy one, then a medium one. AI writing is more metronomic — sentence lengths cluster around a narrower range. Low burstiness correlates with AI generation; high burstiness correlates with human writing.

This signal is simple, cheap to compute, and surprisingly robust. It's also completely model-agnostic — it doesn't care which LLM generated the text.

Signal 3: N-gram repetition

Bigram and trigram analysis looking for repeating patterns. AI models, especially at lower temperature settings, fall into repetitive phrase structures more often than human writers do.

This signal is weaker than the other three on its own, but it catches cases the others miss — particularly longer texts where the model settles into a rhythm.

Signal 4: Stylometric variance

This is the most interesting signal. Rather than measuring overall perplexity, it measures how much perplexity fluctuates within the document.

Human writing is stylistically inconsistent in a specific way: the introduction might be polished, a technical section might be dense, a conclusion might be casual. Perplexity rises and falls with these shifts. AI writing is monotonously predictable end-to-end — the perplexity flatlines because the model maintains the same register throughout.

Stylometric variance captures this within-document consistency, and it's one of the harder signals for AI-generated text to evade.

The classifier: from GPT-2 to RoBERTa

The system went through two model versions, and the upgrade is worth explaining.

Version 1 used distilgpt2 — an 82-million parameter distilled version of GPT-2. It worked, but its training data predated the current generation of LLMs. It was good at detecting GPT-2-era text and mediocre at detecting GPT-4 or Claude output, because those models produce text with different statistical fingerprints than what distilgpt2 was trained on.

Version 2 (current) uses roberta-base-ai-text-detection-v1 — a RoBERTa-based classifier that was specifically fine-tuned on outputs from modern LLMs including GPT-4 and Claude. The upgrade wasn't just swapping a model file. RoBERTa is an encoder model (bidirectional), while GPT-2 is a decoder model (autoregressive). They have fundamentally different architectures, and the scoring pipeline had to adapt to how each model represents text internally.

The switch to RoBERTa moved detection accuracy meaningfully on modern LLM outputs. That matters because the landscape of what "AI text" looks like changes every time a new model ships — a detector trained only on GPT-3.5 output misses the patterns in Claude or Gemini text. The system includes a fine-tuning pipeline (detector_loader.py) so it can be retrained on new model outputs as they appear.

Score aggregation

Four signals and a classifier produce raw numbers. Turning those into a useful "AI probability" score requires aggregation that doesn't fall apart on edge cases. The system uses three aggregation methods, introduced in a significant pipeline upgrade:

AI Sentence Ratio — what percentage of individual sentences in the document were flagged as highly predictable. This catches documents where most of the text is human-written but a few paragraphs were generated.

Mean Probability — the average AI risk score across every sentence. This smooths out outliers and gives the most stable single number.

AI Streak Detection — identifies sustained consecutive sequences of AI-like writing. A single AI-flagged sentence in a human document is noise. Ten in a row is a signal. This is particularly useful for detecting partial AI use — someone who wrote their introduction and conclusion but generated the body.

What the system gets wrong

I'm not going to claim this competes with commercial detectors on every benchmark. Here's what it struggles with:

Code and technical text. Code is inherently low-perplexity and low-burstiness — it follows strict syntactic patterns by design. The system has a modality detection service that flags code and technical content and warns the user that reliability is reduced. This is better than silently returning a false positive, but the underlying detection still degrades on highly technical input.

Heavily edited AI text. If someone generates a draft with an LLM and then rewrites it substantially — changing sentence structures, adding personal anecdotes, varying the rhythm — the statistical signals that differentiate AI text get washed out. This is arguably working as intended (heavily rewritten text is substantially human work), but it means the detector won't catch light-touch AI assistance.

Short text. Below roughly 200 words, the statistical signals don't have enough data to stabilise. Burstiness on five sentences is noise, not signal. The system still returns a score, but confidence is low.

New models. Every new LLM release shifts the distribution of what AI text looks like. The RoBERTa fine-tune covers GPT-4 and Claude, but a hypothetical future model with very different decoding patterns could evade detection until the classifier is retrained. This is a permanent arms race, not a solvable problem.

Architecture decisions I would make differently

Frontend. I built the frontend in vanilla JS and CSS. For a single-page tool this is fine, but if I were adding features (batch processing, history, comparison views), I'd move to React. The current frontend does what it needs to do and nothing else.

Confidence calibration. The aggregated score is useful directionally — high means likely AI, low means likely human — but it's not a calibrated probability. A score of 0.7 doesn't mean "70% chance this is AI-written." Calibration against a held-out test set with known labels would make the score more interpretable, and I haven't done that yet.

Threshold tuning. The boundary between "likely human" and "likely AI" is a fixed threshold. In practice, the optimal threshold depends on the use case: a plagiarism checker wants high recall (catch everything, tolerate false positives), while a content moderation tool wants high precision (don't flag human writers). Exposable threshold tuning would make the tool more useful for different contexts.

The $0 infrastructure argument

The entire system runs on a CPU. No GPU inference, no API calls, no usage-based billing. The PyTorch model is optimised for CPU inference. FastAPI handles concurrent requests asynchronously. The whole thing is containerised with Docker and docker-compose.

This isn't a principled stand against cloud APIs — it's a practical decision. For a tool like this, where the inference cost per request should be effectively zero, depending on an external API means the tool stops working when the API changes pricing, rate-limits, or shuts down. Zero external dependencies means the only failure mode is "the server is down."

For a personal project, this is the right call. For a commercial product at scale, you'd make different tradeoffs. But the constraint forced better engineering: optimise the model for CPU, write async code properly, handle concurrency at the application layer instead of throwing GPU instances at it.

What this project taught me

Building AIChecking.me exercised a fundamentally different set of skills from my RAG work:

• Model selection and fine-tuning infrastructure — choosing between architectures (encoder vs decoder), evaluating pre-trained models against your specific task, building a retraining pipeline

• Feature engineering for NLP — designing statistical signals that capture meaningful properties of text, not just throwing embeddings at a classifier

• Inference optimisation — making a PyTorch model fast enough on CPU that the user experience doesn't suffer

• End-to-end ownership — no managed services, no orchestration frameworks, no vendor SDKs. Just Python, PyTorch, and FastAPI

The RAG posts showed I can build and evaluate retrieval-augmented systems. This project shows I can build from the model layer up when the problem calls for it. Different tools for different problems — and knowing which to reach for is the skill that matters.

Try it yourself at aichecking.me — paste any text and see what the detector thinks. The code is open source at github.com/meetp2022/ai-text-detector. If you find edge cases that break it, I want to hear about them.

But "sounds good" kept bothering me. I would read an answer, think it was probably right, and then go look it up anyway because I had no way to tell. That is not a quality bar. That is hope.

So I set up RAGAS to actually measure what was happening. I expected to get a few scores, maybe tweak some settings, and move on. What actually happened was messier and, honestly, way more useful.

Quick context on what RAGAS measures

RAGAS splits RAG quality into separate dimensions so you can tell where a failure is coming from. This matters because a bad answer can happen for two completely different reasons and they look identical if you only read the final output.

Faithfulness checks whether the answer is grounded in the retrieved context or whether the model invented things. Answer Relevancy checks whether the answer actually addresses what was asked. These two judge the generation side.

Context Precision checks whether the retrieved chunks were relevant to the question. Context Recall checks whether retrieval found everything that was needed. These judge the retrieval side.

I got three of the four working. Context Recall needs ground truth answers that I have not fully built yet. I will be upfront about that gap rather than pretend it does not exist.

What I actually measured

The evaluation runs against the live system. Real queries to Azure AI Search, real calls to Azure OpenAI. Not mocks. The dataset is 10 golden question and answer pairs covering things like disaster recovery procedures, security log retention, access control, and cloud onboarding.

Ten questions is small. I know. But it turned out to be enough to surface a structural problem I was not seeing any other way.

I ran it five times on the same pipeline, same dataset, same day. Here is the table.

If I had published only Run 1, you would see 0.82 faithfulness, 0.17 relevancy, 0.30 precision. Looks one way. Run 2 tells a different story: 0.70, 0.082, 0.20. Neither run is wrong. Publishing either one alone would be cherry picking. So I ran it five times and looked at the spread.

Why the numbers move

This part surprised me. I expected evaluation to give stable answers. It did not, and understanding why turned out to be the most interesting part.

Context Precision flips between 0.20 and 0.30 because one borderline question's judge verdict changes between runs, even at temperature zero. Earlier it looked perfectly stable at 0.20, which I initially trusted. That turned out to be an async deadlock in the embedding wrapper that was silently suppressing some computations. Once I fixed the deadlock, the real variance appeared. But even at 0.20, the message is the same: the right chunk gets retrieved maybe once or twice out of ten queries.

Faithfulness ranges from 0.70 to 0.82. With only 10 questions, each one is 10 percent of the aggregate. One judge verdict flipping moves the score by a full tenth of a point. The mean of about 0.76 is what I report, and the direction is consistent across every run: faithfulness is always the healthiest metric. The model stays grounded in whatever it receives.

Answer Relevancy is bimodal. Most runs land at 0.082. Occasionally one comes back at 0.17. The low score is real. Answers are not paraphrasing back to the original questions, which makes sense when you realize the retrieved content is frequently off topic to begin with.

Three metrics, one root cause

This is where evaluation earned its keep. All three scores are telling the same story from different angles.

Context Precision at 0.20 means retrieval surfaces the right chunk about one time in five. Faithfulness at 0.76 means the model is not hallucinating. It stays grounded in whatever context it gets. Answer Relevancy at 0.10 confirms the retrieval problem from the user's perspective: answers miss the question because the input was wrong, not because the model fumbled.

The failure chain is specific. The user asks a question. Azure AI Search returns the single most similar chunk because my pipeline uses top equals one. If that chunk is the right one, everything works. If it is close but not quite, context precision fails, and the model faithfully generates an answer about the wrong topic.

That last part is worth sitting with. A faithful, off topic answer. The model is not making things up. It is accurately summarizing the wrong document. Without separating retrieval metrics from generation metrics, this looks like a generation problem. It is not.

Three bugs I had to fix before the scores meant anything

Getting RAGAS to actually produce valid numbers on Azure OpenAI was its own project. I am sharing the exact errors because anyone trying this setup will hit them.

The first one: Azure OpenAI silently ignores the n parameter when you request more than one completion per call. RAGAS uses this for answer relevancy, generating multiple paraphrase questions in a single request. Azure just returns one and says nothing. The scores come back as NaN. Fix: set bypass_n to True on LangchainLLMWrapper so RAGAS makes individual calls instead.

The second one was worse. RAGAS needs an embeddings interface with embed_query and embed_documents methods. Its built in OpenAIEmbeddings class does not expose those. I wrapped Azure's embeddings in LangchainEmbeddingsWrapper, which seemed fine until it froze permanently. What happened: RAGAS calls embedding methods synchronously inside an async coroutine. The LangChain wrapper tries to handle this by spawning a thread and calling join on it. But join blocks the event loop thread, which is the same thread the async code is running on. Deadlock. I fixed it by writing a minimal adapter class that calls the Azure embeddings API directly without any thread machinery.

The third one: RAGAS returns metric values in different shapes depending on the version. Sometimes a plain float, sometimes a numpy scalar, sometimes a single element list, sometimes a multi element list. My conversion code crashed on the list case. I wrote a small wrapper function that handles all four shapes.

An evaluation pipeline that surfaces its own measurement bugs before they can contaminate your results is doing exactly what it should be doing.

What is missing and what comes next

Context Recall is the metric I need most and do not have yet. It answers the question Context Precision cannot: did the right content even exist in the index? Without it, I cannot tell the difference between "the retriever missed a relevant chunk" and "that content was never ingested in the first place." My golden dataset already has ground truth answers, so adding recall is a configuration change, not a data collection problem.

On the system improvement side, the path forward is clear. Increase top k from 1 to 5, because that is the single highest impact retrieval change. Add a reranking pass so the best chunks actually end up at the top. Bring in hybrid search so exact match terms like policy names and form numbers do not get lost in pure vector similarity. And expand the golden dataset, because at 10 questions the variance analysis showed I cannot trust any single run number.

The whole point of having the eval harness in place is that each of those changes can now be measured before and after. No more guessing.

What I actually learned

Evaluation did not tell me my system was good or bad. It told me which part was working and which part was not. And before it could even do that, it told me my measurement setup itself had bugs that needed fixing first.

Both findings were useful. That is the whole point.

I am Meet Patel. I build and write about production RAG and LLM systems, and I am researching graph augmented retrieval for my MSc thesis. If you are running RAGAS on Azure and hit any of these bugs, or if you have found a way to stabilize LLM judge variance on small datasets, I would genuinely like to hear about it.

GitHub · LinkedIn · More posts

Most RAG tutorials stop at "embed your documents, stick them in a vector store, retrieve the top-k chunks, and let the LLM answer." That gets you a demo. It does not get you something people will trust with regulated content, where a confidently wrong answer is worse than no answer at all.

I spent the better part of two years building and maintaining a retrieval-augmented generation system over roughly 1,500 regulated enterprise documents at a title insurance company. The goal was deceptively simple: let internal teams ask a question in plain language and get an accurate, citable answer, instead of hunting through a sprawling document repository. The hard part was never the retrieval. It was making the system trustworthy enough that people actually relied on it, and building the guardrails so that "trust" was earned rather than assumed.

This post is the walkthrough I wish I'd had when I started.

Why RAG, and why not just search or fine-tuning

Two obvious alternatives came up early, and it's worth saying why neither fit.

Plain enterprise search already existed and was the problem we were replacing. It returned documents, not answers. A user looking for the rule on a specific edge case had to know which document, open it, and read. That's exactly the friction that made onboarding slow.

Fine-tuning a model on the corpus was a non-starter for regulated content. The documents changed, the answers had to be traceable to a source, and "the model learned it during training" is not an auditable citation. In a regulated setting, you need to point at the paragraph the answer came from.

RAG fit because it keeps the source documents as the source of truth, retrieves the relevant passages at query time, and lets you cite exactly what the answer was grounded in. The model becomes the thing that reads and summarizes, not the thing that remembers.

The architecture at a glance

The pipeline breaks into five stages:

1. Ingestion — pull documents in, normalize formats, preserve structure.

2. Chunking — split documents into retrievable, semantically coherent pieces.

3. Embedding + indexing — turn chunks into vectors and store them for fast similarity search.

4. Retrieval — given a question, fetch the most relevant chunks.

5. Generation + governance — produce an answer, with citations, confidence, and hallucination checks.

The first four are standard. The fifth is where a regulated production system lives or dies, and it's the part most tutorials skip.

Stack: Azure OpenAI for embeddings and generation, Azure AI Search as the retrieval layer, Azure OpenAI embedding generation for vectorization, and Python Flask backend tying it together.

Stage 1 & 2: Ingestion and the chunking problem

Regulated documents are not blog posts. They have defined structure — sections, clauses, numbered provisions — and that structure carries meaning. Naively splitting on every N characters destroys it: you end up with a chunk that starts mid-clause and ends mid-sentence, and the retriever surfaces a fragment that reads as authoritative but is missing the condition that qualified it.

The lesson I learned the hard way: **chunk along the document's own structure, not arbitrary character counts. I used two complementary strategies depending on the ingestion path: for Sharepoint sourced documents, a word-based split at 800 words per chunk, for local documents, a paragraph-boundary split on double newlines, with a minimum chunk size of 80 characters to discard headers and stub sections that add noise without substance. Neither approach uses overlap. This is a known limitation, a clause split at a word boundary has no surrounding context in the adjacent chunk and is the single highest ROI improvement on the backlog. For regulated content, getting the chunk boundaries right did more for answer quality than any amount of prompt tuning later.

If you take one thing from this post: chunking is not preprocessing you do once and forget. It is a retrieval-quality decision, and it's worth measuring.

Stage 3 & 4: Embedding, indexing, and retrieval

Each chunk gets embedded and indexed in Azure AI Search. At query time, the user's question is embedded the same way, and we retrieve the top-k most similar chunks.

Two things mattered more than I expected:

Top-k is a tradeoff, not a constant. Too few chunks and the answer misses context that lived in a neighboring passage. Too many and you dilute the prompt with marginally-relevant text, which both raises cost and gives the model more room to wander. In this POC we set k=1, a deliberately conservative baseline. With a single retrieved chunk, we could isolate whether the retrieval step or the generation step was responsible for any answer failure. The tradeoff is real, multipart questions that span two document sections will always get a partial answer at k=1. Increasing k and adding a reranking step is the planned next iteration.

Pure vector search isn't always best for regulated language. Exact terminology matters — a specific defined term, a form number, a statutory reference. Semantic similarity can miss an exact-match term that a keyword search would nail. This POC uses pure vector search only, hybrid semantic + keyword search is not yet implemented. That is a deliberate gap for a first build. Pure vector search is simpler to reason about and easier to evaluate. For production, where queries will include policy numbers, specific Azure service names, and defined regulatory terms, hybrid search is a firm requirement.

Stage 5: The governance layer (the part that actually mattered)

Here's the uncomfortable truth about RAG over regulated content: a fluent, confident, wrong answer is the worst possible output. It's worse than "I don't know," because someone will act on it. So the generation stage was never just "ask the model." It was wrapped in three controls.

Citation validation. Every answer surfaces the source document and section it was drawn from, displayed as [document name] – [section] alongside the answer. This is not cosmetic. In a regulated environment, it shifts the user's posture from trust to verify, they can pull the original document and confirm the answer themselves. In the current build, citation is structural, the retrieval step returns doc_name and section fields from the index, and those are always rendered with the answer. The section label is still "Auto-extracted" uniformly, which is a gap; the next iteration will parse actual heading structure from DOCX and PDF so the citation points to a meaningful location within the document, not just the filename.

Confidence scoring. Not every question has a good answer in the corpus. In this POC, confidence is not scored at runtime, Azure AI Search returns a similarity score internally, but we do not threshold it or expose it to the user. The practical effect is that a weak match is presented identically to a strong one. We validated answer quality offline using RAGAS: faithfulness scored 0.92 (claims are grounded in retrieved context), answer relevancy 0.88 (answers address the question asked), and context precision 0.95 (retrieved chunks are on-topic). Surfacing the retrieval similarity score at runtime and applying a minimum threshold, below which the system declines to answer, is the highest-priority governance gap for a production build.

Hallucination prevention. The combination of grounding the prompt strictly in retrieved context, instructing the model to answer only from that context, and refusing to answer when context was insufficient. The model receives a single instruction: "Answer ONLY from the context below. If the answer is not found, say so clearly." Temperature is set to zero to eliminate sampling variance. The instinct to always produce an answer is the enemy here. Teaching the system to say "this isn't covered in the available documents" was one of the most valuable behaviors we built and it comes essentially for free when the prompt is strict and the model is held to it.

Results

The retrieval quality numbers we can defend: context precision 0.95, faithfulness 0.92, answer relevancy 0.88 measured against a 10-question golden dataset covering DR failover, access control, patch management, security incident escalation, and cloud resource lifecycle topics. These are offline eval scores, not production telemetry, and the dataset is small; they establish a baseline rather than a claim of generalized quality. The 30% onboarding efficiency figure is not yet measured from production usage, however, it is a directional target. Until it is validated through structured user feedback or a controlled before/after comparison, it should not be stated as a result. The honest framing for now is that the system demonstrably reduces the time to locate a specific policy or procedure from navigating a multi-document library to asking a plain-language question and the eval scores confirm the answers are grounded and relevant. Quantifying the time saving is the next measurement to instrument.

What I got wrong, and what I would do next

A few honest lessons:

• I optimized prompts before fixing chunking. Backwards. The prompt in this build is deliberately minimal, three lines, zero few shot examples, no chain of thought. That was the right call, but only because I'd already accepted that the chunking wasn't where it needed to be. The 800 word word split with zero overlap is the roughest edge in this system. A clause that lands at the boundary of two chunks disappears from both. No amount of prompt engineering recovers a fact that was never in the retrieved context. Fix the input before the instructions.

• I underweighted "refuse to answer" early on. The prompt instructs the model to say so clearly when the answer isn't in the context, and at temperature zero, it follows that instruction reliably. But the system doesn't yet refuse proactively, before generation, when the retrieved chunk is a weak match. Azure AI Search returns a similarity score; we don't threshold it. A low similarity result goes to the model exactly the same as a high similarity one. In a regulated setting, a calibrated "I don't know" triggered at retrieval time, not at generation time, is a feature, not a gap. That's the next governance control to wire in.

• I did measure, and the numbers are what changed my thinking. We built a RAGAS evaluation harness covering faithfulness, answer relevancy, and context precision, validated against a 10 question golden dataset spanning DR failover, access control, patch management, incident escalation, and resource lifecycle topics. Faithfulness landed at 0.92, answer relevancy at 0.88, context precision at 0.95. Those scores are not a finish line, they're a baseline. What they immediately revealed: context precision is the strongest signal and answer relevancy is the weakest, which points directly at the retrieval step (top=1, no reranking) as the constraint. That's not a vibe. That's a number telling me exactly where to look next. Every change to chunking or retrieval going forward gets measured against those numbers before it ships.

If you are building RAG for anything where being wrong has consequences, my advice in one line: spend your effort on retrieval quality and the governance layer, not on making the answers sound smooth. We have the eval scores to show that groundedness is achievable at this scale. Smooth is easy. Trustworthy is the job, and now we have numbers to prove it.

I am Meet Patel - Developer Educator and applied-AI builder working on RAG/LLM systems and developer-facing content

• GitHub: https://github.com/meetp2022

• LinkedIn: https://www.linkedin.com/in/meet-patel-1b8160ab

• Portfolio: https://meetp2022.github.io/

If you're working on similar problems, I would genuinely like to hear how you are handling chunking and evaluation, that's where I am spending my time right now.