1. Define the task before selecting the stack
A RAG initiative often begins with a broad aspiration: make enterprise knowledge easier to use. That is a sensible direction but an incomplete pilot brief. An employee looking up a procedure has different needs from an analyst comparing documents or a service agent preparing a response.
Choose a specific user group and a repeatable task. Describe what the user does today, which information they need and what a useful answer would contain. A good pilot question is narrow enough to evaluate but important enough that an improvement matters.
This framing also reveals when generation is unnecessary. If the task is best served by a relevant document and a clear excerpt, a strong search experience may deliver more value than a conversational answer.
2. Treat source readiness as product work
An index can only retrieve what the ingestion process makes available. Documents with outdated guidance, ambiguous ownership or inconsistent structure can create poor answers even when the search technology performs as designed.
Build a source inventory that records the owner, intended audience, format and update process. Decide how tables, attachments, scanned pages and duplicate versions will be handled. Identify which source should take precedence when similar documents disagree.
These decisions should be made with the people responsible for the knowledge. The pilot is an opportunity to establish a repeatable content process, not simply a one-time document upload.
3. Make authorization part of retrieval
An assistant should not become a shortcut around the permissions in the underlying source systems. Access must be considered when content is ingested and when a user asks a question.
Microsoft’s Azure AI Search documentation describes carrying permission metadata into an index and enforcing access at query time. The implementation varies by source and integration. A project therefore needs an explicit approach for identity, permission updates and content removal.
Test access behavior as a product requirement. Include users with different permissions and questions that would otherwise surface restricted content. A plausible answer is not an acceptable result if its evidence comes from information the user should not see.
4. Separate finding the evidence from writing the answer
When an answer is poor, the first question is where the failure occurred. Did the system fail to find the relevant passage? Did it retrieve the passage but misinterpret it? Was the source itself incomplete?
Evaluate retrieval and generation separately. Build a set of representative questions with expected sources, then examine whether the retrieved context is sufficient. Next, review whether the generated answer is supported by that context and whether its citations point to the correct evidence.
Include questions with no answer in the approved collection. The system should make missing evidence visible rather than turn a knowledge gap into an apparently confident response.
5. Decide who owns the system after the pilot
Knowledge changes. Employees change roles. Source documents are replaced. A successful demonstration does not answer who will manage these events after release.
Assign responsibility for ingestion failures, stale content, evaluation updates and user feedback. Agree which changes require a release review and how the team can revert a problematic configuration. Include latency and operating cost in the service conversation.
The useful output of a pilot is more than a working assistant. It is evidence about user value, technical behavior and the effort required to operate the solution. That evidence supports a more credible decision about whether to expand.
Before you start: identify the user, the task, the approved sources, the access rules and the person who will own the service after the demonstration.