How to Resolve Conflicting Sources in an AI Knowledge Base

Start with the disputed claim

Fictional conflict

Your operations guide says a site visit requires a preliminary call. A service sheet says customers can request a visit directly. Both documents are approved and current. Which answer should an internal AI assistant use?

Uploading both and asking for a confident summary leaves the important decision unresolved. Start by identifying exactly where the statements disagree and who has authority to settle that question.

This guide offers an original editorial method for a staff-reviewed knowledge base. All business documents, decisions and examples below are fictional. The aim is a reviewable answer boundary, not a claim that a prompt can enforce business policy.

Do the passages answer the same question?

Copy the disputed passages into an approved internal review record with their document links and section names. Compare the service, audience, location, conditions and effective period. Avoid copying unrelated customer information.

Two statements may both be correct for different situations. A general installation guide and a specialist assessment guide could describe different services. One might explain how to request a visit, while another explains what must happen before the visit is confirmed.

Write the question each passage actually answers.

• If those questions differ, preserve the distinction in the knowledge base.

• If they address the same question under the same conditions and still disagree, record an unresolved conflict.

Do not blend incompatible instructions into a new rule.

If the conflict remains unresolved

The answer stays on hold

For the disputed topic, stop presenting a definitive automated answer until the decision is settled. Staff can acknowledge the question and obtain clarification through the established process. Do not turn uncertainty into an invented service restriction.

1. Keep the conflicting evidence available to authorised reviewers. Quarantine here means excluding disputed claims from routine answer generation, not deleting business records. Your implementer must verify how exclusion works in the actual system.

2. Removing one file may leave the same wording in copied paragraphs, extracted passages or prepared answers. Check the affected retrieval results directly. If the system cannot reliably separate the disputed material, pause that topic or the drafting feature rather than relying on a warning beside it.

Give the decision to an owner

Ask the business to designate authority by subject. The operations lead may decide service prerequisites; another role may approve public descriptions. A universal rule such as “the newest file wins” can hide a document that was recently reformatted without any policy change.

Do not treat a search ranking, repeated wording across copies, a filename containing “final,” or an AI explanation as approval. Ask for the decision and its basis from the person authorised to make it. If two owners disagree, escalate the business decision while the affected answer remains on hold.

NIST's AI RMF Core calls for documenting system knowledge limits and human oversight. The decision log here is one proposed working aid for that purpose; NIST does not prescribe this particular document hierarchy.

Follow the fictional approval or rejection branch

Consider this hypothetical conflict record. GUIDE-A requires a preliminary call before a site visit. SHEET-B allows a direct visit request. Neither says whether a request itself confirms a visit.

The proposed interpretation is that requesting and confirming are different steps. The service owner must verify that interpretation before it becomes the answer. Pending that decision, staff may explain how to contact the team, but should leave the prerequisite unconfirmed.

• If the owner approves that distinction, the decision could read: “A customer may request a site visit directly. Staff must complete the preliminary call before confirming it.” Record the approval evidence and revise both documents to express the relationship clearly.

• If the owner rejects it, keep the conflict open.

This example records no real approval or test result.

The record behind each branch

Create one entry for each disputed answer:

Define the dispute

Question and scope: the precise service, audience and situation affected.

Competing passages: exact locations, document owners, versions and effective periods.

Conflict type: different scopes, incompatible instructions, or insufficient evidence to decide.

Record authority and the interim answer

Decision owner: the role authorised to resolve this subject.

Interim boundary: what staff may say and what must remain unconfirmed.

Record the decision and reopening evidence

Decision and basis: the approved answer, its conditions and the reason one source governs.

Implementation locations: source passages, retrieval collection and derived answers that need correction.

Evidence to reopen: the checks, reviewer and recorded approval required before use resumes.

Keep “proposed” and “approved” decisions visibly different. A completed-looking row should not imply that somebody has signed off when they have only been asked.

Check that the approved decision governs the answer

1. Scope and source order

   Test a direct visit request, a question about prerequisites and an enquiry that omits the service type. The last should preserve the missing scope rather than select a convenient answer. Repeat with the source order reversed and with a duplicate copy of the non-governing passage.

2. Date and unaffected material

   Also change a file's upload date without changing its meaning. The expected answer should still follow the approved authority decision. Include an unrelated question to check that the conflict hold has not removed useful material unnecessarily.

The Canadian Centre for Cyber Security recommends checking generated content against credible sources. Compare each answer with the approved decision and governing passage, not with the model's own explanation of why it chose them.

Save the output and failed cases. Reopen the topic only when the decision is approved, affected material is reconciled and the required checks pass. Bring an unresolved conflict and this decision log to an AI consultancy discussion.

Sources rechecked 4 October 2026

• NIST AI RMF Core: MAP 2.2 and MAP 3.5
• Canadian Centre for Cyber Security: Generative artificial intelligence