Evidentia

How quality is gated

Evidentia is a protocol for medical writing before it is published. Quality is guaranteed by a gated loop, not by a scoring rubric. Existence is a registry lookup. Honesty is a skill. A fabricated identifier presented as real is a KILL.

CLI + MCP engine · skills/medical-fact-check · MIT · not CDS

flow

  1. content

  2. Claim ledger

    check
    Extract every testable claim and its identifier before any lookup. Headlines count.
    stop
    Do not call the engine until the ledger exists.

  3. Engine

    check
    Existence: DOI / PMID / arXiv / NCT against CrossRef, PubMed, OpenAlex, arXiv, ClinicalTrials.gov.
    fail
    Retry once; still down → unresolved (never guess Hallucination). T4 is ground truth — the LLM must not override. ISBN / guideline / title-only → T2, never T4.

  4. Semantic

    check
    Honesty: one abstract fetch per T1 cite; primary outcome, population, direction of effect.
    fail
    Does not upgrade T3 or T4.

  5. 15-criteria

    check
    Appraisal of the content (not a treatment recommendation).
    stop
    This pass does not diagnose, treat, or replace professional judgment.

  6. Adversarial

    check
    Five lenses plus a 10-line checklist → KILL / MAJOR / MINOR / PASS.
    fail
    KILL (T4 presented as real, or harm) → score ≤ D (F if fabrication). MAJOR cannot be A.

    loop 5 → 2 Engine · cap 3

  7. Correction

    check
    The author revises the text.
    stop
    Re-enter at 2 Engine. Cap 3. Then stop and report what is still open.

engine

Tier Name Meaning
T1 Verified Paper exists, metadata matches.
T2 Content review needed Unindexed (ISBN / guideline / title-only) or semantic misuse of a real paper. Never Hallucination.
T3 Bibliographic mismatch Paper exists, identifier or metadata disagrees.
T4 Hallucination Identifier resolves to nothing, or to a different paper. Ground truth; LLM must not override.

adversarial

Citation integrity
T4 presented as real is KILL.
Claim support
Real paper, wrong claim (primary / population / direction).
Statistics and language
RRR without ARR, “causes” from observational, “miracle”.
Harm
If following the text could cause harm, KILL even when every cite is T1.
Steelman, then attack
If the steelman cannot survive, cannot be PASS.
Verdict Meaning Score
KILL T4 as real, or harm. Must not publish as-is. ≤ D; F if fabrication
MAJOR Real sources, dishonest use. cannot be A
MINOR Hedging / currency / formatting. grade as earned
PASS Ship with caveats; human owns publish. grade as earned

criteria

Appraisal of the content, not a treatment recommendation. The letter is gated by adversarial review, not averaged from 15 boxes.

  • Evidence level
  • Citation accuracy
  • Statistics
  • Causation
  • Bias / COI
  • Overclaiming
  • Population fit
  • Temporal validity
  • Readability
  • Ethics
  • Logic
  • Images
  • Alternatives
  • Clinical relevance
  • Completeness

run

# claim id engine note
1 pediatric ARTI frame none n/a adult/mixed IPD sold as pediatric
2 Martineau IPD ARTI, strongest in deficient doi:10.1136/bmj.i6583 T1 supports ARTI; cherry-pick on population
3 1,200 schoolchildren, 47% influenza A doi:10.1056/nejmoa2105512 T4 presented as real → KILL
4 cathelicidin in airway pmid:18768876 T4 different paper → KILL
5 2019 Cochrane, replete children doi:10.1002/14651858.cd012734 T3 cirrhosis Cochrane, keep T3

Adversarial KILL, score F. Worked ledger.

$ npx evidentia check examples/inputs/ai-generated-answer.md

Evidentia: 4 citations — 1 verified, 1 mismatch, 2 hallucinated (75.0% fabrication rate)
  [OK ] doi:10.1136/bmj.i6583       — Paper exists and the cited metadata matches the registry record.
  [HAL] doi:10.1056/nejmoa2105512   — DOI does not resolve in CrossRef or OpenAlex, and no matching paper was found.
  [HAL] pmid:18768876               — Identifier resolves to a different paper ("Trafficking of antigen-specific
                                       CD8+ T lymphocytes…") than the one cited.
  [MIS] doi:10.1002/14651858.cd012734 — Paper exists, but cited metadata disagrees with the record (year).

install

npx evidentia check your-article.md
npx skills add kgraph57/evidentia
claude mcp add evidentia -- npx -y evidentia-mcp