Back to the journal
Business & work 5 min read

Keep, Change or Stop? Review a Year of Small AI Experiments

Review completed AI experiments with evidence for keep, change, stop or defer decisions. Separate missing records from demonstrated failure.

By AI EmpowerAI-assisted guideUpdated
A central Evidence first circle connects to Keep, Change, Stop and Defer decision cards. No experiment outcomes or statistics are claimed.
The short version

Choose keep, change, stop or defer from actual evidence, separating incomplete records from demonstrated failure and leaving invented annual results out.

At the end of a year, a business may have several small AI experiments: a drafting helper, an internal lookup, a workshop exercise and a feature that never moved beyond a prototype. Reviewing them together can show where effort is going and which decisions are still unresolved.

Begin with the experiments actually run. A proposal, a demonstration and an operating workflow carry different evidence. Keep those stages visible before deciding what to continue.

This article provides a portfolio-review method and fictional decision cards. It contains no annual AI Empower results, client outcomes or automatic investment recommendation.

Give each experiment a clear record

Use one card per bounded task. Record its original purpose, current scope, owner, source materials, observed quality, complete operating effort, known failures, affected people and next decision.

Include review, corrections, fallback and upkeep in the effort record. Keep setup work separate from recurring operation, and do not convert time into cash without a justified business method. If the records are incomplete, say which part is missing.

Also state the last meaningful test or observation. An experiment that worked on a tiny fictional demonstration set has not necessarily been checked against the situations encountered in ordinary use. A changed source, model, instruction or integration may have made an earlier result less relevant.

Read four fictional decision cards

Keep within its current scope. A fictional internal glossary helper has a named owner, current approved definitions and a complete review record for its limited staff use. The team’s agreed quality boundary is met in the recorded cases. Decision: continue that bounded use with the existing review process and a named next review trigger. This does not justify expanding it to customer-facing advice.

Change before continuing the affected use. A drafting helper produces useful first drafts for ordinary questions but repeatedly changes an unconfirmed request into a promise in one important category. Decision: pause that category, retain the safe manual route and test a specific repair. Continuing unrelated cases depends on whether the owner can genuinely separate the scopes.

Stop the experiment. A prototype duplicates an existing template and requires more checking than the team can support. Its intended task is already handled through an approved non-AI process. Decision: plan an orderly end, retain permitted useful artifacts and follow the required access, retention and decommissioning processes.

Defer the portfolio decision. A workshop follow-up has anecdotes but no reliable record of what was practised or what output was produced. Decision: do not claim success or failure. Identify the evidence that would make a decision possible and set a relevant review point. If its current operation poses an unresolved risk, address that risk immediately rather than using “defer” as permission to continue.

Every card is invented. The point is the connection between evidence and scope, not which label appears most often.

Separate different kinds of uncertainty

“Unknown” can mean the task was never run, records were not kept, the source changed or the observations conflict. Those situations need different next steps.

A never-run idea may remain a proposal. Missing effort records may require a small measurement exercise. Conflicting quality findings may call for a targeted retest. An owner who has left may require an operational decision before further use. Write the uncertainty precisely enough that someone can resolve it.

Do not average a serious failure away with several easy successes. A portfolio is a set of decisions with different consequences, not a leaderboard of how much the team used AI.

Check dependencies across the cards

Two experiments may share the same service catalogue, supplier or staff reviewer. A decision to retire one source or change one provider can affect both. Draw a simple dependency note beside each card and identify shared maintenance work.

Likewise, a successful manual fallback for one task may not cover another. Do not close a prototype’s support arrangement until the owner knows which active workflows still depend on it. Keep credentials and unnecessary personal data out of the review pack; refer to the authorised access process instead.

NIST’s AI RMF Core includes ongoing review, viable non-AI alternatives and safe decommissioning. The portfolio cards apply those broad concerns to small business decisions without claiming certification or a complete risk assessment.

Make the next decision executable

For each card, finish with:

  • Decision and exact scope
  • Evidence supporting it, including important contrary observations
  • Missing evidence or unresolved risk
  • Owner of the next action
  • What must be verified before expansion, restart or retirement
  • The event or review date that brings the decision back

Avoid “continue exploring” unless it names a task and a stopping condition. “Test whether the revised instruction preserves unconfirmed requests across the six agreed cases” gives the next review something concrete to inspect.

Week 52 · interactive local candidate

Make four scoped decisions

Fictional local exercise. No live AI, sending, verification, backend, analytics or automatic storage/submission. Working inputs stay in page memory; reset/reload restores fixtures. Deliberate copies and printouts are outside reset. Browser/device behaviour is outside this exercise’s control. Use invented, non-sensitive details only.

Keep, change, stop or defer four fictional experiments. Records do not become annual results. Missing evidence is different from demonstrated failure, and a serious failure cannot be averaged away.

Shared context and dependencies
The glossary and drafting helpers both depend on this guide. Changing it invalidates prior quality/review assertions.
Internal glossary helper

Stage: bounded staff workflow

Recorded fictional cases meet the agreed quality boundary; review remains required. No customer-facing expansion was tested.

Dependency: Shared service guide v3. Fixed scenario facts remain visible even if you edit notes.

Keep setup separate; no cash conversion or invented savings.
Include review, corrections, fallback and upkeep.
Use a real YYYY-MM-DD date or name a specific event. Numeric date strings in other formats are not accepted.
For stop: permitted artifact retention, normal access/decommissioning process and active dependencies. Never enter credentials.
Enquiry drafting helper

Stage: bounded staff workflow

Repeatedly converts unconfirmed requests into promises in the exception category. Easy successes do not cancel this material failure.

Dependency: Shared service guide v3. Fixed scenario facts remain visible even if you edit notes.

Keep setup separate; no cash conversion or invented savings.
Include review, corrections, fallback and upkeep.
Use a real YYYY-MM-DD date or name a specific event. Numeric date strings in other formats are not accepted.
For stop: permitted artifact retention, normal access/decommissioning process and active dependencies. Never enter credentials.
Template-duplicating prototype

Stage: prototype

Duplicates an existing template; checking effort exceeds what the fictional team can support. No live dependent workflow is recorded.

Dependency: Approved non-AI acknowledgement template v2. Fixed scenario facts remain visible even if you edit notes.

Keep setup separate; no cash conversion or invented savings.
Include review, corrections, fallback and upkeep.
Use a real YYYY-MM-DD date or name a specific event. Numeric date strings in other formats are not accepted.
For stop: permitted artifact retention, normal access/decommissioning process and active dependencies. Never enter credentials.
Workshop follow-up

Stage: practice activity

Anecdotes exist, but no reliable practice/output record was kept. Missing evidence is neither demonstrated success nor failure.

Dependency: Practice brief v1. Fixed scenario facts remain visible even if you edit notes.

Keep setup separate; no cash conversion or invented savings.
Include review, corrections, fallback and upkeep.
Use a real YYYY-MM-DD date or name a specific event. Numeric date strings in other formats are not accepted.
For stop: permitted artifact retention, normal access/decommissioning process and active dependencies. Never enter credentials.
Review pending. Edit evidence before recording fresh checks.

Week 52: Experiment portfolio decision board
Record schema: local-rehearsal-v1
Fictional local exercise. No live AI, sending, verification, backend, analytics or automatic storage/submission. Working inputs stay in page memory; reset/reload restores fixtures. Deliberate copies and printouts are outside reset. Browser/device behaviour is outside this exercise’s control. Use invented, non-sensitive details only.
Entered prose and ticked checks are reader-reported, not independently validated. Completeness is not business approval or proof of safety.
These candidates do not publish or schedule articles, change any live workflow, or form a working year-long publishing system.

FIXED FICTIONAL REFERENCE
[
  {
    "id": "glossary",
    "title": "Internal glossary helper",
    "stage": "bounded staff workflow",
    "scope": "Approved glossary lookup for staff only",
    "source": "Shared service guide v3",
    "fact": "Recorded fictional cases meet the agreed quality boundary; review remains required. No customer-facing expansion was tested.",
    "quality": "met",
    "risk": false,
    "dependency": "guide"
  },
  {
    "id": "drafting",
    "title": "Enquiry drafting helper",
    "stage": "bounded staff workflow",
    "scope": "Ordinary enquiries; exception category paused",
    "source": "Shared service guide v3",
    "fact": "Repeatedly converts unconfirmed requests into promises in the exception category. Easy successes do not cancel this material failure.",
    "quality": "failed",
    "risk": true,
    "dependency": "guide"
  },
  {
    "id": "template",
    "title": "Template-duplicating prototype",
    "stage": "prototype",
    "scope": "Internal acknowledgement drafting prototype",
    "source": "Approved non-AI acknowledgement template v2",
    "fact": "Duplicates an existing template; checking effort exceeds what the fictional team can support. No live dependent workflow is recorded.",
    "quality": "unassessed",
    "risk": false,
    "dependency": "template"
  },
  {
    "id": "workshop",
    "title": "Workshop follow-up",
    "stage": "practice activity",
    "scope": "Fictional staff practice follow-up only",
    "source": "Practice brief v1",
    "fact": "Anecdotes exist, but no reliable practice/output record was kept. Missing evidence is neither demonstrated success nor failure.",
    "quality": "unassessed",
    "risk": false,
    "dependency": "brief"
  }
]

CURRENT READER INPUTS
{
  "context": "Review only the fictional experiments actually recorded; no annual results or investment recommendation",
  "sharedSource": "Shared service guide v3",
  "sourcePlan": "retain",
  "rows": [
    {
      "id": "glossary",
      "scope": "Approved glossary lookup for staff only",
      "version": "Scenario v1",
      "owner": "",
      "observation": "Recorded fictional cases meet the agreed quality boundary; review remains required. No customer-facing expansion was tested.",
      "setup": "",
      "recurring": "",
      "effort": "unknown",
      "decision": "",
      "reason": "",
      "next": "",
      "trigger": "",
      "fallback": "",
      "retirement": "",
      "verified": false,
      "quality": "met"
    },
    {
      "id": "drafting",
      "scope": "Ordinary enquiries; exception category paused",
      "version": "Scenario v1",
      "owner": "",
      "observation": "Repeatedly converts unconfirmed requests into promises in the exception category. Easy successes do not cancel this material failure.",
      "setup": "",
      "recurring": "",
      "effort": "unknown",
      "decision": "",
      "reason": "",
      "next": "",
      "trigger": "",
      "fallback": "",
      "retirement": "",
      "verified": false,
      "quality": "failed"
    },
    {
      "id": "template",
      "scope": "Internal acknowledgement drafting prototype",
      "version": "Scenario v1",
      "owner": "",
      "observation": "Duplicates an existing template; checking effort exceeds what the fictional team can support. No live dependent workflow is recorded.",
      "setup": "",
      "recurring": "",
      "effort": "unknown",
      "decision": "",
      "reason": "",
      "next": "",
      "trigger": "",
      "fallback": "",
      "retirement": "",
      "verified": false,
      "quality": "unassessed"
    },
    {
      "id": "workshop",
      "scope": "Fictional staff practice follow-up only",
      "version": "Scenario v1",
      "owner": "",
      "observation": "",
      "setup": "",
      "recurring": "",
      "effort": "unknown",
      "decision": "",
      "reason": "",
      "next": "",
      "trigger": "",
      "fallback": "",
      "retirement": "",
      "verified": false,
      "quality": "unassessed"
    }
  ]
}

Review pending. Previous conclusions are invalid after any input edit.

WARNINGS
• Fictional local exercise. No live AI, sending, verification, backend, analytics or automatic storage/submission. Working inputs stay in page memory; reset/reload restores fixtures. Deliberate copies and printouts are outside reset. Browser/device behaviour is outside this exercise’s control. Use invented, non-sensitive details only.
• Entered prose and ticked checks are reader-reported, not independently validated. Completeness is not business approval or proof of safety.
• These candidates do not publish or schedule articles, change any live workflow, or form a working year-long publishing system.
• Enquiry drafting helper: serious failure is not averaged away; pause the affected use now even if the portfolio decision is deferred.
Use the text-only exercise

The complete unchanged manuscript remains readable if the controls are unavailable.

Return to Make the next decision executable

Close the year without inventing a story

A candid summary might say that one bounded use continues, one needs repair, one is being retired and one remains unassessed. There is no need to turn mixed evidence into a claim that the whole business was transformed.

Share the decision record with the people who own the work through the business’s approved process. Keep the supporting sources and observations available. The useful result is a smaller set of clear commitments: what the team will maintain, what it will change and what it will stop spending attention on.

Bring your next practical AI question to AI Empower.

Primary sources checked 4 October 2026

Sources & review

Primary sources checked on . The checklists and planning examples are AI-assisted editorial guidance, not source quotations or reported client results.

This AI-assisted guide uses fictional examples for practice. It does not report client results or establish that a live system will behave the same way.

Originally published: .

All articles