← All posts PHILOSOPHY

The SafeForge AI Dogma — Why AI Shouldn't Make Safety Decisions

18 April 2026 · updated August 2026 · SafeForge

SafeForge is built on six principles that shape our architecture, our roadmap and every AI feature we ship. They are not aspirations — each one is enforced in code and process, and each one holds up against the AI rulebook as it stands today. The principles come first; the story of why they matter, and the clause-by-clause check against the rulebook, follow.

The dogma

The Six Foundational Principles

AI empowers the decision-maker. AI does not make the decision.
AI's contribution is explicit. You see when AI helped, and when it didn't.
AI operates under consent. Organisations opt in. Users don't surprise anyone.
Authority never transfers. However capable the assistance becomes, every determination remains the engineer's.
AI output is traceable. What was sent, what it cost, what the AI did — all logged.
AI does not bypass safety process. Change Requests, two-person review, audit trails — all apply equally to AI-assisted changes.

How each principle stands against the 2026 rulebook is further down this page.

The why

Why these principles — the five problems they answer

1. The work has outgrown the people

A modest rail signalling project carries hundreds of hazards, a thousand controls, two thousand requirements, and junction tables no one can hold in their head. And the raw scale is the easy part. The hard part is texture: a Systems Safety Engineer is rarely a domain expert in every discipline they must assess, service history lives in archives nobody has time to mine, and the whole record sits behind slow, configuration-managed database systems interlinked with equally archaic test-record systems. Every fact is retrievable and almost nothing is reachable. The practical consequence: hazard management becomes so hard to do continuously that “cleaning up the log” turns into an end-of-project justification task instead of something proactively managed throughout.

2. The deadline at the end of the project

When the dataset is beyond comprehension, the deadline fills the gap: the log is reconciled before an audit rather than maintained, and the safety case is assembled the week before a gate review. The Nimrod Review remains the canonical account of where that leads:

“The Nimrod Safety Case was a lamentable job from start to finish. It was riddled with errors. It missed the key dangers.”
“There has been a yawning gap between the appearance and reality of safety.”
“The task of drawing up the Safety Case became essentially a paperwork and ‘tick-box’ exercise.”
— Charles Haddon-Cave QC, The Nimrod Review (2009)

The people producing that safety case were not unsophisticated. What failed was process and tooling: when continuous engagement is painful, it does not happen, and the record becomes a thing to fix in time for sign-off.

3. Nobody can tell you what good looks like

Ask three safety engineers what a well-managed risk looks like and you will get three answers — not from confusion, but because the genuinely good hazard logs and safety arguments sit unpublished inside primes and consultancies, and the standards leave deliberate room for judgement.

“The biggest challenge is making good practice into common practice… we still haven’t cracked how to fully apply the knowledge and insight which is available and can demonstrably help.”
— Ken Rivers, The Chemical Engineer (2021)
“Accidents are not caused by lack of knowledge, but by a failure to use the knowledge that is available.”
— Trevor Kletz

4. Technical depth without lifecycle scaffolding

The profession has a wide gradient of experience. An engineer can be extraordinary on the failure modes of a particular sensor or the edge cases of an interface — and still be unsupported on the lifecycle questions: what evidence the next gate expects, what independence the verification needs, how to structure the argument a reviewer can follow. That knowledge is poorly published, and the mentors who carried it are retiring faster than they can be replaced. It is not a moral failing; it is an industry that has not given its people the scaffolding they need. This is where assistance is most useful — and most easily misused: a tool that gives the engineer reasonable knowledge at the point of decision can close the gap; a tool that “writes your hazard analysis for you” papers over it.

5. And suddenly, AI is here

Into exactly that landscape arrives generative AI. Tools like Copilot and Claude already give anyone a rapid way to generate content and claims and paste them straight into safety analyses and technical documents — with no governed, safe approach in between. A tool that can draft documents, suggest controls, reject controls and produce confident-sounding justifications in seconds is suddenly in everyone’s hands. And the evidence says people do not check it: in KPMG and the University of Melbourne’s global study, 66% of people rely on AI output without verifying its accuracy, while trust in AI has fallen, not risen. Ungoverned speed, applied to safety analysis, on systems whose failure hurts people — that is the moment the six principles exist for.

How SafeForge addresses it

We call it the SafeForge AI Dogma: AI empowers the decision-maker. AI does not make the decision. In practice that is the Confirmation Pattern: AI proposes, a person disposes, and the audit log records both. When AI drafts a hazard description, a control suggestion, or supporting evidence, the output goes into an editable field marked as unconfirmed. If you accept it, the Safety Expert makes the decision — credited to you and traceable in the audit log.

What AI should be in a safety-critical setting:

A deep-dive literature agent that finds controls and solutions you have not yet thought of, in addition to what you already know.
A reviewer that reads your hazard log overnight and surfaces the gaps in control coverage at 8am.
A rapid literature review of good practice for a specific control, right when you are deciding whether to adopt it.
An organiser that summarises and tracks the tasks and actions required for your next stage gate.
A quality-control aide that points you to the immediate actions where your risk-tracking standards are not being met.

In every case the AI’s contribution is a head start. The decision is still yours, and the accountability sits where it always sat: with the engineer.

What the dogma rules out

AI does not approve change requests. That needs a human reviewer, every time.
AI does not mark hazards closed. That is an explicit human action.
AI does not change data without a change request. Every edit travels the same two-person road your manual edits do.
AI does not run without consent, and access can be revoked at any time.
We will not market AI as “hazard analysis automation”. It is hazard analysis assistance.

A note on AI fallibility

AI gets things wrong — plausibly worded and technically off. The Confirmation Pattern is our defence: unconfirmed by default, read and edited before committing, references treated as user-verifiable rather than asserted fact.

The rulebook check

The six principles against the 2026 rulebook

We wrote these principles before most of today’s AI rules existed, so in August 2026 we checked each one against the current landscape: the EU AI Act (as amended by the 2026 Digital Omnibus), ISO/IEC 42001, the NIST AI Risk Management Framework, and Australia’s National AI Centre guidance. Plain terms, principle by principle.