The Three Problems With AI That Cybersecurity Leaders Can't Ignore

"The world needs AI. But there are big problems."

At the American Petroleum Institute's cybersecurity conference, held in Houston in November 2025, I opened my presentation with that line. This post explores a few of those problems.

AI has immense potential — but it has real problems too. Problems we have to understand if we want to exploit that potential safely and securely.

As cyber practitioners and researchers, we have a duty to recognize these problems before we let AI anywhere near our operations.

Problem 1: The Black Box

Picture this. It's February 2024. The smartest AI on the planet suddenly starts to lose its mind. ChatGPT — used by millions — begins speaking in gibberish. For hours, even its own creators can't explain what's happening.

OpenAI eventually traced it to a low-level bug in how the model selects words during inference, and published the explanation. But here's the part that should give every enterprise leader pause: almost no one downstream — no user, no regulator, no business relying on that model in production — could have diagnosed that themselves. The fix lived entirely inside a system only its creators could see into.

That's the real Black Box problem. Not that an explanation never exists, but that the explanation is locked inside the vendor, while everyone building on top of the model is flying blind.

Peek under the hood of a large language model, and you don't find readable code. You find billions of numerical weights across hundreds of layers — mathematical relationships even their own creators struggle to fully interpret. It's given birth to a whole new discipline: Explainable AI, where researchers are trying to turn mystery into understanding.

But the lesson for the rest of us is simple: AI is not yet accountable in the way the systems we're used to governing are.

Problem 2: AI Bias

AI learns from human data. And human data carries centuries of prejudice and stereotyping.

The danger isn't just that AI is biased — it's that it amplifies it.

A landmark 2024 study from UCL, published in Nature Human Behaviour, found something striking: people who interact with biased AI systems become more biased themselves over time.

That's the feedback loop:
→ Humans feed biased data.
→ AI learns and amplifies it.
→ Humans consume biased output.
→ Their own bias strengthens, feeding back into the next round of data.

In our enterprise world, this could mean maintenance models that quietly prioritize cost over safety, or incident prediction tools that under-represent data from certain facilities or regions — and then train the next generation of decision-makers to see that gap as normal.

Bias in AI doesn't just offend ethics. It can undermine an enterprise's entire safety culture.

Problem 3: The AI Arms Race

The most capable frontier AI systems in the world are still built by a small handful of companies — though that handful is no longer just the usual American names. Chinese labs have gone from a rounding error to a serious share of global AI usage in barely a year, and several more labs worldwide are now shipping frontier-class models in the same quarter, not the same decade.

This isn't a technology race anymore. It's an arms race — and it has more participants than ever, not fewer.

Everyone knows the risks. No one can afford to slow down.

[Check out my post on the recent Fable/Mythos shutdown risk to enterprises.]

The paradox: the faster the race, the thinner the safety margin.

That's the world we're inheriting — one where innovation is outpacing oversight. And that's exactly where we, as stewards of enterprise risk, have to lead with responsibility.

It means we have to harness the power of AI with the same rigor, safety, and resilience we already bring to manufacturing, transport, operations, finance, and legal.

What other AI problems have you run into? I'd genuinely like to hear them — share in the comments.