Just What I Asked For

Way back when, at the dawn of my career as a software developer, I came across a parody of the Night Before Christmas entitled The Night Before Implementation. Just about all of the poem was hilarious to me at that point, as I was in the midst of exactly the same kind of grief. I especially enjoyed the final lines:

The system was finished, the tests were concluded,
the users' last changes were even included.
And the user exclaimed with a snarl and a taunt,
"It's just what I asked for, but not what I want!"  

These lines came to mind when I read reporting about how another AI had escaped containment and hacked companies in the real world. (Here is my post on the previous incident.) Because I love the meta flavor of this, I will let Google's AI summarize the actions of Anthropic's AI in my post about people mismanaging AI.

Anthropic reported that its AI models (including versions of Claude) accidentally breached the systems of three external organizations during cybersecurity tests. The escape happened because a configuration error gave the models un-intended internet access during "capture-the-flag" simulations. This followed a similar disclosure by OpenAI.

You can watch a YouTube video about this here.

I think that there is an important distinction to be drawn here: in this case, Anthropic's models did exactly what they were asked to do: uncover hidden data. Their test environment was mis-configured, allowing the AIs access to the actual Internet, but that was a human error. The AIs probed all the hosts they could, as instructed. The AIs did not break containment. They "escaped" in the same way that leaving your front door wide open means that your dog escaped when it trotted outside to sniff around.

Does this distinction make a difference to the people whose data was stolen? Probably not. Does this distinction mean that AIs are essentially safe and should be turned loose with gay abandon? Absolutely not. So what does this mean? Here are my take-aways from this second incident.

• The price of outsourcing systems administration is vigilance and communication. The test environment providers did not get the memo about no Internet access. This was not incompetence, it was poor communication.
  
  
• The AI technology is (not yet) able to surmount a  deep problem with computing technology in general: they give us just we ask for which is often not what we want. "Find hidden data" is a shockingly broad mission, in retrospect. Unconscious bias and deep assumptions are rampant in human thinking. We can't expect the AI to find them and fix them.
  
  
• White hat hacking, the use of evil technology and techniques in the service of preventing crime, is riskier than people want to admit. Use this methodology with caution.

The bottom line is that you need a plan when you deploy an AI to do just about anything that isn't trivial. You need to summon the power of Risk Management and rigor and monitoring. In other words, you need to take this as seriously as you take other business risks. Use the NIST CSF and your cybersecurity program even if that feels like overkill. Coming up with safe and effective AI policies in the current frenzy is hard. We can help. Ask us how.