Maker vs Breaker

Have you seen any of the recent spate of articles  on how easy it is to 'poison' a Large Language Model (LLM), also called a 'Predictive Artificial Intelligence'? 

Take this one for instance, from Anthropic. The title says it all:

"A small number of samples can poison LLMs of any size"

The once and future nerd in me yearns to expound at length on what it means to poison an LLM and why hacking LLMs is bad. But I gather that the former is beyond the scope of Pythia Cyber's outreach to business people and that the latter is, at some level, pretty obvious. So for the purposes of this post, I will just say that one poisons an LLM by hiding bad stuff in its input stream, which bad stuff is invisible to the human eye but very visible to the LLM's textual analysis. Or, to use Google's AI's rather bombastic wording,

LLM poisoning is a malicious act of intentionally injecting corrupted, misleading, or harmful data into a Large Language Model's (LLM) training dataset to manipulate its behavior, outputs, or decision-making process. [source]

It turns out that it is relatively easy to corrupt an LLM if you can get a tainted document (or 250) into that LLM's training data set. If you have been using Information Technology for any length of time, this is disappointing but not surprising.

However, I do not bring this up to give you yet one more thing to worry about when considering how LLMs fit into your short-term strategic business plans--although you should add this to your list of such concerns--but rather as a jumping off point to a more general issue: Makers versus Breakers and why the playing field is slanted toward the Breakers.

(I know that "makers" is now associated with a movement of technologists who enjoy a small scale engineering, but I need terminology here and common usage keeps robbing me of terminology as fast as that terminology arises. Once upon a time, programmers said "hacker" for good coder and "cracker" for evil coder, but somehow the common usage decided that "hacker" is bad and so that jargon no longer works. Which still irritates me to this day, but I mention this sometimes to avoid confusion when people stumble over older writing by programmers using this terminology. Similarly there were "white hat hackers" and "black hat hackers" but somehow we just have "hacker" and no word for "skilled technologist with a moral compass." So I am using Maker for good guy and Breaker for bad guy instead of hacker and cracker.)

I have always been a Maker: my career has been about applying technology to making businesses processes some combination of faster, easier, more efficient, and more effective. Early on in my technology development career I lived in an Eden of Making: it was hard enough to get things to work that all we worried about was getting new technology to work, then to deploy that new technology, then supporting that new technology and finally to replace that now-not-new technology with something better.

Then came the Fall and we were cast of out Eden and into the Internet.

In the beginning much of my  Making effort, after initial design and coding, went into trying to shield users and systems from honest mistakes, for example, asking "Are you sure? (y/n)" when a user requested that my systems print 1,000 copies of a large document. Gradually more and more of my effort was required to stop users (and administrators) from doing malicious things, from actively exploiting bugs instead of blundering into them. Ease of data exporting had to be balanced with data exfiltration. Ease of system access had to be balanced with unauthorized access. Authorization had to withstand not honest mistakes (eg mistyped user names) or even evil humans typing away at human typing speed but instead sophisticated attack (bots trying to guess passwords at super-human speed). Passwords had to be validated, and not just to avoid "password" or "123" or the clever "password123" but all "dictionary words" and ultimately  passwords had to be shored up by Multi-factor Authentication. All of which took time and effort away from pursuing a better User Experience.

Along the way I became aware of an unpleasant truth: it is generally much easier to break something than it is to make something. I could not avoid the analogy of the archetypal walled city: it is vastly more work to build mammoth stone walls than it is to smash or burn wooden gates. Given gunpowder, it is very much easier to breach a wall than it is to design, erect and maintain a wall. The trick in IT and cybersecurity, as in the rest of life, is to balance security and freedom.

Furthermore, in my experience, there is very little overlap between the Maker and Breaker mindsets. It is jarring for me to think about how to break something. It is unnatural. I don't enjoy doing it. The only cracking I have done has been taking databases in formats proprietary to defunct companies and exporting the data so that the owners of that data could access their data once again. So stop blithely telling your cybersecurity people to "think like hackers" because many of us are not good at it. We can do it when we have to but we will never be as adept at being criminal or malicious as criminals and malcontents are.

Also in my experience, people who enjoy breaking things tend to be better at breaking than they are at making. So hire the reformed evil-doer at your peril.

So if you are not a technologist, keep this painful truth in mind whenever are you tempted to spout off about how technologists keep making things that turn out to be breakable. Remember that those people  charged with protecting the City of Technology in which you live are swimming upstream and that we few of us have the luxury of focusing solely on protecting the city: most of us have make sure that the City has electric service and water service and sewer service and roads as well as walls. By contrast, those people who lay siege to the City have an easier task and fewer distractions; they are going to win sometimes. The goal may be perfect cybersecurity but the expectation cannot be and the responsibility cannot be the technologists alone. Be a good citizen of the City and have reasonable expectations.

Not sure how to turn this abstract advice into concrete policies and plans? Contact us. We can help.