One More Time: Annual Cybersecurity Training Does Not Work*

How many times have we at Pythia Cyber told you that annual cybersecurity training does not work? Answer: a lot. 

Go ahead & don't listen to us. But maybe you should listen to the people who study this.

A recent study by Ho and associates (link here) gave some users at UC San Diego Medical Center standard annual cybersecurity training, while other users got interactive training or were in a control group. In the standard training, participants watched an informational video about phishing attacks. There was no interaction opportunity or embedded quiz; the video was entirely static. In the interactive condition, participants were given a training in which there was an embedded quiz during the training video along with "tips" on avoiding cybersecurity incidents.

At least 12% of participants failed during the year following the standard cybersecurity training when a deliberately introduced phishing attack was launched. That is, at least 12% of all participants clicked on the compromised link. Here is the portion from the paper describing the results (quoted at length):

Based on the use of randomized controlled trials and our statistical models, we find that, in aggregate, users in the four training groups do have a statistically lower failure rate than users in the control group. However, our analysis indicates that this security improvement is quite small: on average, users in the training groups have only a 1.7% lower failure rate than those in the control group, and for several phishing campaigns, at least 10% of users in every group failed the simulated attack.

You'll note in that summary that among those in the control group (no cybersecurity trianing), the rate of clicking on the phishing link was only about 1.7 percentage points higher than among those who got any training. Thus, let's say your baserate of bad-link-clicking behavior is maybe 12%. That means that any type of training, in aggregate and using rough figures, lowered bad behavior rates by 2 percentage points.

That's right: your ROI on cybersecurity training is 2 percentage points.

For comparison, a recent study found that having medical doctors wear a smartwatch to help them improve their well-being was statistically effective. To rub it in: wearing a smartwatch to improve well-being is more effective, thus showing more ROI, than annual cybersecurity training.

And...here comes the asterisk. Ho and associates note a condition in which the training was more effective than standard static training, though still not overall effective: "Employees who completed interactive training sessions [as opposed to standard training] were less likely to fall for phishing scams in subsequent tests. Although these results show that more engaging and dynamic training can be more effective at enhancing employees’ cybersecurity awareness, the improvements produced by this training still fall short given how effective modern phishing attacks are.​"

Yawn? Here's something else we have noted that Ho and colleagues also note: participants were much more likely to click on phishing links that did not look routine ("dress code policy," "vacation policy") than they were on links that looked routine. Just to rub it in again, the proportions of participants who clicked on emails with either of these subject lines was between 27% and 31%. Note that the non-routine links seemed to contain information that was personally relevant to the participants. As we point out a lot, people will ignore policy when it's convenient for them to do so.

It's not that cybersecurity training should not take place. The point is that it's not going to stop some people, and that could be 12% of your work force, from engaging in bad cybersecurity behavior. If you think that a 2 percentage point reduction in potential cybersecurity incidents is meaningful ROI, well, more power to you. We conclude that basic cybersecurity training is good for shifting liability to employee users, which sounds great at a legal level but remember this: people whose information was stolen will sue you anyway because you and the organization, relative to the bad-acting employee, have more money. 

Ask us how you can get real results and better cybersecurity.