NIST Warns of Security and Privacy Risks from Rapid AI System Deployment
8.1.24  AI  The Hacker News

The U.S. National Institute of Standards and Technology (NIST) is calling attention to the privacy and security challenges that arise as a result of increased deployment of artificial intelligence (AI) systems in recent years.

"These security and privacy challenges include the potential for adversarial manipulation of training data, adversarial exploitation of model vulnerabilities to adversely affect the performance of the AI system, and even malicious manipulations, modifications or mere interaction with models to exfiltrate sensitive information about people represented in the data, about the model itself, or proprietary enterprise data," NIST said.

As AI systems become integrated into online services at a rapid pace, in part driven by the emergence of generative AI systems like OpenAI ChatGPT and Google Bard, models powering these technologies face a number of threats at various stages of the machine learning operations.

These include corrupted training data, security flaws in the software components, data model poisoning, supply chain weaknesses, and privacy breaches arising as a result of prompt injection attacks.

"For the most part, software developers need more people to use their product so it can get better with exposure," NIST computer scientist Apostol Vassilev said. "But there is no guarantee the exposure will be good. A chatbot can spew out bad or toxic information when prompted with carefully designed language."

The attacks, which can have significant impacts on availability, integrity, and privacy, are broadly classified as follows -

Evasion attacks, which aim to generate adversarial output after a model is deployed
Poisoning attacks, which target the training phase of the algorithm by introducing corrupted data
Privacy attacks, which aim to glean sensitive information about the system or the data it was trained on by posing questions that circumvent existing guardrails
Abuse attacks, which aim to compromise legitimate sources of information, such as a web page with incorrect pieces of information, to repurpose the system's intended use
Such attacks, NIST said, can be carried out by threat actors with full knowledge (white-box), minimal knowledge (black-box), or have a partial understanding of some of the aspects of the AI system (gray-box).

The agency further noted the lack of robust mitigation measures to counter these risks, urging the broader tech community to "come up with better defenses."

The development arrives more than a month after the U.K., the U.S., and international partners from 16 other countries released guidelines for the development of secure artificial intelligence (AI) systems.

"Despite the significant progress AI and machine learning have made, these technologies are vulnerable to attacks that can cause spectacular failures with dire consequences," Vassilev said. "There are theoretical problems with securing AI algorithms that simply haven't been solved yet. If anyone says differently, they are selling snake oil."