ISG Software Research Analyst Perspectives

Embracing artificial intelligence technologies opens doors for innovation and efficiency. Alongside these opportunities, however, come risks. Threat actors are keenly aware of the potential impact of AI systems and are actively exploring ways to manipulate them. In this Analyst Perspective, I explore the world of adversarial machine-learning threats and provide practical guidance for securing AI systems. Generative AI applications introduce great potential to enterprises, bringing the intersection of innovation and security closer to business needs. Ventana Research asserts that by 2026, one-third of enterprises will establish governance to mitigate risks associated with GenAI software, ensuring ethical considerations, avoiding data and model bias, and safeguarding privacy and data security. GenAI systems can inadvertently generate biased or harmful content. Enterprises must ensure that ethical guidelines are in place to direct AI-generated outputs and prevent unintended consequences. GenAI models also learn from historical data, which may contain biases. Organizations need robust mechanisms to detect and rectify bias during model training and deployment. Plus, GenAI applications often process sensitive data. Implementing strong privacy controls and safeguarding data against breaches is essential. Striking a balance between innovation and digital security becomes paramount for enterprise adoption of GenAI software. Let’s explore the adversarial threats that enterprises must navigate to protect AI systems. A recent report by scientists from the National Institute of Standards and Technology and its collaborators identifies vulnerabilities in AI systems that adversaries can use to create malfunctions. The NIST report provides an overview of four major attack types that AI systems might suffer and corresponding approaches to mitigate damage. The report further classifies attacks according to multiple criteria such as the attacker's goals and objectives, capabilities and knowledge. The leading attack types include abuse, evasion, poisoning and privacy. Abuse attacks involve exploiting AI systems for unintended purposes. Using a language model to generate harmful or inappropriate content by compromising legitimate data sources is an example. Strict access controls, content filtering and regular audits can prevent abuse attacks. Evasion attacks aim to deceive AI models during inference by subtly modifying input data. For example, adversaries alter an image, such as a speed limit sign or road lane marker, to mislead an image-recognition system into misclassifying objects. Robust model architectures, input preprocessing and adversarial training can help mitigate evasion attacks. Poisoning attacks involve injecting malicious data into the training set to compromise model performance. An attacker, for example, modifies training data to bias a sentiment analysis model like a chatbot into utilizing inappropriate language. Rigorous data validation, outlier detection and monitoring during training are essential to prevent poisoning attacks. Privacy attacks exploit model outputs to infer sensitive data about individuals. A membership inference attack reveals whether a specific data point was part of the training set. By asking a chatbot legitimate questions, an attacker may use those responses to identify and exploit a weakness in the model. Differential privacy techniques, model aggregation and data anonymization enhance privacy protection. While enterprises invest significant effort in building robust AI models, attackers face lower barriers. Building attacks on AI systems requires less effort than constructing a robust AI model. Attackers can exploit vulnerabilities without the complexity of model development, training and optimization. Enterprises must recognize this asymmetry and prioritize AI security measures. A wealth of open-source tools and research papers are available for crafting attacks. Adversarial attack libraries, pre-trained models and tutorials are accessible online. Organizations must defend against these documented techniques. Threat actors can test their attacks against various models without rigorous validation. Enterprises often lack comprehensive testing for adversarial scenarios. Rigorous testing, including adversarial examples, is essential during model development. The potential impact of compromising AI systems drives attacker’s motivation. Enterprises may underestimate the attractiveness of AI assets to attackers. Organizations should assess the value of AI models from an adversary’s perspective. As organizations embrace AI, understanding the unique security challenges becomes paramount. Understanding the lower barriers faced by attackers is crucial for effective AI security. Enterprises must proactively address these challenges to safeguard AI systems and data. Essential practices to safeguard AI assets include: Leveraging existing frameworks. Extend existing governance and risk-management programs to cover AI. Rather than creating entirely new policies, integrate AI security considerations into your enterprise’s existing governance structure. This ensures consistency and alignment with overall risk-management practices. Risk assessment. Regularly assess AI-related risks. Understand the unique threats posed by AI systems, including adversarial attacks, data privacy concerns and model vulnerabilities. Update policies and risk assessments accordingly to address emerging risks. Stakeholder engagement. Involve non-technical executives and stakeholders in security discussions. Effective AI security requires collaboration across departments. Engage business leaders, legal teams and compliance officers to ensure a holistic approach to risk management. Quality control. Rigorously validate and curate training data. Poisoning attacks often exploit vulnerabilities in training data. Implement strict data validation processes to prevent malicious injections. Regularly review and clean training datasets. Data privacy. Implement privacy-preserving techniques. Sensitive data used for training AI models must be protected. Techniques like differential privacy can help safeguard individual privacy while maintaining model performance. Consider anonymization methods as well. Data retention. Define data retention policies. Minimize the exposure of sensitive data by retaining it only as long as necessary. Regularly review and update retention policies to align with changing business needs and compliance requirements. Adversarial training. Train models with adversarial examples. Adversarial training exposes models to intentionally crafted adversarial inputs during training. This helps improve model robustness and resilience against attacks. Monitoring and alerts. Continuously monitor model behavior. Set up alerts for suspicious activities, such as unexpected prediction shifts or sudden drops in performance that may indicate tampering. Regularly review model outputs to detect signs of an adversarial attack. Regular updates. Keep models up-to-date with security patches. Just like any enterprise software, AI models may have vulnerabilities. Stay informed about security updates and apply them promptly to mitigate known risks. The convergence of AI’s transformative potential and the increasing sophistication of threat actors requires a proactive approach to safeguarding AI systems. Securing AI systems is an ongoing journey that demands collaboration and adaptability. By following these best practices, enterprises can secure the AI watering hole and contribute to a safer digital future. Regards, Jeff Orr

Analyst Perspectives

Who Poisoned the AI Watering Hole: Adversarial Threats in AI

Our Analyst Perspective Policy

Our Analysts

David Menninger

Jeff Orr

Keith Dawson

Mark Smith

Matt Aslett

Matthew Brown

Robert Kugel

Stephen Hurrell

ISG Software Research

Subscribe to Email Updates

Posts by Month

Posts by Topic

Analyst Perspectives Archive