AI, Adversarial Attacks, and Insider Threats in Life Sciences

Small to medium-sized businesses (SMBs) and enterprises in the life sciences industry already face a number of challenges; these attacks and threats to their AI systems present two more. Learn how to overcome the challenges, starting with an AI risk assessment.

Artificial intelligence (AI) plays a significant role in advancing research and clinical trials in the life sciences industry. But with the integration of AI comes significant security concerns, particularly adversarial attacks and insider threats.

Adversarial Attacks: Methods and Objectives

Adversarial attacks exploit weaknesses in AI and machine learning (ML) models. They are typically carried out by external attackers, but they might be launched by insiders with malicious intent.

These attacks often involve manipulating input data to AI models in subtle ways that cause the models to make incorrect predictions or decisions. Common examples include evasion attacks (modifying an image so an AI fails to recognize it correctly) and poisoning attacks (altering the training data to compromise the model’s integrity).

The primary target of an adversarial attack is the AI model’s algorithm and its input/output mechanism. The attack aims to exploit the model’s vulnerabilities without necessarily having direct access to the underlying system or infrastructure; undermine the reliability, integrity, or confidentiality of AI systems for financial gain; or to cause disruption.

Insider Threats: Methods and Objectives

Insider threats originate within an organization and involve employees, contractors, or anyone who has legitimate access to the organization’s systems and data. This access enables them to exploit a wider range of vulnerabilities across the organization’s digital and physical assets.

Insider threats represent a significant security challenge and encompass malicious, negligent, or unintentional actions. These attacks include data theft, sabotage, and introducing vulnerabilities or backdoors. Unlike adversarial attacks that manipulate an AI model’s input data, insider threats can target AI and ML models and directly alter systems, data, and infrastructure by way of access levels.

Sometimes unintentional actions result in security breaches, but the motivations behind insider threats include financial incentives, personal grievances, and espionage.

Adversarial Attacks and Insider Threats: How They Differ

While adversarial attacks and insider threats have a number of similarities, key differences are:

• Access: Insider threats come from those who have authorized access to an organization’s assets, while adversarial attacks generally come from those who do not, and they must find ways to breach security perimeters.
• Scope of Threat: Adversarial attacks exploit specific vulnerabilities in AI models, whereas insider threats encompass a broader range of malicious activities due to the attacker’s access to and knowledge of internal systems.
• Detection and Prevention: For adversarial attacks, these activities involve strengthening the AI models themselves and their input validation mechanisms. For insider threats, detection and prevention requires a comprehensive approach to monitor user activities, enforce strict access controls, and foster a security-aware culture in the organization.

While one of AI’s roles in life sciences is to help analyze data and perform tasks, generative AI (GenAI) is trained on datasets, uses the predictive aspect of ML algorithms, and gets help from large language models (LLMs) to create content like text and images.

Generally speaking, the lifecycle phases of a GenAI model include:

• Data collection
• Data preprocessing
• Training
• Validation
• Testing
• Production
• Monitoring and updating

Note: This is based on a CRISP Data Model updated for LLM. It’s a linear model; while some companies may not find it flexible enough for Agile development, it works well for describing LLM development phases in this blog.

Adversarial Attacks Against GenAI

Adversarial attacks during the GenAI lifecycle include poisoning, model inversion, evasion, and exploratory attacks. Following are examples of how these attacks could play out during various lifecycle phases of a GenAI model.

• Poisoning Attacks
  • During the Data Collection phase, attackers might tamper with clinical trial data sources and introduce biased or incorrect patient data to skew trial outcomes.
  • During the Data Preprocessing phase, alterations could introduce biases in clinical data and affect trial integrity.
  • During the Training phase when the model is evaluated against the test dataset, attackers could inject malicious data into the training set and cause models predicting drug efficacy or patient outcomes to learn incorrect patterns. This could lead to erroneous conclusions about a drug’s effectiveness or safety.
• Model Inversion Attacks
  • During the Validation phase, attackers might use model outputs to attempt to reverse-engineer patient data or sensitive trial information, which compromises patient and trial confidentiality.
  • During the Testing phase, attackers could extract detailed trial data or proprietary drug information from the model, which puts intellectual property at risk and could cause privacy breaches.
  • During the Production phase, attackers could extract sensitive patient data or proprietary information by analyzing model predictions on new data.
• Evasion Attacks
  • During the Production phase when GenAI is making decisions from input data, an evasion attack makes small, carefully crafted perturbations to the input data that cause the ML model to misclassify it. For example, slightly altering the molecular structure data of a compound to make an AI system misidentify a beneficial molecular structure as ineffective.
• Exploratory Attacks
  • During the Training phase, attackers could determine the model’s behavior to exploit vulnerabilities, which puts the integrity of clinical trial results at risk.
  • During the Validation phase, attackers might probe the model with validation data to discover vulnerabilities and undermine trial validity.
  • During the Testing phase, attackers might probe the model with test data to uncover vulnerabilities and expose critical trial results before official publication.
  • During the Production phase, attackers could determine how the model processes new trial data and compromise future trial data integrity.
  • During the Monitoring and Updating phase, attackers might continuously monitor model performance to identify and exploit new vulnerabilities and affect ongoing clinical trials and future drug development processes.
  • During any phase, exploratory attacks seek to understand the model’s behavior and find vulnerabilities without interfering with its operation. Attackers can use this knowledge to craft more effective evasion or poisoning attacks.

Insider Threats Against GenAI

Insider threats during the GenAI lifecycle include poisoning data, stealing models, tampering with models, exploiting vulnerabilities, and bypassing security controls. Following are examples of how these threats could play out during various lifecycle phases of a GenAI model.

• Poisoning Data
  • In the Data Collection phase, insiders could introduce incorrect patient data or outcomes to skew clinical trial results.
  • In the Data Processing phase, insiders could alter clinical data to introduce biases in trial outcomes.
  • In the Training phase, insiders could inject flawed data to mislead the trial’s safety or efficacy assessments.
• Stealing Models
  • During the Training phase, insiders could steal predictive models of drug efficacy or patient response.
  • During the Monitoring and Updating phase, insiders could exfiltrate sensitive models during reevaluation for regulatory submission.
• Tampering with Models
  • During the Training phase, insiders could modify models to falsely improve or degrade a drug’s projected performance.
  • During the Monitoring and Updating phase, insiders could alter models to influence ongoing trial adjustments or future trial planning.
• Exploiting Vulnerabilities
  • During the Inference phase, insiders could manipulate input data to the deployed model and obtain unapproved drug recommendations.
  • During the Monitoring and Updating phase, insiders could use known system flaws to access or alter trial data post-deployment.
• Bypassing Security Controls
  • During the Data Collection phase, insiders could access restricted trial data under the guise of necessary collection.
  • During the Data Preprocessing phase, insiders could modify data preprocessing tools to introduce or conceal trial data errors.
  • During the Training phase, insiders could gain access to modify trial protocols or data handling procedures.
  • During the Monitoring and Updating phase, insiders could disable or tamper with audit logs to hide unauthorized data access or alterations.

Note that the Validation and Testing phases primarily focus on evaluating the AI model’s performance against a known dataset that wasn’t used during the training phase. These phases are less about interacting with or manipulating the data and more about assessing and ensuring the model’s accuracy, generalizability, and robustness before deployment.

Two reasons for the lack of specific vulnerabilities are:

• Limited interaction with data: In these phases, the data used is already fixed and the main activity is running this data through the model to evaluate performance. Since there’s less active manipulation of data or models, the opportunity for traditional insider threats like data poisoning or model tampering is minimized.
• Objective evaluation: These phases aim to objectively measure the model’s performance. Malicious activities would likely be detected during these evaluations because they would affect the model’s accuracy, precision, and recall and make subversion more difficult without detection.

Understanding the Nuances of This Analysis

The pattern that emerges from this analysis suggests that insider threats are more prevalent and have a higher impact during phases involving active data manipulation, model development, and deployment.

Phases that are more about evaluation and less about direct interaction with data or models present fewer opportunities for insider threats. This doesn’t mean that Validation and Testing phases are immune to threats, but that the nature of threats may focus on undermining the integrity of the evaluation process itself or on indirect forms of sabotage or subversion.

Life sciences organizations must understand these nuances and implement comprehensive security measures across all phases of the AI lifecycle to effectively mitigate the risk of insider threats. This includes technical controls, policies, procedures, and training aimed at detecting, preventing, and responding to insider threats.

It’s important to understand and mitigate risks associated with AI technologies. USDM’s comprehensive risk assessment is based on the NIST AI Risk Management Framework, incorporates GxP guidelines, and delivers a thorough evaluation of these risks.

Safeguard your AI systems and ensure the integrity and reliability of your operations. Contact us today to schedule an AI risk assessment for your organization.