The risks associated with using agentic AI can be significant because agents can take actions, use tools, access data, and operate with limited or no human oversight. To prevent negative outcomes, it is helpful to describe these risks in more detail.
Excessive Privileges and Identity Abuse
AI agents often access multiple organizational systems, including Software-as-a-Service (SaaS) systems, cloud computing resources, application programming interfaces (APIs), and email. If a user is not careful to grant the agent the “least privileges” access, it can lead to situations in which the autonomous agent exceeds its authority when performing tasks. Additionally, if an attacker gains control of the AI agent, the attacker can direct the agent to perform tasks using excess privileges to change or exfiltrate data.
Data Leakage and Exfiltration
Data leakage can occur if AI agents have excessive privileges or expose data during transactions or integrations, in outputs, logs, memory, or downstream actions. Additionally, if agentic AI is compromised, the attacker can potentially access sensitive data through permissions and/or integrations that the agent is already authorized to use.
In Penn’s environment, this is particularly concerning for regulated data (e.g., FERPA, HIPAA) and other confidential data (e.g., PCI-DSS, human resource information, financial data).
Unauthorized Actions
Without predefined constraints on their actions, AI agents may use unintended tools or access sensitive data, leading to unpredictable behavior. Additionally, agentic systems may take actions such as modifying records, sending emails, approving financial transactions, or updating system configurations.
Prompt Injection
Prompt injection is the process of using malicious inputs or commands to manipulate AI systems into unintended or harmful behavior; this is often referred to as “agent hijacking.”
Attackers can insert malicious instructions via prompts or documents, or through systems the agent uses for its work.
Loss of Human Oversight
Users of agentic AI can trust them too much in a variety of ways, including failure to review outputs or missing security incidents, such as privilege escalation by the agent. The human-in-the-loop approach is important for mitigating this risk by embedding consistent human interaction and oversight into agentic AI.