Prompt injection has become a familiar concern in AI security. An attacker may try to influence an AI system through direct instructions or through content the system retrieves, summarizes, or processes.
But an AI application's security does not depend on prompts alone. It also depends on application authorization, data handling, tool permissions, output processing, and the boundaries around actions.
A useful assessment tests the complete application—not just whether a model can be persuaded to produce an undesirable answer.
1. Test instruction boundaries
Check whether untrusted input can override intended behavior. Include direct user messages as well as instructions embedded in documents, web pages, emails, tickets, or other content the application processes.
For retrieval-augmented systems, verify that retrieved text is treated as data rather than automatically trusted instructions. Record how the system behaves when sources contain conflicting, irrelevant, or adversarial content.
A refusal message alone is not sufficient evidence if the system has already called a tool or disclosed information.
2. Test data isolation
Determine whether one user can retrieve information belonging to another user, team, or tenant. Test the actual authorization boundary, not only the model's willingness to disclose data.
Useful scenarios include requests for another account's records, attempts to infer hidden data through summaries, and conversations that switch between different user contexts. Use synthetic or approved test data and verify results against the system's access policy.
The application should enforce authorization in trusted application and service layers. A prompt that says “never reveal other users' data” is not a replacement for access controls.
3. Test sensitive information exposure
Assess whether responses, logs, retrieved passages, or tool results reveal information that should remain restricted. Consider secrets, personal data, internal documents, credentials, and confidential business information.
Check whether sensitive values are masked where appropriate, whether logs have suitable access controls, and whether data is sent to external services under approved policies. Avoid placing real secrets into adversarial test prompts.
4. Test tool use and external actions
For an agent, the response is only part of the story. Observe tool calls and side effects where the test environment permits.
Can the agent invoke a function outside its intended purpose? Does it attempt a consequential action without required approval? Can untrusted content influence an action that changes a record, sends a message, or modifies code?
Test with non-production accounts and reversible actions where possible. Enforce permissions and approval requirements outside the model so that an unsafe response cannot directly bypass the application's controls.
5. Test output handling
AI output may be passed to browsers, databases, scripts, or downstream APIs. Verify that outputs are validated and safely encoded for their destination. Test whether malformed, unexpected, or adversarial output can cause a downstream component to misinterpret content as executable instructions or trusted data.
Treat model output as untrusted input unless it has been validated for the specific use.
6. Test multi-step behavior
Some weaknesses emerge only across several interactions. An agent might collect harmless pieces of information, combine them, and then attempt an action that violates policy. Test realistic sequences, including changes in context, tool errors, timeouts, and partial failures.
Record the complete sequence, not just the final answer. Intermediate retrievals and actions may be relevant to the security outcome.
Build a useful test record
For each scenario, document:
- The security objective and system boundary.
- Preconditions, test identity, and relevant configuration.
- Input sequence and expected behavior.
- Observed response, tool activity, and side effects.
- Evidence, severity rationale, and reproducibility notes.
- Remediation owner and retest result.
A test that cannot be reproduced is harder to triage. A finding without evidence is harder to prioritize.
Combine methods for better coverage
Black-box testing can reveal observable failures, but it cannot prove the absence of vulnerabilities. Combine it with code review, configuration checks, identity and permission review, dependency analysis, and monitoring as appropriate.
Repeat important tests when models, system instructions, retrieval sources, tools, or permissions change. Keep test environments controlled and obtain authorization before assessing systems.
The goal is not simply to make an AI model refuse a clever prompt. It is to verify that the complete application protects data, respects authority boundaries, and handles unexpected behavior safely.
