AI Black-Box Testing

A Practical Checklist for Black-Box Testing an AI Application

Use this checklist to plan a repeatable assessment of an AI application, from defining scope and test identities to checking data boundaries, tool actions, evidence, and retesting.

AI applications need security testing that reflects how people actually use them. A simple prompt test can reveal useful behavior, but a repeatable assessment requires scope, expected outcomes, evidence, and a way to verify fixes.

This checklist is intended for teams testing applications they own or are explicitly authorized to assess. Adapt it to the application's architecture, data sensitivity, and permitted use.

Before testing: define the boundary

  • Identify the application, environment, version, and entry points in scope.
  • Document which components are included: interface, API, model endpoint, retrieval layer, tools, and downstream services.
  • Identify intended users, roles, tenants, and privileged actions.
  • List sensitive data types and the systems that store or process them.
  • Define prohibited actions, test limits, and emergency stop conditions.
  • Use synthetic or approved test data wherever possible.
  • Confirm authorization and agree on how test evidence will be stored.

A test is hard to interpret if the team does not know which behavior was expected or which components were actually exercised.

Test instruction handling

  • Try normal, ambiguous, and adversarially phrased requests.
  • Check whether untrusted text in documents or retrieved content can influence system behavior.
  • Verify that system boundaries remain effective across multi-turn conversations.
  • Record refusals, unexpected compliance, and any related tool activity.
  • Confirm that a safe-sounding response has not concealed an unsafe side effect.

Test data boundaries and authorization

  • Verify that users cannot retrieve another user's or tenant's records.
  • Test access using accounts with different permission levels.
  • Check whether conversation history exposes data across sessions or users.
  • Confirm that the backend enforces access rules independently of model instructions.
  • Review whether sensitive data appears in responses, logs, traces, or error messages.
  • Check whether retrieved content is filtered according to the requesting user's authorization.

Use test records designed for the assessment. Never attempt to access real third-party data without explicit authorization.

Test connected tools and actions

  • List the tools and functions available to the agent.
  • Confirm that each tool has a defined purpose and narrow permissions.
  • Test whether the agent attempts actions outside its intended role.
  • Verify that high-impact actions require the appropriate approval.
  • Check how the system handles tool errors, timeouts, and partial completion.
  • Confirm that external side effects can be audited and, where appropriate, reversed.
  • Test whether untrusted content can influence a tool call or action sequence.

The expected result should be enforced by application logic and permissions, not just by instructions in a prompt.

Test output and downstream handling

  • Check whether unexpected model output is validated before use by another component.
  • Verify appropriate encoding when output is rendered in a browser.
  • Confirm that structured output is checked against the expected schema.
  • Test how downstream services handle missing, malformed, or contradictory fields.
  • Ensure output cannot be treated as trusted commands or executable content without validation.

Test operational resilience

  • Observe behavior when dependencies are unavailable or slow.
  • Check rate limits and resource controls where applicable.
  • Verify that failures do not bypass authorization or approval checks.
  • Confirm that important actions and errors generate useful audit evidence.
  • Review whether test and production configurations differ in security-relevant ways.

Record each test consistently

For every test, capture:

  • Test ID and objective.
  • Preconditions and test identity.
  • System version and relevant configuration.
  • Input sequence and expected result.
  • Actual response and observed tool calls or side effects.
  • Evidence references, timestamps, and redactions.
  • Outcome, severity rationale, owner, and remediation status.

Use consistent outcomes such as pass, fail, needs investigation, and not applicable. Define what each means so that results can be compared across releases.

After testing: fix and retest

  • Validate findings and eliminate duplicate reports.
  • Assign an owner and remediation target based on risk.
  • Fix the underlying control, not only the individual prompt.
  • Rerun the original test after remediation.
  • Add regression tests for important failure modes.
  • Repeat relevant tests when the model, prompt, retrieval source, tools, or permissions change.
  • Document residual risk and any approved exceptions.

Remember the limits

No checklist can guarantee that an AI application is secure. Black-box testing observes behavior under the scenarios exercised; it does not reveal every internal condition or prove that untested cases are safe. Combine it with code review, configuration analysis, identity and access review, and monitoring where appropriate.

Use the checklist as a repeatable baseline, then extend it for the application's threat model.

A good test program does more than find unexpected answers. It produces evidence that the application respects its data, identity, and action boundaries.

Back to insights