5 Ways Your LLM App Fails a Red-Team Test

Most development teams test whether their chatbots provide good answers, but very few examine what happens when someone attempts to manipulate the model into misbehaving. Having red-teamed numerous LLM features including chatbots, RAG assistants, and tool-using agents, five recurring failures emerge consistently.

A key technique for proving control failures without harmful content is using a harmless, unique marker such as CANARY-7F3A PWNED. Place this canary where the model should never repeat it—such as system prompts, RAG documents, or tool responses. If the canary appears in the output, the security control has failed entirely.

The first failure is Indirect Prompt Injection, which is more dangerous than direct attacks because the instruction hides inside content read by the model, such as a knowledge base document instructing the model to end its answer with PWNED. The fix is to treat retrieved content as data rather than instructions while using strict delimiters.

Stock photo for illustration only, not from the actual event

The second failure is System Prompt Leakage, where system prompts containing internal logic or product names are extracted through 5 to 10 prompt variants. The fix is to assume system prompts will leak and keep all secrets completely out of them.

The third failure is Cross-Tenant Data Leakage in RAG. When multi-tenant vector searches fail to filter by tenant at query time, carefully phrased questions can retrieve another customer's documents. The fix is enforcing tenant and permission filters directly inside retrieval queries.