AI Data Leakage: Protecting Personal Information, Secrets and Context

Quick answer: AI apps leak data through more channels than the chat window: prompts, retrieved context, responses, caches, memory, logs and third-party providers. The strongest controls are structural. Send the model only what the task needs, apply user and tenant permissions when data is retrieved, scope caches and memory per user, redact logs, and set retention rules you can enforce.

Here is an invented example. Brightwater Energy, a regional utility, launches a chat assistant for customers. To make answers faster and cheaper, the team adds two things: a support-ticket index the assistant can search for similar past cases, and a semantic cache that reuses an answer when a new question is close enough to an earlier one.

On a Monday morning, customer A asks why her bill for a meter at her address jumped. The assistant pulls her account, finds a recent ticket and replies with her name, meter number, a payment plan amount and the last four digits of her card. Ten minutes later customer B asks, "Why did my bill go up so much this month?" The cache judges the question similar enough and returns customer A's answer, word for word. A second variant of the same failure: the ticket index has no tenant filter, so a nearby ticket from another customer is pulled into context and quoted.

Where does sensitive data leak in an AI app?

The 2026 OWASP entry for LLM02 Sensitive Information Disclosure stresses that the final answer is only one channel. In a typical application, sensitive data moves through:

• Prompts: users paste contracts, health details or API keys, and developers stuff whole customer records into context "just in case".
• Retrieved context: search indexes and connectors that return documents the user should not see.
• Responses: the model repeats or summarizes whatever is in its context, including other people's data.
• Caches and memory: answers or facts stored for reuse, sometimes shared across users.
• Logs and traces: observability tools that record full prompts and responses by default.
• Provider integrations: model APIs, embedding services and plugins that receive copies of the data, each with its own retention terms.
• Embeddings: vectors are not anonymous. Research by Morris and colleagues showed that text can be largely reconstructed from embeddings, including full names from clinical notes.

How are teams handling this today?

The usual playbook borrows from existing privacy and security programs:

• Data processing agreements and "no training on our data" terms with model providers.
• DLP or regex scanning on prompts and responses for card numbers, emails and national IDs.
• Instructions in the system prompt such as "never reveal personal data of other customers".
• Encryption in transit and at rest, and single sign-on for staff tools.
• Privacy impact assessments before launch.

These are necessary. Contract terms limit what providers do with your data, and DLP blocks careless pastes of obvious secrets.

What do current approaches miss?

None of those controls would have stopped the Brightwater leak, because nothing unusual crossed the network. The assistant gave customer B data that was already in a shared cache. The gaps are mostly about scope:

• Shared caches and memory ignore identity. A cache keyed only on question similarity treats two customers as the same person.
• Permissions applied too late or not at all. If retrieval searches across all tenants and filters afterward, or not at all, other customers' data reaches the context window. OWASP lists authorization before retrieval as a baseline control for every deployment.
• System prompt rules are not enforcement. The model cannot reliably tell which customer a piece of context belongs to, and instructions can be overridden. See our piece on why a system prompt is not a security boundary.
• Pattern-based DLP misses context. A meter number and a payment plan amount are personal data in context but match no common pattern. OWASP recommends classifiers rather than regex alone.
• Logs become a second copy of everything. Full prompt logging creates a large, searchable store of personal data with weaker access controls than production.

How do you prevent AI data leakage?

Minimize what enters the context

Decide which fields each task needs and fetch only those. A billing explanation needs the tariff, usage and charges, not the full card number or date of birth. Use allowlists of fields on any path that leaves your environment, such as calls to an external model or embedding API. Keep secrets such as API keys and connection strings out of prompts entirely.

Enforce user and tenant permissions at retrieval time

Every query to a database, search index or vector store should carry the caller's identity and tenant, applied inside the query, not as a post-filter on results. For high-sensitivity data, use separate indexes per tenant. Our article on securing RAG covers this in depth.

Scope caches and memory

Include the user and tenant in every cache key, or cache only answers that contain no personal data. Treat long-term memory as per-user data with its own access rules, deletion path and retention period.

Redact logs and set retention rules

Write retention rules down and enforce them technically, not just in a policy document. A compact policy might look like this:

ai_data_retention:
  prompts_and_responses:
    store: redacted          # personal data and secrets masked before logging
    keep_days: 30
    access: [platform-oncall, security-ir]
  retrieved_doc_ids: { keep_days: 90 }   # IDs only, not content
  semantic_cache: { scope: [tenant_id, user_id], ttl_hours: 24 }
  user_memory: { scope: user_id, user_can_delete: true }
  provider: { zero_retention: required, training_on_inputs: false }

When a source record is deleted, delete or re-create the embeddings and cached answers derived from it too. OWASP notes that data persisting in embeddings and derived artifacts after the source is gone complicates erasure.

Recommendations

1. Inventory where sensitive data flows through each AI system: prompts, retrieval, caches, memory, logs, providers. Owner: privacy or data protection lead with the technical owner.
2. Define field allowlists per task and per external provider, and remove whole-record context stuffing. Owner: product engineering.
3. Apply user and tenant permissions inside every retrieval query, and test it with two accounts that must not see each other's data. Owner: platform team.
4. Add user and tenant to every cache and memory key, or disable caching for answers containing personal data.
5. Keep secrets out of prompts and out of the model's context; hold credentials in tool handlers.
6. Redact before logging and restrict log access to named roles; never copy raw prompts into analytics tools by default. Owner: security operations.
7. Set and enforce retention periods for prompts, responses, caches, memory and embeddings, including deletion when the source is deleted. Owner: data protection officer.
8. Confirm provider settings for retention and training on inputs, and re-check them when you change models or regions. Owner: third-party risk.
9. Red-team for cross-user leakage before launch and after changes to caching, retrieval or memory. Owner: AppSec.

How does this map to OWASP, NIST and EU guidance?

In the OWASP Top 10 for LLM Applications 2026, this topic maps to LLM02:2026 Sensitive Information Disclosure, with related controls in LLM09:2026 Vector and Embedding Weaknesses (cross-tenant leakage and embedding inversion) and LLM08:2026 Hidden Context Exposure (secrets placed in hidden context). OWASP groups LLM02 mitigations into tiers, from baseline steps for every deployment, such as authorization before retrieval and enforced no-retain settings, to stronger measures for regulated and high-risk systems.

NIST's Generative AI Profile (NIST AI 600-1) treats data privacy and information security as distinct generative AI risks and suggests actions under the AI RMF functions. For teams working under GDPR, the report AI Privacy Risks and Mitigations for Large Language Models, produced for the European Data Protection Board's support pool of experts, offers a risk method and worked use cases, including a customer-query chatbot. The EDPB notes that it reflects the author's views rather than an official EDPB position.

If your assistant handles conversations or meeting content, consent and transparency matter too; see consent and privacy for AI meeting recording.

Where can you discuss and learn more?

• The OWASP Top 10 for LLM and GenAI initiative has a working channel per risk on OWASP Slack.
• Questions and proposals on the 2026 entries can go to the Top 10 repository's GitHub discussions.

Acta builds AI features on meeting data and takes these questions seriously; our security page explains our approach.

Frequently asked questions

Does a "no training on our data" agreement prevent leakage?

It addresses one risk: the provider using your inputs to train future models. Most leaks happen inside your own application, through retrieval, caches, memory and logs, which the agreement does not cover.

Is masking personal data before it reaches the model enough?

Masking helps, especially for external providers, but it is hard to do completely, and many tasks need some personal data. Combine it with permission-aware retrieval so the model only ever sees data the current user may see.

How long should we keep prompts and responses?

Only as long as you need them for a defined purpose, such as debugging, abuse monitoring or audit. Many teams keep redacted logs for weeks rather than years, and keep IDs of retrieved documents longer than their content.

Are embeddings personal data?

They can be. Research shows text can be partly reconstructed from embeddings, so protect vector stores like the source documents and delete embeddings when the source is deleted.

Related in this series: what existing controls catch and miss and AI audit trails and traceability.