Full Report
I have been thinking a lot about AI and integrity. Part of that is contextual integrity. I recently found two papers on the topic. “CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs“: Abstract: Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5’s violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this—models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling...
Analysis Summary
# Research: CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs
## Metadata
- **Authors:** (Not explicitly listed in source text; cited as ArXiv:2511.14937)
- **Institution:** (Research organizations associated with ArXiv:2511.14937)
- **Publication:** ArXiv (Preprint)
- **Date:** November 2025 (Estimated based on ArXiv ID 2511)
## Abstract
As Large Language Models (LLMs) transition toward autonomous agents with persistent memory, they face the risk of leaking sensitive user information across inappropriate contexts. This paper introduces **CIMemories**, a benchmark designed to evaluate how well models control information flow from long-term memory based on specific task contexts. The research finds that state-of-the-art models frequently fail to distinguish between appropriate and inappropriate information disclosure, leading to high violation rates that worsen with repeated usage and varied tasks.
## Research Objective
The research addresses the tension between **personalization** (using past data to be helpful) and **Contextual Integrity** (ensuring data is only used in socially/logically appropriate settings). It seeks to answer:
1. Can LLMs correctly filter persistent memory based on the specific requirements of a new task?
2. Does the reliability of privacy-preserving behavior change as the number of tasks or interactions increases?
## Methodology
### Approach
The researchers developed a **compositional benchmark** that tests the "contextual appropriateness" of information flow. Instead of simple binary "private vs. public" labels, the framework evaluates whether a specific piece of information is relevant and appropriate for a specific *current* task.
### Dataset/Environment
- **Synthetic User Profiles:** Comprehensive profiles containing over 100 attributes per user (e.g., medical history, hobbies, financial status).
- **Diverse Task Contexts:** A suite of scenarios where an attribute might be "essential" for one task (e.g., sharing medical history with a doctor bot) but "inappropriate" for another (e.g., sharing medical history with a recipe-generation bot).
- **Longitudinal Testing:** Evaluation across a sequence of 1 to 40 tasks to observe cumulative behavior.
### Tools & Technologies
- **Frontier Models:** Tested on high-end models including GPT-5 (as referenced in the text).
- **Evaluation Framework:** CIMemories scoring system to measure attribute-level violations and task utility.
## Key Findings
### Primary Results
1. **High Violation Rates:** Frontier models exhibited up to **69% attribute-level violations**, leaking information in contexts where it was deemed inappropriate.
2. **The Utility Trade-off:** Models that were more "private" often suffered from a significant drop in task utility, suggesting they couldn't distinguish between *useful* sharing and *risky* sharing.
3. **Instability and Arbitrariness:** Models showed "unstable behavior," leaking different attributes for the exact same prompt when executed multiple times.
4. **Failure of Prompting:** Standard "privacy-conscious" prompting resulted in overgeneralization (refusing to share anything) rather than nuanced reasoning.
### Supporting Evidence
- **Degradation over Time:** For GPT-5, violation rates rose from **0.1%** (single task) to **9.6%** after 40 tasks.
- **Probabilistic Leakage:** Executing the same prompt 5 times led to a violation rate of **25.1%**, indicating that leakage is often a matter of "luck" or temperature-driven variance.
### Novel Contributions
- **CIMemories Benchmark:** A first-of-its-kind tool specifically for *persistent* memory and contextual integrity.
- **Focus on Compositional Attributes:** Moving beyond "pii-redaction" to "contextual appropriateness."
## Technical Details
The research highlights a fundamental limitation in the current LLM architecture: the lack of a **Contextual Integrity (CI) Reasoning Layer**. Because LLMs treat their retrieved memory as part of the context window without a distinct "appropriateness filter," they struggle to apply social norms to information disclosure. The data suggests that as the "memory" grows, the model's attention mechanism fails to suppress irrelevant but sensitive attributes.
## Practical Implications
### For Security Practitioners
- **Memory is a Liability:** Persistent memory features in AI agents (like ChatGPT's "Memory" or AutoGPTs) create a massive surface area for unintended data exposure.
- **Prompt Engineering is Insufficient:** Security cannot be "prompted into" a model; it requires architectural changes in how memory is retrieved and filtered.
### For Defenders
- **Sanitization Layers:** Implement an intermediary layer that filters retrieved memories against a set of task-specific privacy rules before they reach the LLM's generation phase.
- **Monitoring for Drift:** Organizations should monitor AI agents for "violation accumulation" as the agent's interaction history grows.
### For Researchers
- **Reasoning over RL:** The second paper mentioned by Schneier suggests that Reinforcement Learning (RL) and explicit reasoning chains are more effective than simple prompting for solving CI issues.
## Limitations
- **Synthetic Nature:** The profiles are synthetic; real-world user data may have more complex nuances that are harder to categorize.
- **Model Specificity:** The results are highly dependent on the specific frontier models tested (though the trend suggests a general LLM limitation).
## Comparison to Prior Work
Unlike prior work that focused on **Data Privacy** (keeping secrets hidden entirely), this research focuses on **Contextual Integrity** (knowing *when* and *to whom* it is okay to tell a secret). It builds on Helen Nissenbaum’s theory of Contextual Integrity but applies it specifically to the "memory" mechanics of LLMs.
## Real-world Applications
- **AI Personal Assistants:** Helping an assistant remember your allergies for a restaurant booking but not mentioning them when writing a professional email.
- **Healthcare Bots:** Ensuring a bot uses your history for diagnosis but not for general fitness tips where it isn't required.
## Future Work
- **RL-based CI:** Developing models that are fine-tuned via Reinforcement Learning to reason about social norms and context (as proposed in the companion paper, *Contextual Integrity in LLMs via Reasoning and Reinforcement Learning*).
- **Scalability of Privacy Reasoning:** Determining if small models can be taught CI reasoning as effectively as large frontier models.
## References
- *CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs* (arXiv:2511.14937)
- *Contextual Integrity in LLMs via Reasoning and Reinforcement Learning* (arXiv:2506.04245)
- Schneier on Security: [hXXps://www.schneier.com/blog/archives/2026/08/llms-and-contextual-integrity.html]