MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

• Current LLM benchmarks evaluate memorization and action in isolation, failing to reflect how agents use memory to guide future decisions in real-world scenarios.
• MEMORYARENA is a new unified evaluation gym designed to test agents in multi-session 'Memory-Agent-Environment' loops.
• The framework uses human-crafted, interdependent subtasks that require agents to distill past experiences into memory for later use.
• Initial results show a significant performance gap, as models that excel at standard memory recall benchmarks struggle with memory-driven decision-making.

In realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks.

• Existing AI benchmarks often treat interaction history as a flat context, failing to evaluate memory beyond short-term context windows.
• The researchers propose a Memory-Agent-Environment loop where memorization and action are treated as inseparable components of behavior.
• The MEMORYARENA gym introduces interdependent subtasks where success requires tracking latent constraints across multiple sessions.
• New evaluation tasks in MEMORYARENA are intentionally underspecified to ensure agents must recall information from previous interactions to succeed.

We argue that agent memory should be evaluated by treating memorization and action as inseparable components of agentic behavior.

• MEMORYARENA is a new benchmark featuring human-crafted, multi-session tasks that require agents to track and apply information across interdependent subtasks.
• The framework spans four diverse domains, including web shopping, travel planning, information search, and sequential formal reasoning with average traces exceeding 40k tokens.
• While previous benchmarks focused on static retrieval or post hoc recall, MEMORYARENA evaluates whether agents can persistently store and actively utilize memory during execution.
• Experimental results reveal that current state-of-the-art agents fail to maintain latent task states effectively, leading to low completion rates in complex environments.

Despite their strong performance on existing memory benchmarks, these agents exhibit low task completion rates in MEMORYARENA, revealing persistent difficulties in maintaining and exploiting latent task state across sessions.

• Existing AI benchmarks primarily evaluate agents on single-session tasks, failing to measure the role of persistent memory across multiple episodes.
• Current memory benchmarks often focus on static retrieval or question-answering rather than the active application of skills in interdependent task sequences.
• MEMORYARENA introduces cross-task causal dependencies, requiring agents to absorb experiences and apply new understandings to future decisions.
• The benchmark covers diverse domains including web shopping, travel planning, and formal reasoning in mathematics and physics.
• By shifting from fact recall to sequential task completion, researchers aim to see if agents can truly learn from past actions to improve future performance.

MEMORYARENA is the first one designed to assess agent memory using sequential subtasks with causal dependencies across sessions.

• The Bundled Web Shopping environment simulates realistic scenarios where users make sequential, related purchases requiring memory of previous choices.
• Product compatibility serves as a primary constraint, such as ensuring a specific camera lens matches a previously selected camera body.
• Group Travel Planning tests an agent's ability to coordinate shared and individual preferences across multiple trip participants and travel days.
• Progressive Web Search tasks require agents to refine candidate lists by applying new constraints to information retrieved in earlier sessions.
• Formal reasoning environments in math and physics challenge agents to utilize previous derivations to solve complex, multi-step theoretical problems.

Later purchases depend on recalling attributes of earlier items to ensure compatibility and preference consistency.

• MEMORYARENA evaluates agents using interdependent subtasks that require the retention and integration of information across multiple sessions.
• The bundled shopping task uses coarse compatibility trees and fine-grained 'accept-reject' maps to test reasoning about product relationships.
• To ensure a unique solution, human annotators create multi-session instructions featuring incompatible distractors and specific selection constraints.
• Progressive web search tasks are designed to force agents to incrementally add constraints, preventing them from solving the problem in a single interaction.
• The evaluation framework deliberately filters out trivial tasks that do not demand long-term memory or cross-session information reuse.

Solving each session requires the agent to recall prior purchases, identify compatibility constraints, discard negative options, and select a valid product.

• The study implements a filtering phase to exclude queries solvable in one step, focusing purely on tasks requiring long-term memory.
• Researchers decompose complex queries into a series of subqueries that impose a strict causal ordering of information acquisition.
• The group travel benchmark models realistic logistics where participants join a trip incrementally and add potentially conflicting preferences.
• These traveler constraints create intricate dependency chains that require agents to reason about how new requests interact with previous decisions.

I want to stay at a hotel with at least a two-level higher rating than Rebecca’s.

• A new environment for formal reasoning uses research-level math and physics papers to test agents on long-context, multi-step logical arguments.
• Expert PhDs decomposed complex claims into ordered sequences of lemmas and propositions, ensuring that every statement is causally consistent with prior results.
• The test set includes 60 problems that mirror the structure of academic derivations, requiring the reuse of established conclusions over deep dependency chains.
• Evaluation differentiates between single-session tasks and multi-session interactions where agents must carry over information across temporally isolated steps.
• This benchmarking framework addresses the need for persistent memory in agents that must complete interdependent subtasks sequentially.

Verifying a single claim often requires pages of derivations and careful reuse of previously established conclusions, making this setting a natural testbed for evaluating long-term memory and multi-step formal reasoning.

• Complex agentic tasks are often divided into subtasks executed in separate, temporally isolated sessions.
• A persistent memory system is required to bridge these sessions, ensuring information like previous purchases or decisions informs future actions.
• The Memory-Agent-Environment Loop formalizes the process through two core functions: retrieval of task-relevant data and the updating of memory after subtask completion.
• While single-session agents can rely on implicit history, multi-session agents lose access to interaction traces once a session terminates, making explicit memory mandatory.
• This framework allows various memory architectures, such as RAG systems or long-context buffers, to influence an agent's memory-conditioned policy.

Task-relevant information must be selectively stored and retrieved through a persistent memory system in order to support decision-making in later subtasks.

• The MEMORYARENA benchmark evaluates how AI agents handle long-term dependencies across multiple interactive sessions.
• Agents are categorized by their memory architecture, including long-context buffers (0D), external memory with abstraction (1D), and RAG-based systems.
• The 0D memory method stores raw, verbatim history without consolidation, while 1D and 2D methods introduce learned or heuristic distillation mechanisms.
• Experimental results show a consistent decay in success rates as task dependencies span more sessions, suggesting a limit to current agent sustainability.
• High-end models like GPT-5.1-mini and Claude-Sonnet-4.5 were tested alongside specialized memory frameworks like MemGPT and GraphRAG.

The decay trend indicates agents cannot sustain execution as dependencies span more sessions.

• Researchers evaluated various memory systems, including flat, structured, and RAG-based models, within the challenging MEMORYARENA benchmark.
• The study introduces the Task Progress Score (PS) to quantify partial success by measuring the fraction of completed subtasks within a larger objective.
• A significant gap exists between subtask progress and total success, revealing that agents can often solve individual pieces but fail to integrate them into a cohesive whole.
• Group Travel Planning proved to be the most difficult environment, with nearly all methods resulting in near-zero success rates due to complex, interdependent constraints.
• The findings suggest that current AI agents struggle significantly with tasks that require maintaining global consistency across multiple sessions.

This pattern suggests that while agents can make some progress on individual subtasks, they fail to integrate these partial successes into globally consistent solutions dramatically.

• Group Travel Planning is the most difficult task due to its 30-slot itineraries and complex interdependent constraint chains.
• External memory and RAG systems do not consistently outperform raw long-context history due to representation and training mismatches.
• External memory is primarily helpful when tasks push agents beyond their reasoning capacity, reducing noise and attention saturation.
• Agent performance inevitably decays as subtask depth increases, showing that current models cannot sustain execution over deeply interdependent sessions.

Consequently, pairing strong long-context agents with external memory does not reliably produce a “1 + 1>2” effect.

• Neither long-context models nor external memory systems currently provide reliable support for long-horizon execution of interdependent tasks.
• Retrieval-based (RAG) systems prove more robust than external memory for tasks requiring the precise reuse of specific historical data.
• Long-context agents exhibit the lowest latency and remain competitive, whereas external memory systems incur the highest time costs.
• There is no systematic relationship between the complexity of a memory mechanism and its actual execution latency in practice.
• The MEMORYARENA environment models multi-session tasks as partially observable processes where the full state remains hidden from the agent.

Across sessions, the agent never directly observes the full underlying task state (e.g., the latent bundle specification, the evolving set of group constraints, or the intermediate dependencies required by later subtasks).

• MEMORYARENA frames multi-session agent interaction as a partially observable Markov decision process (POMDP) where agents lack direct access to full task states.
• The study identifies 'belief drift' as a primary cause of performance decay, where small errors in state estimation accumulate across sessions to ruin final outcomes.
• External memory is theorized as a tool for belief-state estimation, yet current state-of-the-art systems fail to support the rigorous state tracking required for complex tasks.
• Failures stem from two bottlenecks: memory systems optimized for generic recall rather than state updates, and agents untrained in using memory as structured cues for planning.
• The authors conclude that memory should be evaluated as a functional component of an agentic loop rather than through simple, isolated retrieval benchmarks.

small errors in the agent’s implicit state estimate accumulate across sessions and eventually dominate downstream decisions

• The academic references catalog a wide range of new benchmarks focused on memory and long-context capabilities for large language model agents.
• Current research is shifting from simple task completion metrics toward evaluating multi-session interdependence and episodic memory.
• System assessments like Agencybench are now testing the limits of autonomous agents using contexts as large as one million tokens.
• Innovative tools such as 'agent-as-a-judge' are being proposed to provide more transparent and automated evaluations of agentic search.
• The documents cite various projects aimed at making AI memory production-ready and scalable for real-world software engineering and web tasks.

Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

• The cited literature highlights a shift toward evaluating autonomous agents within extremely long context windows, reaching up to one million tokens in real-world scenarios.
• New benchmarks like Emembench and Agencybench are being developed to specifically test the episodic memory and reasoning capabilities of vision-language model agents.
• The conceptualization of Large Language Models as operating systems, exemplified by MemGPT, indicates a move toward more sophisticated memory management and retrieval architectures.
• Emerging research into 'self-evolving memory' allows agents to engage in test-time learning, enabling them to scale their reasoning memory during active tasks.
• Evaluation environments have evolved from isolated interactions to complex, multi-session scenarios that simulate realistic web browsing and interdependent planning tasks.

Memgpt: Towards llms as operating systems.

• The text details complex benchmarks designed to test the long-term memory and reasoning of autonomous AI agents.
• The 'Bundled web shopping' task requires agents to manage technical compatibility rules and budget constraints across a sequence of purchases.
• Decision-making in the shopping task is interdependent, where an initial choice like a 'gel cleanser' dictates which subsequent toners or treatments are compatible.
• The 'Group Travel Planning' benchmark utilizes structured environment tables to coordinate flights, dining, and attractions within a fixed budget.
• These tasks evaluate an agent's ability to maintain global rules and specific user preferences across multiple sessions and diverse data environments.

Compatibility notes: Gel pairs well with Astringent. Foam pairs well with Pore. Salicylic pairs well with Exfoliating.

• The Group Travel Planning task evaluates an agent's ability to synthesize specific budgetary and dietary constraints from multiple users into a single itinerary.
• Progressive Web Search benchmarks challenge agents to resolve complex, multi-layered queries by breaking them down into logical subqueries and search actions.
• Effective agent performance relies on interdependent multi-session memory to maintain consistency across evolving user requirements.
• The benchmarks highlight the difficulty of cross-referencing disparate data points like educational history, professional awards, and historical society founding dates.

A person who received their B.A. in a university different from where they received their postgraduate degrees got to name a location and was set to participate, at least up to July 2021, at a convention held by a society founded, up to April 2021, more than one decade ago but less than four decades ago.

• The Progressive Web Search task demonstrates how agents must synthesize disparate data points to identify a specific target entity.
• Formal reasoning modules require agents to solve interdependent mathematical lemmas where proofs depend on recalled outputs from earlier steps.
• Algorithms like MDL-Hedge-VC are introduced to facilitate multi-distribution learning and identify optimal hypotheses under various constraints.
• The benchmark structure evaluates an agent's ability to maintain precision and context over long-term, multi-session workflows.

The candidate matches educational disparity, location naming, and multi-author paper participation within the Brunswich/Geobiology context.

• The researchers filtered the WebShop dataset by selecting the top five root categories to minimize noise and ensure a robust data foundation.
• A hierarchical screening template was hand-crafted to enforce physical feasibility and logical self-consistency through dependency and reject maps.
• Large-scale retrieval from over one million items yielded tens of thousands of logically valid item chain combinations for the benchmark.
• Hard negative samples were deliberately included to test an agent's ability to recognize logically mutually exclusive constraints.
• Specific user preferences regarding price or rating are injected to identify a unique ground truth among multiple compatible candidates.

We selected 2 items that are logically mutually exclusive (satisfying the reject map) to serve as “hard negative” samples, thereby testing the model’s understanding of constraints.

• User preferences regarding price and ratings are injected to identify a unique ground truth from a set of compatible distractors.
• The system utilizes a standardized prompt framework to simulate real-world decision-making based on constraints and preferences.
• A final evaluation set consists of 150 high-quality, manually inspected samples to ensure data reliability.
• Memory is retrieved at the session level to optimize computational cost and frequency without sacrificing effectiveness.
• Specific agent protocols, like those for shopping, enforce global rules such as budget management and sequential task execution.

Evaluate All: Never pick the first option; compare all candidates.

• The group travel planning process relies on a 'base traveler' whose fixed itinerary serves as the initial state for subsequent agent-generated plans.
• A specialized memory agent captures execution traces, tool calls, and previous queries to inject relevant context into the model's current generation turn.
• Deep Research Agents utilize a progressive web search framework where reasoning is performed in an interleaved manner across multiple subqueries.
• Memory-guided search rules mandate that every new subquery must build upon the accumulated context of all preceding steps in the research process.
• Strict tool-invocation budgets and exact output formatting are used to benchmark the performance of agents in interdependent multi-session tasks.

Every subquery i must build upon the memory context of all preceding steps (1. . . i−1).

• Sequential Formal Reasoning involves solving math problems by using symbolic reasoning tools and storing reasoning traces back into a memory base.
• The BundledWebShopping task evaluates agents on their ability to manage interdependent purchases while adhering to global rules like budget limits and search styles.
• Memory integration is achieved by injecting retrieved or summarized history into a dedicated memory context block within the agent's input prompt.
• Experiments use a variety of frontier models, including Gemini-3-Flash and Claude-Sonnet-4.5, with strict token limits and timeout protection.
• The system enforces an iterative decision-making process to ensure that technical compatibility and user preferences are evaluated at every step.

Each task necessitates the agent to sequentially complete multiple purchase sub-goals (e.g., 6 items) within a single shopping scenario, while simultaneously satisfying global constraints.

• The study benchmarks agentic memory across multi-session tasks including progressive web search, bundled shopping, and formal logic.
• A specialized decomposition strategy forces models to break complex questions into self-contained subqueries that exclude pronouns to ensure retrieval accuracy.
• Mathematical and physical reasoning experiments require symbolic LaTeX output and strict temperature controls to maintain consistency and reproducibility.
• Latency analysis shows that different memory architectures, such as GraphRAG and Mem0, vary widely in their efficiency for interdependent tasks.
• Case studies identify critical logic failures in agents, such as 'impulse purchase' errors where agents fail to prioritize price, causing budget exhaustion.

Subqueries MUST be completely self-contained and answerable independently- do not use pronouns or references like "this person", "the author", "these conditions", "they", "the movie", etc.

• AI models exhibit different behavioral patterns during tasks, ranging from 'satisficing' by picking the first viable option to 'optimizing' through deep exploration.
• Higher-performing models like Claude-4.5-sonnet and Gemini-3-flash demonstrate the ability to backtrack and compare multiple products to maximize utility.
• Retrieval-Augmented Generation (RAG) systems can fail in sequential tasks when they do not capture specific, nuanced attributes from previous sessions.
• Long-context models generally outperform RAG-based systems in maintaining compatibility constraints across interdependent multi-step activities.

GPT-5.1-mini exhibits 'satisficing' behavior, purchasing the first relevant result (Option 1) immediately.

• The research compares memory architectures like MemGPT and long-context models using complex, multi-user group travel scenarios.
• Long-context models often suffer from 'lost in the middle' syndrome, failing to identify specific details within a massive stream of noise.
• MemGPT demonstrates superior precision in handling relative constraints by using high-density summaries to link cross-traveler dependencies.
• Despite its precision, MemGPT can fail if it does not successfully retrieve critical seed information, such as foundational trip dates and origins.
• The failure of different models across varying scenarios highlights the trade-offs between context density and retrieval reliability in agentic tasks.

Massive token input (20k+ chars) causes ”Lost in the Middle” and instruction drift.

• The Group Travel Planning case study illustrates how memory retrieval failures result in a 'drift' from finalized plans, such as incorrect flight dates and origins.
• A lack of access to previously retrieved data leads to downstream constraint violations, exemplified by a dinner selection that ignores price limits.
• The Progressive Web Search case study tracks model performance across subqueries, identifying Black Sabbath as a band that recorded their debut in a single day.
• Models are tested on their capacity to store and recall specific details like founding members and solo career milestones across multiple sessions.
• Complex multi-step reasoning remains a challenge, as seen when models fail to link biographical constraints to identify a specific album cover designer.

A memory retrieval failure causes drift from the finalized seed plan (wrong date/origin in flight search) and has a downstream constraint violation when selecting dinner.

• The study evaluates the performance of different AI model variations in retrieving and synthesizing information from multi-session search tasks.
• Case Study 1 reveals that even advanced models struggle to link disparate biographical facts to a single founding member of a band.
• Case Study 2 follows a complex query about a Ghanaian doctor, Matthew Arnum Barnor, who studied in Scotland during World War II.
• Model failures are often attributed to context drift and memory noise, where irrelevant data from previous sessions interferes with current tasks.
• Specialized versions like ReasoningBank GPT-5-mini employ protocols to request missing identifiers rather than providing incomplete answers.

I was unable to find any reliable source that ties all of those specific biographical and discographic constraints to a single identifiable founding member.

• The text presents a case study evaluating different AI memory systems through a complex query about a Ghanaian doctor who studied in Scotland.
• While systems like Mem0 and ReasoningBank successfully identified Dr. Matthew Arnum Barnor, the Long Context model failed due to 'semantic drift.'
• The failure occurred because the model's memory was cluttered with irrelevant 'noise' concerning 19th-century maritime disasters like the SS Edmund Fitzgerald.
• The research emphasizes the challenge of preserving context across multi-session tasks without losing accuracy to distracting historical data.
• The document also introduces sequential formal reasoning through mathematical lemmas involving weighted subsets and index segments.

XML-wrapped history contains noise regarding 19th-century maritime disasters (Schooner Abraham Newland 1801; Capt. Morgan).

• The segment conditions facilitate a structured analysis of index sets and weights across discrete time intervals.
• High-probability guarantees link the performance of hypotheses computed in round-based algorithms to the true population loss.
• Uniform convergence bounds are established using VC dimension to control the deviation between empirical estimates and actual values.
• The methodology provides a tight upper bound on the maximum loss over multiple distributions and loss function categories.
• The integration of the Hedge algorithm allows for effective multi-distribution learning with high-confidence statistical results.

With probability at least 1−δ/2 there exists an error term ε(depending on the VC dimension dofH, the sample size m, k and|L| andδ) such that for all i∈[k], ℓ∈ L and all h∈H we have |Lℓi(h)−ˆLℓi(h)| ≤ε.

• The document compares the performance of different AI memory systems, such as MemGPT and Mirix, through complex mathematical case studies.
• The analysis involves technical proofs that utilize uniform convergence and empirical minimizer properties to bound loss across multiple distributions.
• The Hedge algorithm is implemented over multiple rounds to minimize regret against the best fixed distribution in a hypothesis space.
• Subqueries evaluate the ability of memory-augmented agents to recall and apply specific lemmas to solve sequential formal reasoning tasks.
• The study demonstrates how sample complexity for VC classes can be bounded with high probability when specific algorithmic parameters are met.

Running Hedge across T rounds with step size eta and applying the Hedge regret bound gives that the average regret against the best fixed loss/distribution is small.

• The analysis employs VC uniform convergence to bound deviations between empirical and population losses across multiple distributions and hypotheses.
• A mathematical lower bound is established for (p,q,x)-segment lengths, demonstrating logarithmic dependency on distribution and loss types.
• By integrating Hedge algorithm regret guarantees with deviation bounds, the final output policy is proven to approximate the minimax loss.
• Detailed sample-size requirements are specified, ensuring high-probability performance based on VC dimension, distribution count, and desired accuracy.

Using the uniform deviation bound on both sides of the inequality in step 2 we get with probability at least1−δ...

• The text specifies the necessary sample size for Algorithm 2 to achieve reliable performance across varied data distributions.
• These generalization bounds are calculated based on the VC dimension and the total number of loss functions involved.
• If the sample complexity exceeds the calculated threshold, the algorithm's output is guaranteed to reach a near-optimal state.
• The resulting policy minimizes the worst-case expected loss within a specified accuracy margin with high probability.
• This mathematical trace confirms that the output policy satisfies the uniform generalization bound under stated conditions.

In other words, the policy output by Algorithm 2 achieves the near-optimal worst-case expected loss across all distributions and losses, within an additive (ε) margin, with high probability, provided the sample complexity exceeds the above threshold.