According to the LangChain blog, New Computer significantly improves the memory search system by integrating LangSmith, achieving 50% higher recall and 40% higher accuracy compared to previous baselines.
About the new computer
New Computer is the team behind Dot, the first personal AI designed to truly understand its users. Dot’s long-term memory system evolves by observing linguistic and behavioral cues, providing timely and personalized support to create a perception of true understanding.
A brief overview of Dot’s Agentic Memory
The innovative agent memory system developed by New Computer dynamically generates or precomputes documents for future retrieval. Unlike standard augmented retrieval generation (RAG) methods, this system structures information during memory generation, ensuring accurate and efficient retrieval as the memory accumulates.
Dot’s memory contains meta fields like status (e.g. completed or in progress) and date/time fields like start date or due date, which act as additional filters during high-frequency queries.
Improving memory retrieval using LangSmith
New Computer used LangSmith to rapidly iterate on a labeled example dataset. To maintain user privacy, synthetic data was generated, creating synthetic users with background stories generated by LLM. The team stored queries and available memory in the LangSmith dataset, labeled the associated memory for each query, and defined evaluation metrics such as precision, recall, and F1.
The experiment started with a baseline system that retrieved relevant memories using semantic search. A variety of techniques were tested to evaluate performance, including similarity search and keyword methods such as BM25. In some cases, pre-filtering by meta-fields was required for effective performance.
LangSmith’s SDK and experiment UI allowed New Computer to efficiently run and evaluate these experiments, and significantly improved the memory system.
Adjusting conversation prompts with LangSmith
Dot’s responses are generated by dynamic conversational prompts that integrate relevant memories, tool usage, and highly contextual action instructions. To optimize the prompts, synthetic users generated a wide range of queries, allowing the team to examine the global effects of prompt changes using LangSmith’s Experimental Comparison View.
In failure cases, we improved iteration speed while evaluating and adjusting conversation prompts by adjusting prompts directly within the LangSmith UI.
What’s next for new computers?
New Computer aims to deepen the relationship between humans and AI, and Dot continues to improve its ability to adapt to user preferences and provide personalized experiences. With a recent launch bringing in a new wave of users and a 45% conversion rate to the paid tier of the app, the partnership with LangChain and the use of LangSmith remain key to simulating complex human-AI interactions.
Image source: Shutterstock