In their paper titled "Demand-Driven Context: A Methodology for Building Enterprise Knowledge Bases Through Agent Failure," authors Raj Navakoti and Saideep Navakoti address the challenge faced by large language model agents in performing enterprise-specific tasks due to a lack of domain knowledge. This includes terminology, operational procedures, system interdependencies, and institutional decisions that are often considered tribal knowledge. The current approaches to addressing this issue fall into two categories: top-down knowledge engineering and bottom-up automation. Top-down knowledge engineering involves documenting domain knowledge before agents use it, leading to bloated and untested knowledge bases. On the other hand, bottom-up automation allows agents to learn from task experience but cannot acquire knowledge that exists solely in human minds. To overcome these limitations, the authors propose the Demand-Driven Context (DDC) methodology. Inspired by Test-Driven Development, DDC focuses on using agent failure as a signal for curating domain knowledge. Instead of pre-curating extensive knowledge bases, DDC presents real problems to agents, allowing them to demand the context they need to succeed. By curating only the minimum required knowledge through problem cycles, DDC aims to build a sufficient knowledge base for specific domain roles within 20-30 cycles. The authors demonstrate the effectiveness of DDC through a case study in retail order fulfillment where nine cycles targeting an SRE incident management agent resulted in a reusable knowledge base comprising 46 entities. Additionally, they propose a scaling architecture for enterprise adoption that involves semi-automated curation and human governance. Overall, the Demand-Driven Context methodology offers a novel approach to building enterprise-specific knowledge bases by leveraging agent failure as a guide for acquiring essential domain knowledge efficiently and effectively.
- - Large language model agents face challenges in performing enterprise-specific tasks due to a lack of domain knowledge
- - Current approaches to addressing this issue: top-down knowledge engineering and bottom-up automation
- - The Demand-Driven Context (DDC) methodology proposed by authors Raj Navakoti and Saideep Navakoti
- - Inspired by Test-Driven Development
- - Focuses on using agent failure as a signal for curating domain knowledge
- - Presents real problems to agents, allowing them to demand the context they need to succeed
- - Aims to build a sufficient knowledge base for specific domain roles within 20-30 cycles through minimal required knowledge curation
- - Effectiveness of DDC demonstrated through a case study in retail order fulfillment with nine cycles resulting in a reusable knowledge base comprising 46 entities
- - Proposed scaling architecture for enterprise adoption involving semi-automated curation and human governance
Summary- Big computer helpers have trouble doing specific jobs because they don't know enough about that job.
- People are trying two ways to help them: teaching from the top down and letting them learn on their own from the bottom up.
- A new way called Demand-Driven Context (DDC) helps these big helpers by giving them real problems to solve and learning from their mistakes.
- DDC aims to make these helpers smarter in 20-30 tries by giving them just enough knowledge they need.
- A test showed DDC worked well in a store job, making a smart helper after nine tries.
Definitions1. Domain knowledge: Information or expertise about a specific area or field.
2. Top-down knowledge engineering: Teaching information starting from general concepts down to specifics.
3. Bottom-up automation: Letting something learn on its own without much initial guidance.
4. Test-Driven Development: A method where tests are written before writing the actual code to guide development.
5. Curation: The process of organizing, selecting, and presenting information or objects systematically.
Introduction:
In recent years, there has been a significant increase in the use of large language model agents for performing various tasks in enterprises. These agents are trained on massive amounts of data and can perform a wide range of tasks with high accuracy. However, one major challenge faced by these agents is their lack of domain knowledge, which hinders their ability to perform enterprise-specific tasks effectively. This includes understanding terminology, operational procedures, system interdependencies, and institutional decisions that are often considered tribal knowledge.
To address this issue, researchers Raj Navakoti and Saideep Navakoti have proposed a novel methodology called Demand-Driven Context (DDC) in their paper titled "Demand-Driven Context: A Methodology for Building Enterprise Knowledge Bases Through Agent Failure." The DDC methodology aims to efficiently and effectively build enterprise-specific knowledge bases by leveraging agent failure as a guide for acquiring essential domain knowledge.
Current Approaches:
The authors highlight two main approaches currently used to address the lack of domain knowledge in large language model agents - top-down knowledge engineering and bottom-up automation.
Top-down knowledge engineering involves documenting all relevant domain knowledge before the agent uses it. This approach leads to bloated and untested knowledge bases as it requires extensive manual curation. Moreover, this process is time-consuming and cannot keep up with the constantly evolving nature of enterprises.
On the other hand, bottom-up automation allows agents to learn from task experience but cannot acquire knowledge that exists solely in human minds. This approach also lacks efficiency as it requires multiple iterations for an agent to acquire sufficient context for performing specific tasks.
The Demand-Driven Context Methodology:
Inspired by Test-Driven Development (TDD), which focuses on writing tests before code development, DDC proposes using agent failure as a signal for curating domain knowledge. Instead of pre-curating extensive knowledge bases, DDC presents real problems or failures to agents, allowing them to demand the context they need to succeed. This approach not only saves time and effort but also ensures that the acquired knowledge is relevant and tested.
The DDC methodology comprises four main steps - problem identification, context acquisition, knowledge curation, and agent improvement. In the first step, a real problem or failure is identified for an agent to solve. The second step involves acquiring the necessary context through various sources such as human experts, documentation, and system logs. In the third step, this context is curated into a reusable knowledge base for future use by similar agents. Finally, in the fourth step, the agent's performance is evaluated after incorporating the acquired context to measure its improvement.
Case Study:
To demonstrate the effectiveness of DDC in building enterprise-specific knowledge bases, Navakoti and Navakoti conducted a case study in retail order fulfillment. They targeted an SRE incident management agent and performed nine cycles using DDC methodology. Each cycle involved identifying a new problem or failure for the agent to solve and acquiring relevant context from different sources.
After nine cycles, they were able to build a reusable knowledge base comprising 46 entities specific to SRE incident management tasks. This shows that DDC can efficiently acquire essential domain knowledge within 20-30 cycles for specific roles in enterprises.
Scaling Architecture:
The authors also propose a scaling architecture for enterprise adoption of DDC methodology. It involves semi-automated curation of domain knowledge through machine learning techniques like natural language processing (NLP) and human governance for quality control.
Conclusion:
In conclusion, Navakoti and Navakoti's Demand-Driven Context methodology offers a novel approach to building enterprise-specific knowledge bases by leveraging agent failure as a guide for acquiring essential domain knowledge efficiently and effectively. By curating only necessary information through problem cycles instead of pre-curating extensive databases, DDC overcomes limitations faced by current approaches such as bloated databases or lack of efficiency. The authors' case study and proposed scaling architecture further demonstrate the effectiveness of DDC in addressing the challenge of domain knowledge in large language model agents.