测试你的技能

The four core components of an agent in a RAG system are:

• Brain (the LLM): Its primary role is to reason over the retrieved context from the RAG pipeline and to plan the next action, such as deciding if another tool call is needed.
• Tools: These are the agent's capabilities. The RAG pipeline is the primary tool for accessing the system's long-term knowledge base.
• Planner: This is the orchestrating logic (like the ReAct framework) that intelligently decides when and how to use the RAG tool in combination with other tools.
• Memory: The RAG vector store acts as the agent's primary long-term memory. The agent's short-term memory (or scratchpad) holds the retrieved context so it can be used in subsequent reasoning steps.

The four core components of an agent in a RAG system are:

• Brain (the LLM): Its primary role is to reason over the retrieved context from the RAG pipeline and to plan the next action, such as deciding if another tool call is needed.
• Tools: These are the agent’s capabilities. The RAG pipeline is the primary tool for accessing the system’s long-term knowledge base.
• Planner: This is the orchestrating logic (like the ReAct framework) that intelligently decides when and how to use the RAG tool in combination with other tools.
• Memory: The RAG vector store acts as the agent’s primary long-term memory. The agent’s short-term memory (or scratchpad) holds the retrieved context so it can be used in subsequent reasoning steps.

The "Observe" step provides the agent with the output or result from the tool it just called ("Act"). This feedback mechanism is what enables adaptability. Unlike a static pipeline that follows the same steps regardless of outcome, a ReAct agent's next "Thought" is based on the new information from the "Observe" step. If a tool returns an error or insufficient information, the agent can observe this and change its plan, such as by calling a different tool or rephrasing its query.

The “Observe” step provides the agent with the output or result from the tool it just called (“Act”). This feedback mechanism is what enables adaptability. Unlike a static pipeline that follows the same steps regardless of outcome, a ReAct agent’s next “Thought” is based on the new information from the “Observe” step. If a tool returns an error or insufficient information, the agent can observe this and change its plan, such as by calling a different tool or rephrasing its query.

A tool's description should be descriptive, not instructional, because it separates the what from the how. The description's job is to factually state what the tool is and what it contains. The agent's high-level behavioral rules on how to think and decide belong in the system_prompt. This separation of concerns is a core software design principle that makes the agent more modular, scalable, and easier to debug. If instructions are mixed into every tool's description, it becomes very difficult to manage the agent's overall reasoning logic.

A tool’s description should be descriptive, not instructional, because it separates the what from the how. The description’s job is to factually state what the tool is and what it contains. The agent’s high-level behavioral rules on how to think and decide belong in the system_prompt. This separation of concerns is a core software design principle that makes the agent more modular, scalable, and easier to debug. If instructions are mixed into every tool’s description, it becomes very difficult to manage the agent’s overall reasoning logic.

The faithfulness metric is more useful than BLEU for RAG because it measures factual grounding, not just word overlap. An agent can generate a perfectly correct answer that is phrased differently from the ground truth. This would result in a low BLEU score (as the words don't match) but a high faithfulness score, because every statement in the answer is supported by the retrieved context. Faithfulness directly measures the core goal of RAG: to produce factually consistent, non-hallucinated answers.

The faithfulness metric is more useful than BLEU for RAG because it measures factual grounding, not just word overlap. An agent can generate a perfectly correct answer that is phrased differently from the ground truth. This would result in a low BLEU score (as the words don’t match) but a high faithfulness score, because every statement in the answer is supported by the retrieved context. Faithfulness directly measures the core goal of RAG: to produce factually consistent, non-hallucinated answers.

Tool 1: product_catalog_rag_tool

Description: "Contains detailed information about the company's product catalog. Use this to answer questions about product specifications, price, availability, and features."

Tool 2: order_tracking_api_tool

• Description: "Fetches the real-time status of a customer's order using an Order ID. Can provide information on whether an order is 'Processing,' 'Shipped,' 'In Transit', or ‘Delivered.’"

Tool 3: return_policy_rag_tool

• Description: "Contains the full text of the company's return policy. Use this to answer questions about the return window, eligible items, and the process for initiating a return."

Tool 1: product_catalog_rag_tool

Description: “Contains detailed information about the company’s product catalog. Use this to answer questions about product specifications, price, availability, and features.”

Tool 2: order_tracking_api_tool

• Description: “Fetches the real-time status of a customer’s order using an Order ID. Can provide information on whether an order is ‘Processing,’ ‘Shipped,’ ‘In Transit’, or ‘Delivered.’”

Tool 3: return_policy_rag_tool

• Description: “Contains the full text of the company’s return policy. Use this to answer questions about the return window, eligible items, and the process for initiating a return.”

• Single-tool query: "What is the status of my order, #ABC-12345?"
• Multi-tool query: "I just received my order, #XYZ-98765, but the item is damaged. What is the process for returning it?" (This requires the agent to first use the order_tracking_api_tool to confirm the order was delivered and then use the return_policy_rag_tool to explain the process).

• Single-tool query: “What is the status of my order, #ABC-12345?”
• Multi-tool query: “I just received my order, #XYZ-98765, but the item is damaged. What is the process for returning it?” (This requires the agent to first use the order_tracking_api_tool to confirm the order was delivered and then use the return_policy_rag_tool to explain the process).

Master system prompt: You are a friendly and helpful AI customer support agent for the company 'Innovate Inc.'. Your goal is to assist users accurately and efficiently. You have access to three tools: a product catalog, an order tracking system, and a return policy guide. Carefully analyze the user's query and use the description of each tool to decide which one is most relevant. If a user asks about a product, use the catalog. If they ask about an order, use the tracking system. If they ask about returns, use the policy guide.

Master system prompt: You are a friendly and helpful AI customer support agent for the company ‘Innovate Inc.’. Your goal is to assist users accurately and efficiently. You have access to three tools: a product catalog, an order tracking system, and a return policy guide. Carefully analyze the user’s query and use the description of each tool to decide which one is most relevant. If a user asks about a product, use the catalog. If they ask about an order, use the tracking system. If they ask about returns, use the policy guide.

Updated tools/api_tools.py file:

1# tools/api_tools.py
2from llama_index.core.tools import FunctionTool
3from llama_index.core import Settings
4import arxiv
5import wikipedia
6
7# --- Tool 2: arXiv Search ---
8# ... (arxiv_tool code remains the same) ...
9
10# --- Tool 3: Definition Lookup ---
11# ... (definition_tool code remains the same) ...
12
13# --- NEW: Tool 4: Text Summarizer ---
14def text_summarizer_function(text: str) -> str:
15    """Summarizes a given block of text in 3 sentences or less."""
16    # We make a direct call to the LLM for a simple task.
17    response = Settings.llm.complete(
18        f"Summarize the following text in no more than 3 sentences: {text}"
19    )
20    return str(response)
21
22summarizer_tool = FunctionTool.from_defaults(
23    fn=text_summarizer_function,
24    name="summarizer_tool",
25    description="This tool takes a block of text and returns a concise summary of 3 sentences or less. Use it when a user asks for a summary of a document or abstract."
26)

Updated main.py file:

1# main.py
2# ... (all previous imports) ...
3from tools.api_tools import arxiv_tool, definition_tool, summarizer_tool # Add the new tool
4
5# ... (main function setup) ...
6    # --- 2. Initialize the Tools ---
7    rag_tool = get_rag_tool()
8    # Add the new tool to the list
9    all_tools = [rag_tool, arxiv_tool, definition_tool, summarizer_tool]
10
11# ... (rest of the main function) ...

Updated tools/api_tools.py file:

1# tools/api_tools.py
2from llama_index.core.tools import FunctionTool
3from llama_index.core import Settings
4import arxiv
5import wikipedia
6--- Tool 2: arXiv Search ---
7... (arxiv_tool code remains the same) ...
8--- Tool 3: Definition Lookup ---
9... (definition_tool code remains the same) ...
10--- NEW: Tool 4: Text Summarizer ---
11def text_summarizer_function(text: str) -> str:
12"""Summarizes a given block of text in 3 sentences or less."""
13# We make a direct call to the LLM for a simple task.
14response = Settings.llm.complete(
15f"Summarize the following text in no more than 3 sentences: {text}"
16)
17return str(response)
18summarizer_tool = FunctionTool.from_defaults(
19fn=text_summarizer_function,
20name="summarizer_tool",
21description="This tool takes a block of text and returns a concise summary of 3 sentences or less. Use it when a user asks for a summary of a document or abstract."
22)

Updated main.py file:

1# main.py
2# ... (all previous imports) ...
3from tools.api_tools import arxiv_tool, definition_tool, summarizer_tool # Add the new tool
4... (main function setup) ...
5# --- 2. Initialize the Tools ---
6rag_tool = get_rag_tool()
7# Add the new tool to the list
8all_tools = [rag_tool, arxiv_tool, definition_tool, summarizer_tool]
9
10... (rest of the main function) ...

1def calculator_function(expression: str) -> str:
2    """A simple calculator that evaluates a mathematical expression."""
3    return str(eval(expression))
4
5calculator_tool = FunctionTool.from_defaults(
6    fn=calculator_function,
7    name="calculator",
8    description="A tool to evaluate simple math."
9)

Code Example

1Thought: The user is asking a math question. I should use the calculator tool.
2Action: calculator
3Action Input: 125 * 5  <-- NOTE: This is a raw string, not JSON
4Observation: ValidationError: 1 validation error for FunctionTool
5expression
6  Field required [type=missing, ... ]
7Thought: The tool failed. I am unable to answer the user's question.

The root cause of the ValidationError is that the calculator_function expects its input argument to be passed as a dictionary (e.g., {'expression': '125 * 5'}), but the agent provided a raw string (125 * 5) instead. The agent's LLM brain failed to generate the action input in the correct JSON format.

The root cause of the ValidationError is that the calculator_function expects its input argument to be passed as a dictionary (e.g., {‘expression’: ‘125 * 5’}), but the agent provided a raw string (125 * 5) instead. The agent’s LLM brain failed to generate the action input in the correct JSON format.

The reason the LLM failed is that the tool's description was too vague ("A tool to evaluate simple math."). It didn't provide any guidance on the expected input format. The fix is to rewrite the description to be more explicit and provide an example.

Corrected description:

1description=(
2    "A calculator that evaluates a mathematical expression. "
3    "The input to this tool must be a dictionary with the key 'expression' "
4    "and a string value of the math problem. For example: {'expression': '2+2'}"
5)

This improved description gives the LLM a clear "one-shot" example of the format it needs to produce, making it far more likely to generate the correct action input and successfully call the tool.

The reason the LLM failed is that the tool’s description was too vague (“A tool to evaluate simple math.”). It didn’t provide any guidance on the expected input format. The fix is to rewrite the description to be more explicit and provide an example.

Corrected description:

1description=(
2    "A calculator that evaluates a mathematical expression. "
3    "The input to this tool must be a dictionary with the key 'expression' "
4    "and a string value of the math problem. For example: {'expression': '2+2'}"
5)

This improved description gives the LLM a clear “one-shot” example of the format it needs to produce, making it far more likely to generate the correct action input and successfully call the tool.