.NET: [Issue]: Enable agentic, in-run dynamic loading of tools

[vladbuinceanu]

Language

.NET

Type of Issue

Feature Request

Microsoft.Agents.AI

No response

Microsoft.Agents.AI.Abstractions

No response

Microsoft.Agents.AI.OpenAI

No response

Microsoft.Agents.AI.AzureAI

No response

Microsoft.Agents.AI.Anthropic

No response

Microsoft.Agents.AI.Hosting

No response

Microsoft.Agents.AI.Workflows

No response

Other .NET Packages

No response

agent-framework-core

No response

agent-framework-azure-ai

No response

agent-framework-anthropic

No response

agent-framework-azurefunctions

No response

Other Python Packages

No response

Description

The current ChatClientAgent design allows passing per-run configuration via ChatOptions. However, there is no clean or supported way to dynamically load new tools while an agent is already running, without stopping the agent and starting a new run.

What I am looking for is agentic behavior, where the agent itself can decide that it needs access to a certain “skill”, and then load that skill within the same run.

A “skill” in this context consists of:

• Additional instructions
• One or more tools (AIFunction instances)

---

Example flow:

1. Agent starts with a minimal instruction set and toolset
2. During reasoning, the agent determines it needs a specific skill (e.g. "weather", "data_analysis", "calendar_management")
3. Agent calls a tool such as get_skill("weather")
4. get_skill returns: Skill-specific instruction and a set of new tools
5. Agent immediately incorporates those instructions and tools and continues execution, same run, same agent

Code Sample

Additional Context

No response

[markwallace-microsoft]

@westey-m Can we do this via a context provider?

[westey-m]

@vladbuinceanu, we support this type of behavior via AIContextProviders. They are essentially plugins for ChatClientAgent that allows pre-processing of the user input using whatever mechanism is appropriate and can inject Instructions, Tools or additional Messages into the main run. The implementation of the AIContextProvider may of course include AI invocations.

You can see a sample implementation of an AIContextProvider here that is used to extract memories from the conversation and inject them automatically in future agent runs.

Also see this AIContextProvider from semantic kernel, which we have not yet ported to Agent Framework, but uses vector similarity search to achieve something similar. (note that the AIContextProvider interface in SK is different to AF).

[vladbuinceanu]

@westey-m Thank you for the answer, I looked into AIContextProvider and the code sample you shared. I assume you are referring to the InvokingAsync method to dynamically load instructions and tools, but this method still seems to operate before the agent starts generating its response.

What I am trying to achieve is slightly different, I’m looking for a way for the agent to decide during its internal reasoning / tool loop that it needs a new skill, and then immediately gain access to the corresponding tools and instructions, and finally uses the new tool and generates the answer for the user.

[westey-m]

I’m looking for a way for the agent to decide during its internal reasoning / tool loop that it needs a new skill

This is not really a feature of LLMs. If you don't advertise the list of functions available to an LLM, it's not going to ask for ones it doesn't know about. Even if you prompt the LLM, to ask for another tool if it doesn't have a tool it needs, it's rather unlikely that it is going to ask for a tool you have, unless you have told it what you have.

In cases where folks have too many tools available to advertise to the LLM in one go, this problem is typically solved by pre-processing, since you have a lot of control over just solving this one problem, i.e. what tools should I advertise to the LLM from the available list.

[vladbuinceanu]

In cases where folks have too many tools available to advertise to the LLM in one go, this problem is typically solved by pre-processing, since you have a lot of control over just solving this one problem, i.e. what tools should I advertise to the LLM from the available list.

That’s actually why I was inspired by Anthropic’s Advanced Tool Use . The idea is to not advertise everything up front, instead, the agent would know it can load certain skills when needed.

So rather than giving it 50 tools, the prompt just says it can load things like calendar_management, data_analysis, etc. When the agent realizes “I need calendar stuff,” it calls something like get_skill("calendar_management") and gets the instructions + tools for that skill, then continues in the same run.

So I want to move the reasoning at the skill level and injecting the actual tools only when they’re needed instead of loading all tools + definitions upfront.

[westey-m]

Ah, thanks for the example. That's quite interesting.

So, there is a search tool, which when called, rather than returning anything meaningful, results in more tools being advertised to the LLM on the next LLM invocation.

We do support modifying the ChatOptions during a function loop from within a Function, but it doesn't look like those modifications are currently taken into account for the actual function invocation, they are just taken into account for the advertising. Or at least, we currently cache a map of tools at the start of the function loop, and this map does not get refreshed if ChatOptions.Tools is modified.

CC @stephentoub

[stephentoub]

See dotnet/extensions#6865. cc: @MackinnonBuck

@westey-m, I'm unclear what's modifying what when. What ChatOptions is being mutated such that it'd be expected for https://github.com/dotnet/extensions/blob/99383237b2b86d4d85b4f5908b938e9e43c97db7/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs#L430 to pick up those changes?