Support for Alternative LLM Providers (Beyond OpenAI)

[pravintargaryen]

I wanted to ask if there's any plan to support alternative LLM providers, such as:

• Local models via vLLM, llama.cpp, or Ollama
• Other cloud providers like Mistral, Together AI, or Cohere

This would provide more flexibility for users who prefer self-hosted or non-OpenAI options. Would you be open to adding support for this? I'd be happy to help test or contribute if needed.

Looking forward to your thoughts!

[georg-wolflein]

Hi @pravintargaryen, that's an excellent suggestion! As we are using litellm to perform the LLM calls, this should not be too difficult. In fact, all LLM requests are made via the LLM.completion function in toolmaker/llm/__init__.py:

ToolMaker/toolmaker/llm/__init__.py

Lines 129 to 136 in d37e0db

	def completion[T: BaseModel | str](
	self,
	model: str,
	messages: Sequence[ChatCompletionMessageParam],
	*,
	response_format: type[T] | None = None,
	tools: Sequence[ChatCompletionToolParam] | None = None,
	) -> ParsedChatCompletionMessage[T]:

Specifically, this is the part where the LLM call is made:

ToolMaker/toolmaker/llm/__init__.py

Lines 158 to 163 in d37e0db

	response = completion_factory(model)(
	model=model,
	messages=list(messages),
	tools=tools,
	response_format=response_format if is_typed else None,
	)

Here, completion_factory(model) usually is just litellm.completion.

If you have a local LLM that supports function calling and structured outputs, you should be able to modify this file to use the local LLM instead. I would happily accept a PR for local LLM support and/or support for configuring different LLM model providers (right now the LLM models are set via environment variables here).

I am also happy to support if you run into problems, but I currently do not have the bandwidth to implement/test this with local LLMs.

PS: I tried to get ToolMaker to work with Claude 3.5, but Claude does not support using structured outputs and function calling at the same time. For this, I created a dirty workaround, where I used Claude with function calling, giving it an extra function __finish__ that would act like a structured output. However, I was not yet able to extensively test how well this approach works.

[pravintargaryen]

@georg-wolflein Thanks for the detailed response! It’s great to hear that litellm is already being used, which should make this easier to implement. I’ll look into modifying completion_factory(model) to support local LLMs and make the model provider configurable via environment variables.

I'll start by testing with LLM Model providers and see how well function calling and structured outputs work. If I run into any issues, I’ll reach out for guidance.

Appreciate the support, and I’ll update you as I make progress!

[pravintargaryen]

Hi @georg-wolflein,

I’ve written a simple local_llm function to support function calling and structured outputs with a local LLM server. It uses an endpoint (configurable via LOCAL_LLM_ENDPOINT) to send requests and dynamically appends the JSON schema if response_format is provided.

Here’s the current implementation:
`
def local_llm(
model: str,
messages: Sequence[ChatCompletionMessageParam],
tools: Sequence[ChatCompletionToolParam] | None = None,
response_format: type[BaseModel] | None = None,
) -> ModelResponse:
"""Call a local LLM server (e.g., Ollama, vLLM) with function calling and structured responses."""

LOCAL_LLM_ENDPOINT = os.getenv("LOCAL_LLM_ENDPOINT", "http://localhost:8000/v1/chat/completions")

payload = {
    "model": model,
    "messages": [dict(msg) for msg in messages],
    "tools": tools or [],
}

# Handle function calling (if tools are provided)
if tools:
    payload["tools"] = [dict(tool) for tool in tools]
    payload["tool_choice"] = "required"

# Handle structured output (if response_format is provided)
if response_format:
    schema = response_format.model_json_schema()
    prompt = f"Please respond using this JSON schema: {schema}\n\n"
    payload["messages"].append({"role": "system", "content": prompt})

try:
    response = requests.post(LOCAL_LLM_ENDPOINT, json=payload)
    response.raise_for_status()
    return response.json()  
except requests.RequestException as e:
    logger.error(f"Local LLM request failed: {e}")
    raise RuntimeError("Failed to communicate with local LLM.")

For mock testing, I'm using a simple Flask API endpoint before eventually testing the use case with the Ollama local model, starting with Mistral

`
from flask import Flask, request, jsonify

app = Flask(name)

@app.route("/v1/chat/completions", methods=["POST"])
def chat_completions():
data = request.json
response = {
"choices": [{
"message": {
"role": "assistant",
"content": "This is a mock response.",
"tool_calls": data.get("tools", []), # Simulate function calling
},
"finish_reason": "stop"
}]
}
return jsonify(response)

if name == "main":
app.run(port=8000)

`

Does this approach align with what you're looking for?

Thanks!

[georg-wolflein]

Hi @pravintargaryen, thank you for your efforts.

However, I think a simpler solution would be to just stick with litellm. I know that for example llama.cpp and ollama are are perfectly compatible with litellm.

For example, check out the ollama/litellm documentation. As far as I can tell, you would only need to modify these lines in toolmaker:

def completion_factory(model: str) -> ChatCompletion:
    if model == "o3-mini":
        # Sometimes o3-mini is flaky, so we retry a few times
        return litellm_completion_retry_on_api_error
    elif "claude" in model:
        return claude_completion
    elif "ollama" in model:
        return functools.partial(litellm.completion, api_base=os.getenv("LOCAL_LLM_ENDPOINT"))
    return litellm.completion

Then, in your environment set

LLM_MODEL=ollama/llama3.3
LLM_MODEL_REASONING=ollama/llama3.3
LOCAL_LLM_ENDPOINT=http://localhost:...

Let me know if this works for you.

[pravintargaryen]

Hi @georg-wolflein,

Thank you for the suggestion! This simpler approach seems more efficient. I'll update completion_factory accordingly and configure the environment variables as you described. I'll test it soon and report back if I encounter any issues.

Thanks

[pravintargaryen]

Hi @georg-wolflein

Submitted PR with the changes you mentioned : #2

However while testing I had the following issues since I'm using Windows

1. os.getuid() & os.getgid() functions replaced with default values of -1 in utils/io.py and toolmaker/cli.py

2. Had some issues with backward "\" slashes as well in toolmaker/runtime/client.py while building docker image (using docker desktop) since I'm on windows so tweaked with lines 163 and 164

3. Though getting this error at the moment

Once the dockerfile path issues are resolved, I will be able to create and run the tool with test cases running on local llm.

[pravintargaryen]

Hi @georg-wolflein,

I thought it would be best if you could take a look at this repo - https://github.com/ComposioHQ/composio