LiteLLM (as a client)

litellm.completion used as a client library, sending to plugboard's Chat Completions.

Connect

Prefix the model with openai/ and set api_base to the /v1 base (LiteLLM adds /chat/completions):

def plugboard(model: str, **kwargs):
    return litellm.completion(model=f"openai/{model}", api_base=os.environ["PLUGBOARD_BASE_URL"], api_key=os.environ["PLUGBOARD_KEY"], **kwargs)

Completion

response = litellm.completion(
    model="openai/fast",
    api_base=os.environ["PLUGBOARD_BASE_URL"],
    api_key=os.environ["PLUGBOARD_KEY"],
    messages=[{"role": "user", "content": "Reply with exactly: pong"}],
)

Streaming

Ask for usage in the last chunk with stream_options:

chunks = list(
    plugboard(model, messages=[{"role": "user", "content": "Reply with exactly: pong"}], stream=True, stream_options={"include_usage": True})
)

Tool calls

LiteLLM hands the call back as tool_calls; your code runs the tool and sends a tool message.

first = plugboard(model, messages=messages, tools=[TOOL])
message = first.choices[0].message
call = message.tool_calls[0]
# Send the assistant turn back as LiteLLM typed it, then the result under the same call id.
messages += [message.model_dump(exclude_none=True), {"role": "tool", "tool_call_id": call.id, "content": TOOL_RESULT}]
second = plugboard(model, messages=messages, tools=[TOOL])

Structured output

response = plugboard(model, messages=[{"role": "user", "content": PROMPT}], response_format=Meal)
meal = Meal.model_validate_json(response.choices[0].message.content or "")

Limits

• LiteLLM fetches its price map from GitHub on import. Set LITELLM_LOCAL_MODEL_COST_MAP=True to skip it.