AnythingLLM

A self-hosted assistant with RAG and agents. Use its Generic OpenAI LLM provider.

Connect

In Settings → AI Providers → LLM → Generic OpenAI: Base URL http://127.0.0.1:4010/v1, your plugboard key, the model name typed by hand (such as fast), and the model's real context window and max tokens. Or set the same values as environment variables. In Docker (plugboard must listen where the container can reach it, not only on 127.0.0.1):

services:
  anythingllm:
    image: mintplexlabs/anythingllm:latest
    ports:
      - "3001:3001"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - STORAGE_DIR=/app/server/storage
      - LLM_PROVIDER=generic-openai
      - GENERIC_OPEN_AI_BASE_PATH=http://host.docker.internal:4010/v1
      - GENERIC_OPEN_AI_API_KEY=${PLUGBOARD_KEY_ANYTHINGLLM}
      - GENERIC_OPEN_AI_MODEL_PREF=fast
      - GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=200000
    volumes:
      - ./storage:/app/server/storage

The file is in examples/anythingllm/.

What works

Chat, streaming, and agent tool calls.

Limits

• There is no model discovery; a wrong name fails on the first chat.
• Set the context window and max tokens to the model's real values. Values that are too low truncate the context without an error.
• If agent tool turns misbehave, try PROVIDER_DISABLE_NATIVE_TOOL_CALLING to fall back to prompt-based tools.
• Embeddings are not served by plugboard; use another embedding provider for documents.