LangChain4j

LangChain4j's langchain4j-open-ai module.

Connect

Add dev.langchain4j:langchain4j-open-ai to your build (the version is in the example's pom.xml). BASE_URL is http://127.0.0.1:4010/v1 and KEY is your plugboard key.

private static ChatModel chatModel(String model) {
  return OpenAiChatModel.builder().baseUrl(BASE_URL).apiKey(KEY).modelName(model).build();
}

private static StreamingChatModel streamingModel(String model) {
  return OpenAiStreamingChatModel.builder().baseUrl(BASE_URL).apiKey(KEY).modelName(model).build();
}

baseUrl ends in /v1; modelName is an alias such as fast.

Chat

ChatResponse response = chatModel(model).chat(ChatRequest.builder().messages(UserMessage.from(PROMPT)).build());

Tool calls

Plugboard returns the call. Your code runs the tool and sends the result back.

private static final ToolSpecification WEATHER = ToolSpecification.builder()
    .name("get_weather")
    .description("Current weather for a city.")
    .parameters(JsonObjectSchema.builder().addStringProperty("city").required("city").build())
    .build();

ChatResponse first = chat.chat(ChatRequest.builder().messages(messages).toolSpecifications(WEATHER).build());
AiMessage call = first.aiMessage();
messages.add(call);
for (ToolExecutionRequest request : call.toolExecutionRequests()) {
  messages.add(ToolExecutionResultMessage.from(request, "{\"temp_c\": 18, \"sky\": \"cloudy\"}"));
}
ChatResponse last = chat.chat(ChatRequest.builder().messages(messages).toolSpecifications(WEATHER).build());

Streaming

streamingModel(model).chat(request, new StreamingChatResponseHandler() {
  @Override
  public void onPartialResponse(String partial) {
    text.append(partial);
  }

  @Override
  public void onCompleteResponse(ChatResponse response) {
    done.complete(response);
  }

  @Override
  public void onError(Throwable error) {
    done.completeExceptionally(error);
  }
});

The runnable project is in examples/langchain4j/.

What works

Chat, streaming, tool calls (@Tool with AI Services, or ToolExecutionResultMessage by hand), streamed tool calls and structured output.