Frameworks and SDKs

Every major AI framework already knows how to talk to this server, because each one ships an OpenAI-compatible provider and this server serves that dialect behaviorally (API Compatibility). This guide is the recipe book: for each dominant stack, the exact provider configuration, then chat, streaming, embeddings, and tool calls, and where a framework feature meets a documented limit.

1What every recipe shares#

Three values configure every framework below, and they never change per stack:

• Base URL: http://your-server:PORT/v1 for the OpenAI dialect. Paths without the /v1 prefix are aliased to it, so a client that constructs /chat/completions directly still lands.
• API key: minted in the admin console, sent as the Bearer header. On a loopback-only install with anonymous access, a placeholder string satisfies SDKs that require one.
• Model id: a catalog id from GET /v1/models (browsable in Models). Frameworks generally require the field; where one lets you omit it, the server's default for the capability answers (Choosing and Operating Models).

Prefer the OpenAI dialect for frameworks: it is the shape they test against, and it carries the key natively. The Ollama dialect exists for clients that already speak it (Coming from Ollama); a framework wrapping the Anthropic SDK inherits the auth-token nuance from Cloud-to-Local Migration. Editor assistants (Continue, Cline, Zed) have their own guide: Coding Assistants.

2The official OpenAI SDKs#

The reference clients. Everything the dialect serves (streaming, response_format schemas, tools, embeddings) is reachable through them unchanged. The SDKs' other sub-clients (audio, images, moderations, batches, fine-tuning, assistants) have no counterpart here and answer honestly that the endpoint does not exist, through every framework below; what is served is the table in API Compatibility.

Python:

from openai import OpenAI
client = OpenAI(base_url="http://your-server:PORT/v1", api_key="lmk_...")

reply = client.chat.completions.create(model="<model-id>",
    messages=[{"role": "user", "content": "Say hello."}])

Pass stream=True for token deltas, tools=[...] for function calling, response_format={"type": "json_schema", ...} for enforced structured outputs, and use client.embeddings.create(...) against the same base URL for vectors. Vision-capable models take images as image_url content parts on the same chat call, from any framework that passes them through (Vision).

JavaScript / TypeScript:

import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://your-server:PORT/v1", apiKey: "lmk_..." });

const stream = await client.chat.completions.create({
  model: "<model-id>", messages, stream: true });
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");

.NET (the official OpenAI package):

using OpenAI;
using System.ClientModel;

var client = new OpenAIClient(new ApiKeyCredential("lmk_..."),
    new OpenAIClientOptions { Endpoint = new Uri("http://your-server:PORT/v1") });
var chat = client.GetChatClient("<model-id>");
var completion = chat.CompleteChat("Say hello.");

The SDKs' Responses-API clients also work: POST /v1/responses is served, with stateful chaining and file_search over local vector stores (Responses and Vector Stores).

3LangChain#

Use the OpenAI provider classes with a base URL; the rest of LangChain composes on top.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings

llm = ChatOpenAI(base_url="http://your-server:PORT/v1",
                 api_key="lmk_...", model="<model-id>")
emb = OpenAIEmbeddings(base_url="http://your-server:PORT/v1",
                       api_key="lmk_...", model="<embedding-model-id>",
                       check_embedding_ctx_length=False)

• Chat and streaming: llm.invoke(...) and llm.stream(...) work as documented; the stream is ordinary SSE deltas underneath.
• Tools: llm.bind_tools([...]) rides the OpenAI tool contract. Arguments arrive whole and schema-constrained (Function Calling), so LangChain's argument parsing does not hit half-JSON.
• Structured output: llm.with_structured_output(Schema, method="json_schema") maps to the enforced response_format, which here is a guarantee rather than a request (Structured Outputs).
• Embeddings: set check_embedding_ctx_length=False so LangChain sends your text as strings instead of pre-tokenizing it with OpenAI's tokenizer, which local models do not share. Embed corpora and queries with the mode asymmetry in mind (Embeddings and Reranking).

Two retrieval stages sit outside the OpenAI dialect, here and in every framework below. Reranking has no OpenAI-shaped equivalent: call the native /lmkit/v1/rerank/score or /lmkit/v1/rerank/rerank endpoints from a custom retriever or plain HTTP. And rather than operating a separate vector database, the server's Search engine and OpenAI-compatible vector stores are endpoints a retriever can call.

4LlamaIndex#

LlamaIndex separates "the OpenAI service" from "something OpenAI-shaped"; use the OpenAILike classes so it does not second-guess unknown model names.

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(api_base="http://your-server:PORT/v1", api_key="lmk_...",
                 model="<model-id>", is_chat_model=True,
                 is_function_calling_model=True)

The two flags matter: they tell LlamaIndex to use the chat endpoint and to route its agent and tool abstractions through native tool calls; stream_chat streams as everywhere else (section 8). For embeddings, the companion OpenAILikeEmbedding class (package llama-index-embeddings-openai-like) takes the same api_base, api_key, and model_name. Retrievers, query engines, and agents built on these run unchanged; the server-side retrieval and reranking endpoints from section 3 apply here too.

5The Vercel AI SDK#

Use the @ai-sdk/openai-compatible provider; it exists precisely for servers like this one.

import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { streamText } from "ai";

const lmkit = createOpenAICompatible({ name: "lmkit",
  baseURL: "http://your-server:PORT/v1", apiKey: "lmk_...",
  supportsStructuredOutputs: true });

const result = streamText({ model: lmkit("<model-id>"), prompt: "Say hello." });

generateText with tools runs the client-side tool loop. generateObject reaches the enforced json_schema channel only with supportsStructuredOutputs set as above; without it the provider sends plain JSON mode, validated client-side. embed / embedMany with lmkit.textEmbeddingModel("<id>") hit /v1/embeddings. One deployment reality: the AI SDK runs in your app's backend, so the server must be reachable from where that backend runs. Local development hits localhost; a deployed app pairs with this server on the same host or private network, behind TLS and real keys (Going Live).

6Semantic Kernel and Microsoft.Extensions.AI#

.NET teams have two first-party paths, and the choice is the interesting part.

Over HTTP, against this server. Semantic Kernel's OpenAI connector accepts a custom endpoint:

var kernel = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion(modelId: "<model-id>",
        endpoint: new Uri("http://your-server:PORT/v1"), apiKey: "lmk_...")
    .Build();

For Microsoft.Extensions.AI, wrap the official OpenAI client:

using Microsoft.Extensions.AI;

IChatClient chat = new OpenAIClient(new ApiKeyCredential("lmk_..."),
        new OpenAIClientOptions { Endpoint = new Uri("http://your-server:PORT/v1") })
    .GetChatClient("<model-id>").AsIChatClient();

Both abstractions' function invocation and structured-output features ride the same OpenAI contract as everything above. For vectors, the same OpenAIClient serves GetEmbeddingClient("<embedding-model-id>") against /v1/embeddings, and Extensions.AI wraps it the way it wraps chat (AsIEmbeddingGenerator()).

In-process, no server. Because this product is built on the LM-Kit.NET SDK, a .NET application can host the engine directly: the LM-Kit.NET.SemanticKernel package registers AddLMKitChatCompletion and AddLMKitTextEmbeddingGeneration on the kernel builder, and LM-Kit.NET.ExtensionsAI provides AddLMKitChatClient and AddLMKitEmbeddingGenerator for IChatClient / embedding-generator consumers. The model loads inside your process; no HTTP, no key, no second deployment.

The decision, plainly:

The two are not exclusive: prototyping against the server and embedding the SDK in the shipped product (or the reverse) is a supported trajectory, because both speak the same model catalog and engine.

7LiteLLM as a router#

Where LiteLLM fronts your model traffic, this server registers as an ordinary OpenAI-compatible deployment:

model_list:
  - model_name: local-chat
    litellm_params:
      model: openai/<model-id>
      api_base: http://your-server:PORT/v1
      api_key: lmk_...

The openai/ prefix tells LiteLLM to speak the OpenAI dialect at your api_base. Chat, streaming, embeddings, and tool calls route through unchanged, and LiteLLM's fallback and routing policies can mix this deployment with hosted ones during a migration (Cloud-to-Local Migration). Keys still matter: give LiteLLM its own key so its traffic is attributable in the request trail.

8Streaming, across all of them#

Every recipe above streams the same way underneath: stream: true on /v1/chat/completions produces SSE deltas that start when the model starts producing, the Anthropic dialect streams its documented SSE events, and the Ollama dialect streams NDJSON. Four facts save debugging time:

• Tool-call arguments arrive whole, streaming included. A framework that accumulates argument fragments still works, because a complete argument payload is a valid single fragment (Function Calling).
• Token usage arrives in the last chunk, on request. Send stream_options: {"include_usage": true} and the final pre-[DONE] chunk carries the usage block; frameworks that report usage on streams set it.
• Dropping the connection cancels the generation. A client that aborts mid-stream stops the model server-side; the request records as 499 in the request trail: recorded, never an error (Errors, Retries, and Jobs).
• A stream that arrives in one burst is a proxy, not the server. Reverse proxies buffer SSE by default; the fix is a proxy setting, covered in Behind a Reverse Proxy.

9Stated plainly#

• One base URL, one key, one model id: every framework recipe is those three values in that framework's provider settings.
• Prefer each framework's OpenAI-compatible provider; it is the tested shape and carries the key natively.
• Structured output and tool calling through framework abstractions land on enforced server-side contracts, not best-effort parsing; where a provider gates that behind a flag (the Vercel AI SDK's supportsStructuredOutputs), set it.
• .NET teams choose per application between this server over HTTP and the LM-Kit.NET SDK in-process; both are first-party.