Set up the gateway and the packages
Install FreeLLMAPI and add free keys from two or more providers on the dashboard Keys page. Copy the unified key. Your code uses only this key. The provider keys stay in FreeLLMAPI.
Install FreeLLMAPI and the LangChain packages
curl -fsSL https://freellmapi.co/install.sh | bash
# Python
pip install -U langchain langchain-openai
# JavaScript
npm install @langchain/openai @langchain/core zodEnvironment variables
export FREELLMAPI_BASE_URL=http://localhost:3001/v1
export FREELLMAPI_API_KEY=<unified-key>Use model="auto" to let the router select a model. You can also use a routing strategy, for example auto:fast or auto:smart. A named profile such as auto:coding and a fixed model ID from the catalog also work.
To see which models your keys can call now, request GET /v1/models?execution_status=ready. The full list also contains models without a key. A model with the status exhausted has keys, but all of them are at a limit at this moment. That status clears when the cooldowns end.
List the models that can answer now
curl "http://localhost:3001/v1/models?execution_status=ready" \
-H "Authorization: Bearer <unified-key>"Chat with ChatOpenAI
In Python, set base_url and api_key. The max_retries value goes to the OpenAI client, which retries 429 and 5xx errors with exponential backoff.
Python: chat and streaming
import os
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="auto",
base_url=os.environ["FREELLMAPI_BASE_URL"],
api_key=os.environ["FREELLMAPI_API_KEY"],
temperature=0.2,
max_retries=3,
timeout=60,
include_response_headers=True,
)
msg = llm.invoke("Explain retrieval-augmented generation in two sentences.")
print(msg.content)
print("Served by:", msg.response_metadata["headers"].get("x-routed-via"))
for chunk in llm.stream("Write a haiku about SQLite."):
print(chunk.content, end="", flush=True)In JavaScript, put the base URL in configuration.baseURL. The configuration object goes to the official OpenAI SDK client.
JavaScript: chat and streaming
import { ChatOpenAI } from "@langchain/openai";
const llm = new ChatOpenAI({
model: "auto",
apiKey: process.env.FREELLMAPI_API_KEY,
temperature: 0.2,
maxRetries: 3,
configuration: { baseURL: process.env.FREELLMAPI_BASE_URL },
});
const msg = await llm.invoke("Explain retrieval-augmented generation in two sentences.");
console.log(msg.content);
for await (const chunk of await llm.stream("Write a haiku about SQLite.")) {
process.stdout.write(String(chunk.content));
}The x-routed-via header shows the provider and model that answered. If the router changed provider, x-fallback-trail shows each failed attempt and the reason.
Tools and agents
FreeLLMAPI passes OpenAI-style tools to each provider and returns tool_calls in OpenAI format. For Gemini, it translates the request into Google's format and back. Use bind_tools for one call, or create_agent for a loop that runs the tools.
Python: bind_tools and create_agent
from langchain.agents import create_agent
from langchain_core.tools import tool
@tool
def get_weather(city: str) -> str:
"""Return the current weather for a city."""
return f"It is 18 C and cloudy in {city}."
# One call: the model returns the tool call, your code runs it
msg = llm.bind_tools([get_weather]).invoke("What is the weather in Dublin?")
print(msg.tool_calls)
# Agent loop: LangChain runs the tool and sends the result back
agent = create_agent(
model=llm,
tools=[get_weather],
system_prompt="You answer weather questions. Use the tool.",
)
result = agent.invoke({"messages": [{"role": "user", "content": "Weather in Dublin?"}]})
print(result["messages"][-1].content)JavaScript: bindTools
import { tool } from "@langchain/core/tools";
import { z } from "zod";
const getWeather = tool(
async ({ city }) => `It is 18 C and cloudy in ${city}.`,
{
name: "get_weather",
description: "Return the current weather for a city.",
schema: z.object({ city: z.string() }),
},
);
const reply = await llm.bindTools([getWeather]).invoke("What is the weather in Dublin?");
console.log(reply.tool_calls);Only some free models support tool calling. With auto, keep tool-calling models in your fallback chain, or make a profile for agents. These free models support tools today:
| Model | Free on | Context | Free limits (best offer) | Tools | Vision |
|---|---|---|---|---|---|
| GLM 5.3 | UnoRouter | 1M | 1 req/min 1 req/min per model | Yes | No |
| GLM 5.3 Flash | NVIDIA NIM, UnoRouter | 1M | 40 req/min 40 RPM | Yes | Yes |
| Gemini 3.8 Flash | Google AI Studio | 1M | Recurring provider quota | Yes | Yes |
| Qwen3.8 Flash Next | AMD Radeon Cloud | 262K | Recurring daily account allowance | Yes | No |
| Qwen3.8 Flash | ElectronHub | 1M | $0.25/week shared recurring credit | Yes | Yes |
| Gemini 3.7 Flash | Google AI Studio | 1M | 10 req/min 20 req/day 250,000 tokens/min ~3M | Yes | Yes |
| GPT-5.6 Luna | ElectronHub | 1M | $0.25/week shared recurring credit | Yes | Yes |
| Dots3 Note Preview | Kilo Gateway, OpenRouter | 512K | 20 req/min 50 req/day 50 req/day account-wide (1,000 after a $10 purchase) | Yes | Yes |
| DeepSeek V4 Flash | AMD Radeon Cloud, BazaarLink, ElectronHub | 1M | 10 req/min 50 req/day Shared rate limit | Yes | No |
| Qwen3.8 27B | ElectronHub, Groq, OVH AI Endpoints | 262K | 30 req/min 250 req/day 70,000 tokens/min ~6M | Yes | Yes |
Showing 10 of 109 matching free models. See all models.
Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.
Embeddings with OpenAIEmbeddings
By default, OpenAIEmbeddings in Python splits text with an OpenAI tokenizer and sends tokens. Most other providers need raw text. Set check_embedding_ctx_length=False to send text.
Python: embeddings
from langchain_openai import OpenAIEmbeddings
emb = OpenAIEmbeddings(
model="bge-m3", # one embedding family, 1024 dimensions
base_url=os.environ["FREELLMAPI_BASE_URL"],
api_key=os.environ["FREELLMAPI_API_KEY"],
check_embedding_ctx_length=False,
)
vectors = emb.embed_documents(["FreeLLMAPI pools free tiers.", "LangChain builds LLM apps."])
print(len(vectors), len(vectors[0]))JavaScript: embeddings
import { OpenAIEmbeddings } from "@langchain/openai";
const emb = new OpenAIEmbeddings({
model: "bge-m3",
apiKey: process.env.FREELLMAPI_API_KEY,
configuration: { baseURL: process.env.FREELLMAPI_BASE_URL },
});
const vector = await emb.embedQuery("What does FreeLLMAPI do?");
console.log(vector.length);Embedding failover in FreeLLMAPI never changes the model. It only moves between providers that serve the same family. This keeps all vectors in one vector store compatible. Select a family and keep it. The free embedding models today are:
| Model | Provider | Dimensions | Max input | Free limit |
|---|---|---|---|---|
Gemini Embeddinggemini-embedding-001 | Google AI Studio | 3072 | 2048 | 100 rpm · 1K req/day |
Nemotron Embed VL 1Bnvidia/llama-nemotron-embed-vl-1b-v2 | NVIDIA NIM | 2048 | 8192 | ~40 rpm |
Nemotron 3 Embed 1B (OR free)nvidia/nemotron-3-embed-1b:free | OpenRouter | 2048 | 32768 | 20 rpm · shared free daily cap |
SEA-LION E5 Embedding 600Maisingapore/SEA-LION-E5-Embedding-600M | SEA-LION | 1024 | 8192 | free · 10 rpm |
SEA-LION ModernBERT Embedding 300Maisingapore/SEA-LION-ModernBERT-Embedding-300M | SEA-LION | 1024 | 8192 | free · 10 rpm |
SEA-LION ModernBERT Embedding 600Maisingapore/SEA-LION-ModernBERT-Embedding-600M | SEA-LION | 1024 | 8192 | free · 10 rpm |
BGE-M3@cf/baai/bge-m3 | Cloudflare Workers AI | 1024 | 8192 | 10K neurons/day (shared) |
EmbeddingGemma 300M@cf/google/embeddinggemma-300m | Cloudflare Workers AI | 768 | 2048 | 10K neurons/day (shared) |
Qwen3 Embedding 0.6B@cf/qwen/qwen3-embedding-0.6b | Cloudflare Workers AI | 1024 | 8192 | 10K neurons/day (shared) |
Cohere Embed v4embed-v4.0 | Cohere | 1536 | 128000 | 1K calls/mo (shared w/ chat) |
Gemini Embedding 2gemini-embedding-2 | Google AI Studio | 3072 | 8192 | free-tier rate limited |
Nemotron 3 Embed 1Bnvidia/nemotron-3-embed-1b | NVIDIA NIM | 2048 | 4096 | ~40 rpm |
PLaMo Embedding 1B (CF)@cf/pfnet/plamo-embedding-1b | Cloudflare Workers AI | 2048 | 4096 | 10K neurons/day (shared) |
BGE Small EN v1.5 (CF)@cf/baai/bge-small-en-v1.5 | Cloudflare Workers AI | 384 | 512 | 10K neurons/day (shared) |
BGE Base EN v1.5 (CF)@cf/baai/bge-base-en-v1.5 | Cloudflare Workers AI | 768 | 512 | 10K neurons/day (shared) |
BGE Large EN v1.5 (CF)@cf/baai/bge-large-en-v1.5 | Cloudflare Workers AI | 1024 | 512 | 10K neurons/day (shared) |
Mistral Embedmistral-embed | Mistral | 1024 | 8192 | free mode · monthly cap (shared) |
Codestral Embedcodestral-embed | Mistral | 1536 | 8192 | free mode · monthly cap (shared) |
Llama Text Embed v2 (Pinecone)llama-text-embed-v2 | Pinecone | 1024 | 2048 | 5M tokens/mo per model |
Multilingual E5 Large (Pinecone)multilingual-e5-large | Pinecone | 1024 | 507 | 5M tokens/mo per model |
Nemotron 3 Embed 1B (Speka)nvidia/nemotron-3-embed-1b | Speka | 2048 | 8192 | $1/month shared across Speka models |
OpenAI: Text Embedding 3 Small (ElectronHub)text-embedding-3-small | ElectronHub | 1536 | 8192 | $0.25/week shared free credits |
OpenAI: Text Embedding 3 Large (ElectronHub)text-embedding-3-large | ElectronHub | 3072 | 8192 | $0.25/week shared free credits |
Nemotron 3 Embed 8B (Inferbase)nvidia-nemotron-3-embed-8b | Inferbase | 4096 | 8192 | 5,000 requests/month; 100 requests and $0.50 usage/day shared |
Qwen 3 Embedding 0.6B (Inferbase)qwen3-embedding-0-6b | Inferbase | 1024 | 8192 | 5,000 requests/month; 100 requests and $0.50 usage/day shared |
Qwen 3 Embedding 4B (Inferbase)qwen3-embedding-4b | Inferbase | 2560 | 8192 | 5,000 requests/month; 100 requests and $0.50 usage/day shared |
Qwen 3 Embedding 8B (Inferbase)qwen3-embedding-8b | Inferbase | 4096 | 8192 | 5,000 requests/month; 100 requests and $0.50 usage/day shared |
LFM2.5 Embedding 350M (LLMTR)liquid/lfm-2.5-embedding-350m-free | LLMTR | 1024 | 512 | Ongoing zero-price route; provider terms apply |
BGE-M3 (SEA-LION)BAAI/bge-m3 | SEA-LION | 1024 | 8192 | free · 10 rpm |
Cohere Embed English v3embed-english-v3.0 | Cohere | 1024 | 512 | 1K calls/mo (shared w/ chat) |
Qwen3 Embedding 8B (OVH)Qwen3-Embedding-8B | OVH AI Endpoints | 4096 | 32768 | free · anonymous 2 req/min per IP |
BGE-M3 (OVH)bge-m3 | OVH AI Endpoints | 1024 | 8192 | free · anonymous 2 req/min per IP |
BGE Multilingual Gemma2 (OVH)bge-multilingual-gemma2 | OVH AI Endpoints | 3584 | 8192 | free · anonymous 2 req/min per IP |
Mistral: Codestral Embed (ElectronHub)codestral-embed | ElectronHub | 1536 | 8192 | $0.25/week shared free credits |
Mistral: Mistral Embed (ElectronHub)mistral-embed | ElectronHub | 1024 | 8192 | $0.25/week shared free credits |
Google: Gemini Embedding 2 (ElectronHub)gemini-embedding-2 | ElectronHub | 3072 | 8192 | $0.25/week shared free credits |
Google: Gemini Embedding 001 (ElectronHub)gemini-embedding-001 | ElectronHub | 3072 | 2048 | $0.25/week shared free credits |
OpenAI: Text Embedding Ada 002 (ElectronHub)text-embedding-ada-002 | ElectronHub | 1536 | 8192 | $0.25/week shared free credits |
Cohere Embed English Light v3embed-english-light-v3.0 | Cohere | 384 | 512 | 1K calls/mo (shared w/ chat) |
Cohere Embed Multilingual v3embed-multilingual-v3.0 | Cohere | 1024 | 512 | 1K calls/mo (shared w/ chat) |
Cohere Embed Multilingual Light v3embed-multilingual-light-v3.0 | Cohere | 384 | 512 | 1K calls/mo (shared w/ chat) |
Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.
Structured output, sessions and model choice
with_structured_output returns a Pydantic object in place of text. By default, LangChain uses function calling for this. That method works on every free model that supports tools.
Python: structured output
from pydantic import BaseModel, Field
class Ticket(BaseModel):
category: str = Field(description="billing, bug or other")
urgent: bool
classifier = llm.with_structured_output(Ticket)
ticket = classifier.invoke("The app crashes when I pay. Please fix it today.")
print(ticket.category, ticket.urgent)Do not use method="json_schema" with auto. Not every free provider supports strict JSON schema output, and the router can select a different provider for each request.
FreeLLMAPI keeps a conversation on one model for 30 minutes. It finds the conversation from the X-Session-Id header. If there is no header, it uses a hash of the first user message. For a chat app with many users, send one session ID per user:
Python: one session per user
def llm_for(user_id: str) -> ChatOpenAI:
return ChatOpenAI(
model="auto",
base_url=os.environ["FREELLMAPI_BASE_URL"],
api_key=os.environ["FREELLMAPI_API_KEY"],
default_headers={"X-Session-Id": f"user-{user_id}"},
max_retries=3,
)Select the model string for each part of your app:
| Task in your app | Model string | Reason |
|---|---|---|
| Chat answers for users | auto | Follows the chain that you set on the dashboard |
| Agent with tools | auto:<profile> | A profile that has only tool-calling models |
| Classification, routing, short summaries | auto:fast | Ranks models by measured speed |
| Hard reasoning steps | auto:smart | Ranks models by intelligence |
| Tests that must be repeatable | A fixed model ID | Same model each time, but it stops at that model's limit |
Free tiers have no SLA. Latency changes during the day, and the strongest models reach their daily limits first. Test your chains with the models that your chain will actually use.
Rate limits and retries
FreeLLMAPI tries the next provider when one returns 429. Your code sees a 429 only when every key in the chain is at its limit. Then the response has a Retry-After header. Many daily limits reset at midnight UTC.
For batch jobs, add a second retry layer with with_retry. It retries the whole runnable with exponential backoff and jitter.
Python: retry a chain on rate limits
import openai
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_messages([
("system", "Summarize the text in one sentence."),
("user", "{text}"),
])
chain = (prompt | llm).with_retry(
retry_if_exception_type=(openai.RateLimitError, openai.APITimeoutError),
wait_exponential_jitter=True,
stop_after_attempt=5,
)
# batch() runs requests in parallel; max_concurrency keeps the rate low
results = chain.batch(
[{"text": t} for t in ["First document.", "Second document."]],
config={"max_concurrency": 4},
)| Problem | Cause | Fix |
|---|---|---|
openai.AuthenticationError | Wrong unified key | Copy the key again from the Keys page |
openai.NotFoundError on every call | The base URL has no /v1 | Use http://localhost:3001/v1 |
openai.RateLimitError | All keys in the chain are at their limit | Add providers, reduce max_concurrency or wait for Retry-After |
| Embedding error about tokens or input type | LangChain sends tokens, not text | Set check_embedding_ctx_length=False |
tool_calls is empty | The model does not support tools | Use a model from the tools table or a tools profile |
| Reasoning text is missing | ChatOpenAI keeps only standard OpenAI fields | Read the answer in content; extra provider fields are not kept |
Frequently asked questions
Can I use a free LLM API with LangChain?
Yes. Any OpenAI-compatible API works with ChatOpenAI. Set base_url to http://localhost:3001/v1 and api_key to your FreeLLMAPI unified key. The router uses free provider tiers.
How do I set a custom base URL in LangChain ChatOpenAI?
In Python, pass base_url to ChatOpenAI. In JavaScript, pass configuration: { baseURL } to the constructor. The OPENAI_BASE_URL environment variable also works in Python.
Do LangChain agents work with free models?
Yes, if the model supports tool calling. Use create_agent with a ChatOpenAI model and keep tool-calling models in your FreeLLMAPI chain.
Why does OpenAIEmbeddings fail with a non-OpenAI provider?
By default it sends token IDs. Most other providers need raw text. Set check_embedding_ctx_length=False in Python.
How do I handle 429 errors in LangChain?
Set max_retries on ChatOpenAI and add with_retry to the chain for batch jobs. FreeLLMAPI already moves to the next provider before your code sees a 429.
Related guides
- Free LLM for LlamaIndex
- Free LLM API in Python
- Free LLM API for RAG
- Free embeddings API
- Free LLM API with function calling
- Free LLM API for AI agents
Sources
- LangChain docs: ChatOpenAI integration (Python) (checked 20 November 2026)
- LangChain docs: ChatOpenAI integration (JavaScript) (checked 20 November 2026)
- LangChain docs: agents and create_agent (checked 20 November 2026)
- LangChain source: OpenAIEmbeddings note on OpenAI-compatible APIs (checked 20 November 2026)
- OpenAI Python SDK: retries and errors (checked 20 November 2026)
- FreeLLMAPI docs: REST API (tool calling, embeddings, response headers) (checked 20 November 2026)