Setup
Install FreeLLMAPI. It runs in Docker. Open the dashboard at http://localhost:3001, add free keys from two or more providers on the Keys page and copy the unified key. Then install the Python package.
Your Python code uses only the unified key. The provider keys stay in FreeLLMAPI, encrypted in its database. If you change or add a provider key, you do not change your code.
Install
curl -fsSL https://freellmapi.co/install.sh | bash
python3 -m pip install -U openai
export FREELLMAPI_API_KEY=<unified-key>client.py
import os
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:3001/v1",
api_key=os.environ["FREELLMAPI_API_KEY"],
max_retries=3, # the SDK retries 429, 408, 409 and 5xx errors
timeout=60,
)
resp = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Name three uses of Python in one line."}],
)
print(resp.choices[0].message.content)The model auto follows your fallback chain on the dashboard. You can also use auto:fast, auto:smart or auto:reliable, a profile such as auto:coding, or a fixed model ID.
Streaming and response headers
Stream the answer
from client import client
stream = client.chat.completions.create(
model="auto:fast",
messages=[{"role": "user", "content": "Write a haiku about Python."}],
stream=True,
)
for chunk in stream:
if chunk.choices:
print(chunk.choices[0].delta.content or "", end="", flush=True)
print()Each response has an X-Routed-Via header with the provider and model that answered. To read headers with the SDK, use with_raw_response and call parse() to get the normal object.
Read the routing headers
from client import client
raw = client.chat.completions.with_raw_response.create(
model="auto",
messages=[{"role": "user", "content": "hi"}],
)
print("Served by:", raw.headers.get("x-routed-via"))
print("Failed attempts:", raw.headers.get("x-fallback-attempts", "0"))
print(raw.parse().choices[0].message.content)Tool calling
Send OpenAI-style tools. The router passes them to OpenAI-compatible providers and translates them for Gemini. The answer comes back in the same format for every provider.
One tool call and the final answer
import json
from client import client
tools = [{
"type": "function",
"function": {
"name": "get_exchange_rate",
"description": "Get the exchange rate between two currencies.",
"parameters": {
"type": "object",
"properties": {
"base": {"type": "string"},
"quote": {"type": "string"},
},
"required": ["base", "quote"],
},
},
}]
messages = [{"role": "user", "content": "How many euros is 250 US dollars?"}]
first = client.chat.completions.create(model="auto", messages=messages, tools=tools)
msg = first.choices[0].message
if msg.tool_calls:
messages.append(msg)
for call in msg.tool_calls:
args = json.loads(call.function.arguments)
result = {"base": args["base"], "quote": args["quote"], "rate": 0.92} # your real lookup
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
final = client.chat.completions.create(model="auto", messages=messages, tools=tools)
print(final.choices[0].message.content)
else:
print(msg.content)Some providers return a tool call as plain text. FreeLLMAPI converts these answers into real tool_calls, so your code reads one format. Always check msg.tool_calls before you read msg.content, because a model can also answer without a tool.
Use a model with tool calling. These free models support tools and have the fastest measured speed today:
| Model | Free on | Context | Free limits (best offer) | Tools | Vision |
|---|---|---|---|---|---|
| Qwen3.8 27B | ElectronHub, Groq, OVH AI Endpoints | 262K | 30 req/min 250 req/day 70,000 tokens/min ~6M | Yes | Yes |
| Nemotron 3.5 Lightning | ElectronHub, Kilo Gateway, OpenRouter | 1M | 20 req/min 50 req/day 50 req/day account-wide (1,000 after a $10 purchase) | Yes | No |
| GPT-OSS 120B | Cloudflare Workers AI, ElectronHub, Groq +1 more | 131K | 30 req/min 1,000 req/day 8,000 tokens/min 200,000 tokens/day ~6M | Yes | No |
| GPT-OSS 20B | Cloudflare Workers AI, Groq, NVIDIA NIM +1 more | 131K | 30 req/min 1,000 req/day 8,000 tokens/min 200,000 tokens/day ~6M | Yes | No |
| GPT-OSS Safeguard 20B | Groq | 131K | 30 req/min 1,000 req/day 8,000 tokens/min 200,000 tokens/day ~6M | Yes | No |
| Gemini 3.5 Flash Lite | ElectronHub, Google AI Studio | 1M | 15 req/min 20 req/day 250,000 tokens/min ~3M | Yes | Yes |
| Step 3.7 Flash | ElectronHub | 256K | $0.25/week shared recurring credit | Yes | Yes |
| Gemini 3.1 Flash-Lite | DreamPrompting, Google AI Studio | 1M | 100 req/min 5,000 req/day 500,000 tokens/day 500K tokens/day · 5,000 req/day | Yes | Yes |
| Nemotron 3.5 Lightning 30B | Requesty | 1M | 200 req/day 200 rpd | Yes | No |
| Gemini 2.5 Flash-Lite | ElectronHub, Google AI Studio | 1M | 15 req/min 20 req/day 250,000 tokens/min ~3M | Yes | Yes |
Showing 10 of 146 matching free models. See all models.
Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.
Async batches inside free limits
AsyncOpenAI sends many requests at the same time. Free tiers limit requests per minute, so limit the concurrency with a semaphore. The example also retries when every provider is at its limit.
batch.py
import asyncio
import os
import openai
from openai import AsyncOpenAI
client = AsyncOpenAI(
base_url="http://localhost:3001/v1",
api_key=os.environ["FREELLMAPI_API_KEY"],
max_retries=2,
)
limit = asyncio.Semaphore(4) # at most 4 requests at a time
async def summarize(text: str) -> str:
async with limit:
for attempt in range(5):
try:
resp = await client.chat.completions.create(
model="auto:fast",
messages=[
{"role": "system", "content": "Summarize in one sentence."},
{"role": "user", "content": text},
],
)
return resp.choices[0].message.content
except openai.RateLimitError as err:
header = err.response.headers.get("retry-after", "")
wait = int(header) if header.isdigit() else 2 ** attempt
await asyncio.sleep(min(wait, 60))
return "ERROR: rate limited"
async def main():
docs = [f"Document {i}: Python is a programming language." for i in range(20)]
results = await asyncio.gather(*(summarize(d) for d in docs))
for r in results:
print(r)
await client.close()
asyncio.run(main())The router returns 429 only when every key in the chain is at its limit. The response then has a Retry-After header. A small semaphore value and more provider keys give the best results for large batches.
Choose a model string for each task
The model string tells the router how to select a model. You can use a different string in each call. You do not need to change the dashboard.
| Task | Model string | Reason |
|---|---|---|
| Chat answers for users | auto | Follows the fallback chain that you set on the dashboard |
| Classification, extraction, short summaries | auto:fast | Ranks enabled models by measured speed |
| Hard questions, code review, planning | auto:smart | Ranks enabled models by intelligence |
| Scheduled jobs that must finish | auto:reliable | Ranks enabled models by recent success rate |
| A separate chain for one app | auto:<profile> | Uses a named fallback chain |
| Tests that must be repeatable | A fixed model ID | Same model each time, but it stops at that model's limit |
To see which models your keys can call now, request the model list with a filter. The full list also contains models that have no key yet.
List the models that can answer now
import os
import httpx
r = httpx.get(
"http://localhost:3001/v1/models",
params={"execution_status": "ready"},
headers={"Authorization": f"Bearer {os.environ['FREELLMAPI_API_KEY']}"},
timeout=30,
)
r.raise_for_status()
for m in r.json()["data"]:
print(m["id"])A model with the status exhausted has keys, but all of them are at a limit now. The status clears when the cooldowns end. A model with the status needsKey has no enabled key.
For a conversation, send an X-Session-Id header with extra_headers. FreeLLMAPI then keeps that conversation on one model for 30 minutes. Without the header, it uses a hash of the first user message as the session key.
Embeddings
Embed text
from client import client
resp = client.embeddings.create(
model="bge-m3", # one embedding family
input=["Python is a programming language.", "A snake is a reptile."],
)
print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dimensions")Embedding failover never changes the model. It only moves between providers that serve the same family, so your vectors stay compatible. The value auto uses the default family that you set on the dashboard.
Send many texts in one input list. One request with 32 texts uses one request from the free limit, not 32. Keep each text under the maximum input of the family. Store the family name with your vectors, so that you embed queries with the same family later.
Errors and fixes
| Exception | Cause | Fix |
|---|---|---|
openai.AuthenticationError | Wrong unified key | Copy the key again from the Keys page |
openai.APIConnectionError | FreeLLMAPI is not running, or wrong port | Open http://localhost:3001 in the browser |
openai.NotFoundError | The base URL has no /v1 | Use http://localhost:3001/v1 |
openai.RateLimitError | All keys in the chain are at their limit | Add providers, lower the concurrency, wait for Retry-After |
openai.BadRequestError with auto:name | The profile name does not exist | Check the profile name on the dashboard |
openai.UnprocessableEntityError (422) with images | No vision model is in your chain | Enable a vision model in the fallback chain |
Free tiers have no SLA. Latency changes during the day. The strongest models reach their daily limits first, and many limits reset at midnight UTC. Do not use a free gateway for a service that must answer within a fixed time.
Frequently asked questions
Is there a free LLM API for Python?
Yes. Many providers have free tiers that work with the openai Python package. FreeLLMAPI combines them behind one base URL and one key, with failover between providers.
How do I use the openai Python library with a different base URL?
Pass base_url to OpenAI() or AsyncOpenAI(), for example base_url="http://localhost:3001/v1". Pass your gateway key as api_key.
How do I handle rate limit errors in Python?
Set max_retries on the client. Catch openai.RateLimitError, read the Retry-After header and wait. Limit concurrency with an asyncio semaphore.
Does streaming work with free LLM APIs in Python?
Yes. Pass stream=True and read chunk.choices[0].delta.content. FreeLLMAPI streams in the OpenAI format for every provider.
Can I use async Python with a free LLM API?
Yes. Use AsyncOpenAI with the same base_url and key. Keep the number of parallel requests small, because free tiers limit requests per minute.
Related guides
- Free OpenAI-compatible API base URLs
- Free LLM API for LangChain
- Free LLM API for AI agents
- Free LLM API with function calling
- Free LLM API rate limits
Sources
- OpenAI Python SDK README (async, retries, raw responses, errors) (checked 20 November 2026)
- FreeLLMAPI docs: REST API (chat, streaming, tool calling, embeddings, headers) (checked 20 November 2026)
- FreeLLMAPI docs: degraded mode and failover (checked 20 November 2026)