Guide

Free LLM API in Python

The official openai Python package works with any OpenAI-compatible API. You change the base URL and the key. This guide uses FreeLLMAPI, which pools 45 free provider tiers behind one key and moves to the next provider when a limit stops a request. Each example below is complete and runs as it is.

Setup

Install FreeLLMAPI. It runs in Docker. Open the dashboard at http://localhost:3001, add free keys from two or more providers on the Keys page and copy the unified key. Then install the Python package.

Your Python code uses only the unified key. The provider keys stay in FreeLLMAPI, encrypted in its database. If you change or add a provider key, you do not change your code.

Install

curl -fsSL https://freellmapi.co/install.sh | bash

python3 -m pip install -U openai
export FREELLMAPI_API_KEY=<unified-key>

client.py

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key=os.environ["FREELLMAPI_API_KEY"],
    max_retries=3,   # the SDK retries 429, 408, 409 and 5xx errors
    timeout=60,
)

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Name three uses of Python in one line."}],
)
print(resp.choices[0].message.content)

The model auto follows your fallback chain on the dashboard. You can also use auto:fast, auto:smart or auto:reliable, a profile such as auto:coding, or a fixed model ID.

Streaming and response headers

Stream the answer

from client import client

stream = client.chat.completions.create(
    model="auto:fast",
    messages=[{"role": "user", "content": "Write a haiku about Python."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
print()

Each response has an X-Routed-Via header with the provider and model that answered. To read headers with the SDK, use with_raw_response and call parse() to get the normal object.

Read the routing headers

from client import client

raw = client.chat.completions.with_raw_response.create(
    model="auto",
    messages=[{"role": "user", "content": "hi"}],
)
print("Served by:", raw.headers.get("x-routed-via"))
print("Failed attempts:", raw.headers.get("x-fallback-attempts", "0"))
print(raw.parse().choices[0].message.content)

Tool calling

Send OpenAI-style tools. The router passes them to OpenAI-compatible providers and translates them for Gemini. The answer comes back in the same format for every provider.

One tool call and the final answer

import json
from client import client

tools = [{
    "type": "function",
    "function": {
        "name": "get_exchange_rate",
        "description": "Get the exchange rate between two currencies.",
        "parameters": {
            "type": "object",
            "properties": {
                "base": {"type": "string"},
                "quote": {"type": "string"},
            },
            "required": ["base", "quote"],
        },
    },
}]

messages = [{"role": "user", "content": "How many euros is 250 US dollars?"}]
first = client.chat.completions.create(model="auto", messages=messages, tools=tools)
msg = first.choices[0].message

if msg.tool_calls:
    messages.append(msg)
    for call in msg.tool_calls:
        args = json.loads(call.function.arguments)
        result = {"base": args["base"], "quote": args["quote"], "rate": 0.92}  # your real lookup
        messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
    final = client.chat.completions.create(model="auto", messages=messages, tools=tools)
    print(final.choices[0].message.content)
else:
    print(msg.content)

Some providers return a tool call as plain text. FreeLLMAPI converts these answers into real tool_calls, so your code reads one format. Always check msg.tool_calls before you read msg.content, because a model can also answer without a tool.

Use a model with tool calling. These free models support tools and have the fastest measured speed today:

Fast free models with tool calling
ModelFree onContextFree limits (best offer)ToolsVision
Qwen3.8 27BElectronHub, Groq, OVH AI Endpoints262K30 req/min
250 req/day
70,000 tokens/min
~6M
YesYes
Nemotron 3.5 LightningElectronHub, Kilo Gateway, OpenRouter1M20 req/min
50 req/day
50 req/day account-wide (1,000 after a $10 purchase)
YesNo
GPT-OSS 120BCloudflare Workers AI, ElectronHub, Groq +1 more131K30 req/min
1,000 req/day
8,000 tokens/min
200,000 tokens/day
~6M
YesNo
GPT-OSS 20BCloudflare Workers AI, Groq, NVIDIA NIM +1 more131K30 req/min
1,000 req/day
8,000 tokens/min
200,000 tokens/day
~6M
YesNo
GPT-OSS Safeguard 20BGroq131K30 req/min
1,000 req/day
8,000 tokens/min
200,000 tokens/day
~6M
YesNo
Gemini 3.5 Flash LiteElectronHub, Google AI Studio1M15 req/min
20 req/day
250,000 tokens/min
~3M
YesYes
Step 3.7 FlashElectronHub256K$0.25/week shared recurring creditYesYes
Gemini 3.1 Flash-LiteDreamPrompting, Google AI Studio1M100 req/min
5,000 req/day
500,000 tokens/day
500K tokens/day · 5,000 req/day
YesYes
Nemotron 3.5 Lightning 30BRequesty1M200 req/day
200 rpd
YesNo
Gemini 2.5 Flash-LiteElectronHub, Google AI Studio1M15 req/min
20 req/day
250,000 tokens/min
~3M
YesYes

Showing 10 of 146 matching free models. See all models.

Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.

Async batches inside free limits

AsyncOpenAI sends many requests at the same time. Free tiers limit requests per minute, so limit the concurrency with a semaphore. The example also retries when every provider is at its limit.

batch.py

import asyncio
import os

import openai
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="http://localhost:3001/v1",
    api_key=os.environ["FREELLMAPI_API_KEY"],
    max_retries=2,
)
limit = asyncio.Semaphore(4)  # at most 4 requests at a time

async def summarize(text: str) -> str:
    async with limit:
        for attempt in range(5):
            try:
                resp = await client.chat.completions.create(
                    model="auto:fast",
                    messages=[
                        {"role": "system", "content": "Summarize in one sentence."},
                        {"role": "user", "content": text},
                    ],
                )
                return resp.choices[0].message.content
            except openai.RateLimitError as err:
                header = err.response.headers.get("retry-after", "")
                wait = int(header) if header.isdigit() else 2 ** attempt
                await asyncio.sleep(min(wait, 60))
        return "ERROR: rate limited"

async def main():
    docs = [f"Document {i}: Python is a programming language." for i in range(20)]
    results = await asyncio.gather(*(summarize(d) for d in docs))
    for r in results:
        print(r)
    await client.close()

asyncio.run(main())

The router returns 429 only when every key in the chain is at its limit. The response then has a Retry-After header. A small semaphore value and more provider keys give the best results for large batches.

Choose a model string for each task

The model string tells the router how to select a model. You can use a different string in each call. You do not need to change the dashboard.

TaskModel stringReason
Chat answers for usersautoFollows the fallback chain that you set on the dashboard
Classification, extraction, short summariesauto:fastRanks enabled models by measured speed
Hard questions, code review, planningauto:smartRanks enabled models by intelligence
Scheduled jobs that must finishauto:reliableRanks enabled models by recent success rate
A separate chain for one appauto:<profile>Uses a named fallback chain
Tests that must be repeatableA fixed model IDSame model each time, but it stops at that model's limit

To see which models your keys can call now, request the model list with a filter. The full list also contains models that have no key yet.

List the models that can answer now

import os
import httpx

r = httpx.get(
    "http://localhost:3001/v1/models",
    params={"execution_status": "ready"},
    headers={"Authorization": f"Bearer {os.environ['FREELLMAPI_API_KEY']}"},
    timeout=30,
)
r.raise_for_status()
for m in r.json()["data"]:
    print(m["id"])

A model with the status exhausted has keys, but all of them are at a limit now. The status clears when the cooldowns end. A model with the status needsKey has no enabled key.

For a conversation, send an X-Session-Id header with extra_headers. FreeLLMAPI then keeps that conversation on one model for 30 minutes. Without the header, it uses a hash of the first user message as the session key.

Embeddings

Embed text

from client import client

resp = client.embeddings.create(
    model="bge-m3",  # one embedding family
    input=["Python is a programming language.", "A snake is a reptile."],
)
print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dimensions")

Embedding failover never changes the model. It only moves between providers that serve the same family, so your vectors stay compatible. The value auto uses the default family that you set on the dashboard.

Send many texts in one input list. One request with 32 texts uses one request from the free limit, not 32. Keep each text under the maximum input of the family. Store the family name with your vectors, so that you embed queries with the same family later.

Errors and fixes

ExceptionCauseFix
openai.AuthenticationErrorWrong unified keyCopy the key again from the Keys page
openai.APIConnectionErrorFreeLLMAPI is not running, or wrong portOpen http://localhost:3001 in the browser
openai.NotFoundErrorThe base URL has no /v1Use http://localhost:3001/v1
openai.RateLimitErrorAll keys in the chain are at their limitAdd providers, lower the concurrency, wait for Retry-After
openai.BadRequestError with auto:nameThe profile name does not existCheck the profile name on the dashboard
openai.UnprocessableEntityError (422) with imagesNo vision model is in your chainEnable a vision model in the fallback chain

Free tiers have no SLA. Latency changes during the day. The strongest models reach their daily limits first, and many limits reset at midnight UTC. Do not use a free gateway for a service that must answer within a fixed time.

Use every one of these through one key. FreeLLMAPI is an open-source, self-hosted router that puts all these free tiers behind a single OpenAI-compatible endpoint and fails over when one is rate-limited. Browse the catalog or go live.

Frequently asked questions

Is there a free LLM API for Python?

Yes. Many providers have free tiers that work with the openai Python package. FreeLLMAPI combines them behind one base URL and one key, with failover between providers.

How do I use the openai Python library with a different base URL?

Pass base_url to OpenAI() or AsyncOpenAI(), for example base_url="http://localhost:3001/v1". Pass your gateway key as api_key.

How do I handle rate limit errors in Python?

Set max_retries on the client. Catch openai.RateLimitError, read the Retry-After header and wait. Limit concurrency with an asyncio semaphore.

Does streaming work with free LLM APIs in Python?

Yes. Pass stream=True and read chunk.choices[0].delta.content. FreeLLMAPI streams in the OpenAI format for every provider.

Can I use async Python with a free LLM API?

Yes. Use AsyncOpenAI with the same base_url and key. Keep the number of parallel requests small, because free tiers limit requests per minute.

Sources

All guides · Browse all free models

Keep reading

All posts →
Guide

Best free LLM APIs 2026

Ten genuinely free LLM APIs worth using — ranked, with real limits.

Guide

Unlimited free AI tokens?

No free AI API is unlimited. The closest real thing: 10.1 billion free tokens a month behind one key.

Guide

How to get a free LLM API key

Get a working key in minutes — no credit card required.