Guide

Free LLM for LlamaIndex

LlamaIndex has two classes for OpenAI-compatible APIs: OpenAILike for chat and OpenAILikeEmbedding for embeddings. This guide points both at FreeLLMAPI, which pools 45 free provider tiers behind one key. You build a retrieval-augmented generation (RAG) index, query it and add an agent. The code handles rate limits.

Install the packages

Install FreeLLMAPI and add free keys from two or more providers on the dashboard Keys page. Copy the unified key. Then install the LlamaIndex packages.

Install

curl -fsSL https://freellmapi.co/install.sh | bash

pip install -U llama-index-core \
  llama-index-llms-openai-like \
  llama-index-embeddings-openai-like

export FREELLMAPI_BASE_URL=http://localhost:3001/v1
export FREELLMAPI_API_KEY=<unified-key>

Do not use the plain OpenAI class from llama-index-llms-openai with model names that OpenAI does not know. That class checks model names and context sizes against a list of OpenAI models. OpenAILike does not, so you set these values yourself.

Configure the LLM and the embedding model

settings.py

import os
from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai_like import OpenAILikeEmbedding

BASE = os.environ["FREELLMAPI_BASE_URL"]
KEY = os.environ["FREELLMAPI_API_KEY"]

Settings.llm = OpenAILike(
    model="auto",
    api_base=BASE,
    api_key=KEY,
    is_chat_model=True,              # use /v1/chat/completions
    is_function_calling_model=True,  # needed for FunctionAgent
    context_window=32000,            # the smallest window in your chain
    max_retries=5,
    timeout=90,
)

Settings.embed_model = OpenAILikeEmbedding(
    model_name="bge-m3",  # one family, 1024 dimensions
    api_base=BASE,
    api_key=KEY,
    embed_batch_size=16,
    max_retries=10,
)
  • is_chat_model=True: without it, LlamaIndex sends requests to the legacy completions endpoint.
  • context_window: LlamaIndex uses this value to decide how many retrieved chunks fit in the prompt. With auto, the router can select different models. Use the smallest context window in your chain.
  • is_function_calling_model=True: set it only when your chain has tool-calling models.
  • embed_batch_size: each batch is one request. Larger batches use fewer requests per minute.

A RAG prompt carries several retrieved chunks, so the chat model needs room for them. Make a fallback chain profile with long-context models for RAG. Then set model="auto:<profile>" and raise context_window to the smallest window in that profile. These free models have the largest context windows today:

Free models with the largest context windows
ModelFree onContextFree limits (best offer)ToolsVision
OpenAI: GPT-6 LunaElectronHub1.1M$0.25/week shared free creditsNoNo
Gemini 3.8 FlashAPINex, Google AI Studio1M5 req/min
10M weighted tokens/day shared across APINex free routes; 3x token multiplier; 5 requests/minute/account
YesYes
Gemini 3.7 FlashGoogle AI Studio1M10 req/min
20 req/day
250,000 tokens/min
~3M
YesYes
DeepSeek V4 FlashAMD Radeon Cloud, BazaarLink, ElectronHub1M10 req/min
50 req/day
Shared rate limit
YesNo
DeepSeek V4 Flash 0731ElectronHub, Router91M10,000 credits/month (~$1), shared; account-anniversary resetNoNo
Gemini 3.6 FlashGoogle AI Studio1M10 req/min
20 req/day
250,000 tokens/min
~3M
YesYes
GLM-5.2ModelScope, Router91M100 req/day
Daily login: 200 credits + 50 for linked Alibaba Cloud; 24h expiry; shared; noncommercial experience
NoNo
Gemini 3.5 FlashGoogle AI Studio1M10 req/min
20 req/day
250,000 tokens/min
~3M
YesYes

Showing 8 of 115 matching free models. See all models.

Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.

To keep one user on one model, send an X-Session-Id header. OpenAILike accepts default_headers, for example default_headers={"X-Session-Id": "user-42"}. FreeLLMAPI keeps a session on the same model for 30 minutes. This gives answers in the same style across one chat.

Build and query a RAG index

The script reads files from a folder, makes embeddings, saves the index and answers a question. Run it once to build the index. Later runs load the saved index and do not embed the files again.

rag.py

import os
import settings  # noqa: F401  (sets Settings.llm and Settings.embed_model)
from llama_index.core import (
    SimpleDirectoryReader,
    StorageContext,
    VectorStoreIndex,
    load_index_from_storage,
)

PERSIST = "./storage"


def load_index() -> VectorStoreIndex:
    if os.path.isdir(PERSIST):
        return load_index_from_storage(StorageContext.from_defaults(persist_dir=PERSIST))
    docs = SimpleDirectoryReader("./docs").load_data()
    index = VectorStoreIndex.from_documents(docs, show_progress=True)
    index.storage_context.persist(persist_dir=PERSIST)
    return index


if __name__ == "__main__":
    query_engine = load_index().as_query_engine(similarity_top_k=4, streaming=True)
    response = query_engine.query("How do I rotate the API keys?")
    response.print_response_stream()
    for node in response.source_nodes:
        print("\nSource:", node.metadata.get("file_name"), round(node.score or 0, 3))
Important: keep one embedding family for the whole index. FreeLLMAPI never changes the embedding model during failover. It only moves between providers that serve the same family. If you change the family, delete ./storage and build the index again.

These embedding models are free in today's catalog. The dimensions column shows the vector size of each family.

ModelProviderDimensionsMax inputFree limit
Gemini Embedding
gemini-embedding-001
Google AI Studio30722048100 rpm · 1K req/day
Nemotron Embed VL 1B
nvidia/llama-nemotron-embed-vl-1b-v2
NVIDIA NIM20488192~40 rpm
Nemotron 3 Embed 1B (OR free)
nvidia/nemotron-3-embed-1b:free
OpenRouter20483276820 rpm · shared free daily cap
SEA-LION E5 Embedding 600M
aisingapore/SEA-LION-E5-Embedding-600M
SEA-LION10248192free · 10 rpm
SEA-LION ModernBERT Embedding 300M
aisingapore/SEA-LION-ModernBERT-Embedding-300M
SEA-LION10248192free · 10 rpm
SEA-LION ModernBERT Embedding 600M
aisingapore/SEA-LION-ModernBERT-Embedding-600M
SEA-LION10248192free · 10 rpm
BGE-M3
@cf/baai/bge-m3
Cloudflare Workers AI1024819210K neurons/day (shared)
EmbeddingGemma 300M
@cf/google/embeddinggemma-300m
Cloudflare Workers AI768204810K neurons/day (shared)
Qwen3 Embedding 0.6B
@cf/qwen/qwen3-embedding-0.6b
Cloudflare Workers AI1024819210K neurons/day (shared)
Cohere Embed v4
embed-v4.0
Cohere15361280001K calls/mo (shared w/ chat)
Gemini Embedding 2
gemini-embedding-2
Google AI Studio30728192free-tier rate limited
Nemotron 3 Embed 1B
nvidia/nemotron-3-embed-1b
NVIDIA NIM20484096~40 rpm
PLaMo Embedding 1B (CF)
@cf/pfnet/plamo-embedding-1b
Cloudflare Workers AI2048409610K neurons/day (shared)
BGE Small EN v1.5 (CF)
@cf/baai/bge-small-en-v1.5
Cloudflare Workers AI38451210K neurons/day (shared)
BGE Base EN v1.5 (CF)
@cf/baai/bge-base-en-v1.5
Cloudflare Workers AI76851210K neurons/day (shared)
BGE Large EN v1.5 (CF)
@cf/baai/bge-large-en-v1.5
Cloudflare Workers AI102451210K neurons/day (shared)
Mistral Embed
mistral-embed
Mistral10248192free mode · monthly cap (shared)
Codestral Embed
codestral-embed
Mistral15368192free mode · monthly cap (shared)
Llama Text Embed v2 (Pinecone)
llama-text-embed-v2
Pinecone102420485M tokens/mo per model
Multilingual E5 Large (Pinecone)
multilingual-e5-large
Pinecone10245075M tokens/mo per model
Nemotron 3 Embed 1B (Speka)
nvidia/nemotron-3-embed-1b
Speka20488192$1/month shared across Speka models
OpenAI: Text Embedding 3 Small (ElectronHub)
text-embedding-3-small
ElectronHub15368192$0.25/week shared free credits
OpenAI: Text Embedding 3 Large (ElectronHub)
text-embedding-3-large
ElectronHub30728192$0.25/week shared free credits
Nemotron 3 Embed 8B (Inferbase)
nvidia-nemotron-3-embed-8b
Inferbase409681925,000 requests/month; 100 requests and $0.50 usage/day shared
Qwen 3 Embedding 0.6B (Inferbase)
qwen3-embedding-0-6b
Inferbase102481925,000 requests/month; 100 requests and $0.50 usage/day shared
Qwen 3 Embedding 4B (Inferbase)
qwen3-embedding-4b
Inferbase256081925,000 requests/month; 100 requests and $0.50 usage/day shared
Qwen 3 Embedding 8B (Inferbase)
qwen3-embedding-8b
Inferbase409681925,000 requests/month; 100 requests and $0.50 usage/day shared
LFM2.5 Embedding 350M (LLMTR)
liquid/lfm-2.5-embedding-350m-free
LLMTR1024512Ongoing zero-price route; provider terms apply
BGE-M3 (SEA-LION)
BAAI/bge-m3
SEA-LION10248192free · 10 rpm
Cohere Embed English v3
embed-english-v3.0
Cohere10245121K calls/mo (shared w/ chat)
Qwen3 Embedding 8B (OVH)
Qwen3-Embedding-8B
OVH AI Endpoints409632768free · anonymous 2 req/min per IP
BGE-M3 (OVH)
bge-m3
OVH AI Endpoints10248192free · anonymous 2 req/min per IP
BGE Multilingual Gemma2 (OVH)
bge-multilingual-gemma2
OVH AI Endpoints35848192free · anonymous 2 req/min per IP
Mistral: Codestral Embed (ElectronHub)
codestral-embed
ElectronHub15368192$0.25/week shared free credits
Mistral: Mistral Embed (ElectronHub)
mistral-embed
ElectronHub10248192$0.25/week shared free credits
Google: Gemini Embedding 2 (ElectronHub)
gemini-embedding-2
ElectronHub30728192$0.25/week shared free credits
Google: Gemini Embedding 001 (ElectronHub)
gemini-embedding-001
ElectronHub30722048$0.25/week shared free credits
OpenAI: Text Embedding Ada 002 (ElectronHub)
text-embedding-ada-002
ElectronHub15368192$0.25/week shared free credits
Cohere Embed English Light v3
embed-english-light-v3.0
Cohere3845121K calls/mo (shared w/ chat)
Cohere Embed Multilingual v3
embed-multilingual-v3.0
Cohere10245121K calls/mo (shared w/ chat)
Cohere Embed Multilingual Light v3
embed-multilingual-light-v3.0
Cohere3845121K calls/mo (shared w/ chat)

Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.

Add an agent with FunctionAgent

FunctionAgent uses tool calling. You can give it the query engine as a tool and add your own Python functions.

agent.py

import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import QueryEngineTool
from llama_index.core import Settings
from rag import load_index

docs_tool = QueryEngineTool.from_defaults(
    query_engine=load_index().as_query_engine(similarity_top_k=4),
    name="project_docs",
    description="Answers questions about the project documentation.",
)

def open_tickets(team: str) -> int:
    """Return the number of open tickets for a team."""
    return {"billing": 4, "platform": 11}.get(team, 0)

agent = FunctionAgent(
    tools=[docs_tool, open_tickets],
    llm=Settings.llm,
    system_prompt="Use the tools. Cite the document name in each answer.",
)

async def main():
    result = await agent.run("How many platform tickets are open, and how do we rotate keys?")
    print(result)

asyncio.run(main())

For agent work, select models with tool calling. Make a fallback chain profile with only these models, and use auto:<profile> as the model name.

Free models with tool calling and 128K+ context
ModelFree onContextFree limits (best offer)ToolsVision
GLM 5.3UnoRouter1M1 req/min
1 req/min per model
YesNo
GLM 5.3 FlashNVIDIA NIM, UnoRouter1M40 req/min
40 RPM
YesYes
Gemini 3.8 FlashGoogle AI Studio1MRecurring provider quotaYesYes
Qwen3.8 Flash NextAMD Radeon Cloud262KRecurring daily account allowanceYesNo
Qwen3.8 FlashElectronHub1M$0.25/week shared recurring creditYesYes
Gemini 3.7 FlashGoogle AI Studio1M10 req/min
20 req/day
250,000 tokens/min
~3M
YesYes
GPT-5.6 LunaElectronHub1M$0.25/week shared recurring creditYesYes
Dots3 Note PreviewKilo Gateway, OpenRouter512K20 req/min
50 req/day
50 req/day account-wide (1,000 after a $10 purchase)
YesYes
DeepSeek V4 FlashAMD Radeon Cloud, BazaarLink, ElectronHub1M10 req/min
50 req/day
Shared rate limit
YesNo
Qwen3.8 27BElectronHub, Groq, OVH AI Endpoints262K30 req/min
250 req/day
70,000 tokens/min
~6M
YesYes

Showing 10 of 109 matching free models. See all models.

Live data from the FreeLLMAPI catalog (version 2026.11.19, checked 19 November 2026). Free limits change often. The catalog is checked every day.

Keep ingestion inside free limits

Ingestion sends many embedding requests in a short time. Free embedding tiers have low limits. For example, some allow 10 requests per minute and others count calls per month. Plan the first ingestion before you start it.

  • Use large chunks. Fewer chunks mean fewer vectors and fewer requests. A chunk size near 1,024 tokens is a good start.
  • Use batches. Set embed_batch_size to 16 or more. One request then carries many chunks.
  • Save the index. Persist the index after the first run. Do not embed the same files again.
  • Keep retries on. OpenAILikeEmbedding retries up to 10 times by default. Keep this value for long ingestion jobs.
  • Ingest in parts. For thousands of files, ingest one folder at a time.

Queries are cheaper than ingestion. Each query uses one embedding request and one chat request. A chat request with four chunks of 1,024 tokens sends about 5,000 tokens. Free tiers with low tokens-per-minute limits can reject this. FreeLLMAPI then tries the next provider.

Troubleshooting

ProblemCauseFix
Unknown model 'auto'You used the OpenAI classUse OpenAILike
Requests go to /v1/completionsis_chat_model is not setSet is_chat_model=True
The agent does not call toolsis_function_calling_model is off, or the model has no tool callingSet it to True and use a tools profile
Error 429 during ingestionThe embedding family is at its limit on all providersUse larger batches, wait for Retry-After and run again
Bad answers after a change of embedding modelVectors from two families in one indexDelete the saved index and build it again
Context length errorcontext_window is larger than the model that answeredSet a smaller value or use a long-context profile

To see which provider answered a request, use the FreeLLMAPI dashboard. The Analytics page shows requests, latency and errors for each provider and model. Each response also has an X-Routed-Via header. If the router changed provider, the X-Fallback-Trail header names each failed attempt and the reason.

Use every one of these through one key. FreeLLMAPI is an open-source, self-hosted router that puts all these free tiers behind a single OpenAI-compatible endpoint and fails over when one is rate-limited. Browse the catalog or go live.

Frequently asked questions

Can I use LlamaIndex with a free LLM?

Yes. Use OpenAILike with api_base set to an OpenAI-compatible URL. FreeLLMAPI gives one URL for many free provider tiers. Each tier has its own limits.

What is the difference between OpenAI and OpenAILike in LlamaIndex?

The OpenAI class checks model names against OpenAI models. OpenAILike accepts any model name. You set the context window and the chat and function-calling flags yourself.

Which free embedding model is best for LlamaIndex?

Select a family with enough free requests for your ingestion and keep it for the whole index. The bge-m3 family is free on several providers, so failover can move between them.

Does FunctionAgent work with free models?

Yes, if the model supports tool calling and you set is_function_calling_model=True. Use a fallback chain profile that has only tool-calling models.

How do I avoid rate limits when I build a LlamaIndex index?

Use large chunks, set embed_batch_size to 16 or more, persist the index and ingest large folders in parts.

Sources

All guides · Browse all free models

Keep reading

All posts →
Guide

Best free LLM APIs 2026

Ten genuinely free LLM APIs worth using — ranked, with real limits.

Guide

Unlimited free AI tokens?

No free AI API is unlimited. The closest real thing: 10.1 billion free tokens a month behind one key.

Guide

How to get a free LLM API key

Get a working key in minutes — no credit card required.