GLM 5.2 with LlamaIndex: Build RAG Pipelines in Python

LlamaIndex has become the go-to framework for building retrieval-augmented generation (RAG) pipelines in Python. But its documentation and examples almost always assume you're using OpenAI. If you want to swap in a more cost-effective model — like GLM 5.2 (model name: glm-4-plus) from Zhipu AI — the path forward isn't obvious.

This tutorial shows you exactly how to connect GLM 5.2 to LlamaIndex, from a basic document Q&A setup all the way to a ReAct agent with tool calling. You'll write real Python, understand what each piece does, and come away with a working RAG pipeline that costs significantly less to run than an equivalent GPT-4o setup.

The Pain Point: LlamaIndex Docs Assume OpenAI

If you've tried to use a non-OpenAI model with LlamaIndex, you know the friction. The docs show OpenAI(), the examples show OpenAI(), and the error messages reference OpenAI keys. Developers unfamiliar with the GLM API assume it requires a special integration package or a custom LLM wrapper.

It doesn't. GLM 5.2 exposes an OpenAI-compatible REST API at https://open.bigmodel.cn/api/paas/v4/. That means you can use llama-index-llms-openai — the same package used for GPT-4o — and simply point it at Zhipu's endpoint. No new abstractions, no monkey patching, no custom provider class.

Why GLM 5.2 for Your RAG Pipeline

Before diving into code, here's why GLM 5.2 is worth considering for production RAG workloads.

Cost. At $1.40 per million input tokens and $4.40 per million output tokens (Zhipu API pricing as of writing), GLM 5.2 runs 3–4x cheaper than GPT-4o for identical LlamaIndex pipelines. If your pipeline processes thousands of documents per day, this difference compounds quickly.

Context window. GLM 5.2 supports a 1,048,576-token context — a full million tokens. For RAG, this matters: you can stuff entire codebases, legal documents, or research paper collections into a single context and query them without aggressive chunking strategies that risk losing relevant passages.

Benchmark quality. GLM 5.2 scores 89% on GPQA Diamond and 62.1% on SWE-bench Pro, putting it solidly in the top tier for reasoning and code tasks. Its Artificial Analysis Quality Index sits at 51, and it runs at 158 tokens per second on benchmark hardware — fast enough for interactive Q&A use cases.

MIT license. The weights are available under MIT on HuggingFace, which matters if your organization has restrictions on proprietary model APIs. You can run the model on your own infrastructure if needed, or use the Zhipu API for convenience.

Zhipu also offers complementary models for a complete pipeline: embedding-3 (2048-dimensional embeddings) and embedding-2 (1024-dimensional) for vector indexing, GLM-4-Long optimized for long-document tasks, and GLM-4-Air for lightweight/high-throughput workloads. You can mix and match based on your latency and cost constraints.

If you want to explore GLM 5.2's capabilities before writing code, glm5.app provides a playground and model overview to get familiar with the model's behavior.

Prerequisites

Install the required packages:

pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai

Set your Zhipu API key as an environment variable:

export ZHIPU_API_KEY="your-zhipu-api-key-here"

You can get a Zhipu API key at open.bigmodel.cn. For a quick overview of the GLM 5.2 API parameters and available endpoints, see the GLM 5.2 API guide.

Step 1: Configure GLM 5.2 as Your LlamaIndex LLM

The key insight is that llama-index-llms-openai's OpenAI class accepts api_base and api_key overrides. Point those at Zhipu's endpoint and you're done:

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="glm-4-plus",
    api_base="https://open.bigmodel.cn/api/paas/v4/",
    api_key=os.environ["ZHIPU_API_KEY"],
    temperature=0.1,
    max_tokens=2048,
)

# Quick sanity check
response = llm.complete("What is retrieval-augmented generation in one sentence?")
print(response.text)

If the response prints correctly, your connection is working. From here, every LlamaIndex component that accepts an llm parameter will use GLM 5.2.

Step 2: Set Up GLM Embeddings

For a full RAG pipeline you also need embeddings. Zhipu's embedding-3 model (2048-dimensional) is the highest-quality option; embedding-2 (1024-dimensional) is faster and cheaper. Both are accessible through the same OpenAI-compatible endpoint:

from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    model="embedding-3",
    api_base="https://open.bigmodel.cn/api/paas/v4/",
    api_key=os.environ["ZHIPU_API_KEY"],
    embed_batch_size=10,
)

Set these as your global defaults so every LlamaIndex component picks them up automatically:

from llama_index.core import Settings

Settings.llm = llm
Settings.embed_model = embed_model

Step 3: Build a Document Q&A Pipeline

Now load documents, build a vector index, and run queries. Create a folder called ./docs and drop in any PDF, .txt, or Markdown files you want to query.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Load documents from a local folder
documents = SimpleDirectoryReader("./docs").load_data()
print(f"Loaded {len(documents)} document chunks")

# Build a vector index (embeddings are computed here)
index = VectorStoreIndex.from_documents(documents)

# Create a query engine
query_engine = index.as_query_engine(
    similarity_top_k=5,  # retrieve top-5 chunks
    response_mode="tree_summarize",
)

# Ask a question
response = query_engine.query(
    "What are the main conclusions of the research?"
)
print(response.response)
print("\nSources:")
for node in response.source_nodes:
    print(f"  - {node.metadata.get('file_name', 'unknown')} (score: {node.score:.3f})")

The SimpleDirectoryReader handles PDF parsing, text splitting, and metadata extraction. VectorStoreIndex stores embeddings in memory by default — for production, swap in a persistent vector store like Chroma or Weaviate using their respective LlamaIndex integrations.

Step 4: Persist the Index

Computing embeddings for large document sets is slow and costs money. Persist your index to disk and reload it on subsequent runs:

from llama_index.core import StorageContext, load_index_from_storage

# Save
index.storage_context.persist(persist_dir="./index_store")

# Load on next run
storage_context = StorageContext.from_defaults(persist_dir="./index_store")
index = load_index_from_storage(storage_context)
query_engine = index.as_query_engine(similarity_top_k=5)

Step 5: Sub-Question Query Engine for Complex Questions

When a question spans multiple topics, a single retrieval step often misses relevant context. LlamaIndex's SubQuestionQueryEngine breaks the question into sub-questions, retrieves context for each, then synthesizes a final answer — all using your configured LLM (GLM 5.2 in our case):

from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata

# Wrap the base query engine as a tool
tools = [
    QueryEngineTool(
        query_engine=query_engine,
        metadata=ToolMetadata(
            name="research_docs",
            description="Answers questions about the research documents",
        ),
    )
]

sub_question_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=tools,
    use_async=True,
)

response = sub_question_engine.query(
    "Compare the methodology and conclusions across the different papers"
)
print(response.response)

GLM 5.2's 1M-token context means it can synthesize answers across many retrieved chunks without hitting length limits — a meaningful advantage over models with smaller context windows.

Step 6: ReAct Agent with Tool Calling

GLM 5.2 supports function calling, which means LlamaIndex's ReActAgent works out of the box. Here's an agent that can both query your document index and perform web searches (or any other tool you add):

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import FunctionTool

def get_current_date() -> str:
    """Returns today's date."""
    from datetime import date
    return str(date.today())

date_tool = FunctionTool.from_defaults(fn=get_current_date)

doc_tool = QueryEngineTool(
    query_engine=query_engine,
    metadata=ToolMetadata(
        name="document_search",
        description=(
            "Search the loaded research documents for factual information. "
            "Input should be a natural language question."
        ),
    ),
)

agent = ReActAgent.from_tools(
    tools=[doc_tool, date_tool],
    llm=llm,
    verbose=True,
    max_iterations=10,
)

response = agent.chat(
    "What does the research say about long-term outcomes, "
    "and how recent is this information relative to today?"
)
print(response.response)

The verbose=True flag prints each reasoning step so you can watch GLM 5.2 decide which tool to call and when. GLM 5.2's tool-use accuracy is strong enough that it rarely calls the wrong tool or loops unnecessarily.

Want to try GLM 5.2 directly before wiring it into a pipeline? Explore the model on glm5.app — it lets you test prompts, compare outputs, and understand the model's behavior before committing to integration work.

Choosing the Right GLM Model for Each Pipeline Stage

Zhipu offers several models you can mix into your LlamaIndex setup. Here's a practical mapping:

For most RAG pipelines, glm-4-plus for synthesis and embedding-3 for retrieval is the recommended starting point.

Streaming Responses

For interactive applications, enable streaming so the UI updates token-by-token:

streaming_engine = index.as_query_engine(streaming=True)

response = streaming_engine.query("Summarize the key findings")
response.print_response_stream()

GLM 5.2's API supports streaming natively, and the OpenAI-compatible adapter handles the SSE parsing transparently.

Troubleshooting

AuthenticationError: Double-check that ZHIPU_API_KEY is set and that you're passing it as api_key= (not relying on OPENAI_API_KEY). The api_base must be https://open.bigmodel.cn/api/paas/v4/.

Embedding dimension mismatch: If you switch between embedding-2 (1024-dim) and embedding-3 (2048-dim), delete your persisted index and rebuild — stored vectors must match the current model's dimensions.

Slow first query: The first query triggers embedding computation for all loaded documents. Subsequent queries against a persisted index are fast.

model not found error: Confirm the model name is exactly glm-4-plus. GLM 5.2 is the marketing name; the API model identifier is glm-4-plus.

Summary

Integrating GLM 5.2 into LlamaIndex requires no custom LLM class or special provider package. The OpenAI-compatible endpoint at https://open.bigmodel.cn/api/paas/v4/ lets you use llama-index-llms-openai directly, with model="glm-4-plus" as the only model-specific setting. From there, the full LlamaIndex feature set — vector indexing, sub-question engines, ReAct agents, streaming — works without modification.

The practical payoff: a RAG pipeline with GLM 5.2 costs 3–4x less than an equivalent GPT-4o setup, offers a 1M-token context window for large document collections, and delivers benchmark quality that holds up for production workloads.

Sources

• Zhipu AI GLM-4-Plus API documentation: https://open.bigmodel.cn/dev/api
• LlamaIndex OpenAI LLM integration: https://docs.llamaindex.ai/en/stable/examples/llm/openai/
• LlamaIndex embeddings: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings/
• LlamaIndex ReActAgent: https://docs.llamaindex.ai/en/stable/examples/agent/react_agent/
• Artificial Analysis GLM-4-Plus benchmark: https://artificialanalysis.ai/models/glm-4-plus
• GLM-4-Plus on HuggingFace (MIT license): https://huggingface.co/THUDM/glm-4