GLM 5 — Next-Generation Frontier ModelGLM 5
GLM 5.2 on Hugging Face: Load and Run with Transformers

GLM 5.2 on Hugging Face: Load and Run with Transformers

How to load and run GLM 5.2 (GLM-4) on Hugging Face using the Transformers library — model IDs, hardware requirements, quantization, and inference code examples.

Zhipu AI's GLM-4 family has become one of the most capable open-weight model series available on Hugging Face. Whether you want to run inference locally, fine-tune for a specific domain, or deploy at the edge without sending data to an external API, the THUDM checkpoints on Hugging Face give you a practical on-ramp. This tutorial walks through everything you need: the right model IDs, hardware requirements, quantization options, and working inference code.

Pain Point: Why Local GLM Inference Matters

Most developers first encounter GLM-4 through the Zhiyu AI API, which serves the full-scale GLM-4-Plus frontier model (a 753B total / 40B active MoE architecture). That API is excellent for production use — fast at 158 tokens per second, with a 1M-token context window — but it is not the right tool for every situation.

Common reasons developers want a local checkpoint instead:

  • Privacy and compliance — sending code, documents, or user data to a third-party API is blocked by policy in many organizations.
  • Fine-tuning — adapting the model on proprietary data requires access to weights.
  • Edge and embedded deployment — latency-sensitive or offline scenarios where API round-trips are unacceptable.
  • Cost control — sustained high-volume workloads can be cheaper to run on owned hardware than on a per-token API.

The Hugging Face ecosystem solves all four. The THUDM organization publishes open-weight GLM-4 checkpoints under the MIT license, which gives you maximum flexibility to use, modify, and redistribute.

Which Model Should You Use?

Before writing any code, you need to pick the right checkpoint. The GLM-4 series on Hugging Face has several variants:

CheckpointParametersBest For
THUDM/glm-4-9b-chat9BChat/instruction following, easiest to run
THUDM/glm-4-9b9BBase model, fine-tuning starting point
GLM-4-Plus (Zhipu API)753B total / 40B activeFrontier-quality, production API only

For most developers, THUDM/glm-4-9b-chat is the right starting point. It is instruction-tuned for dialogue, fits on a single consumer GPU at full precision, and uses the same architecture family as the larger frontier models. The full GLM-4-Plus scale model is not available as open weights — that capability is served exclusively through the API.

For a deeper look at the API version and how it compares in benchmarks, see the GLM 5.2 API guide on this blog.

Competitive Differentiator

GLM-4-9B-Chat is one of the strongest open-weight 9B models for Chinese-English bilingual tasks. At release, it outperformed several competing 9B-class models on standard Chinese language benchmarks while remaining fully competitive on English tasks. The MIT license is unusually permissive for a model of this quality — unlike some competing checkpoints released under custom research licenses, you can use GLM-4-9B-Chat in commercial products without restriction. And because it integrates natively with the Hugging Face Transformers library, you can drop it into any existing Transformers-based pipeline without custom inference code.

Hardware Requirements

Before you install anything, confirm your environment can handle the checkpoint:

  • Full precision (bfloat16): approximately 18 GB VRAM. A single NVIDIA RTX 3090 or RTX 4090 (24 GB) is sufficient.
  • 8-bit quantization: approximately 9-10 GB VRAM.
  • 4-bit quantization: approximately 5-6 GB VRAM. Fits on cards like an RTX 3060 12 GB or RTX 4070.
  • CPU-only: possible via Accelerate's disk offloading, but inference will be very slow — only practical for testing.

For production local deployment, a single RTX 4090 at 4-bit quantization gives you a good balance of quality and speed.

Installation

Install the required packages. Using a fresh virtual environment is recommended to avoid dependency conflicts.

pip install transformers torch accelerate bitsandbytes
  • transformers — Hugging Face model hub client and inference utilities
  • torch — PyTorch backend (install the CUDA variant from pytorch.org for GPU support)
  • accelerate — device-mapping and multi-GPU support
  • bitsandbytes — quantization backend for 4-bit and 8-bit inference

Loading the Model at Full Precision

If your GPU has 24 GB of VRAM, the simplest approach is to load at bfloat16 precision with no quantization. This gives the best output quality.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "THUDM/glm-4-9b-chat"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",          # automatically assigns layers across available GPUs
    trust_remote_code=True,
)
model.eval()

# Build a simple chat prompt using the model's chat template
messages = [
    {"role": "user", "content": "Explain the difference between MoE and dense transformer architectures in two paragraphs."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )

# Decode only the newly generated tokens
generated = output_ids[0][inputs.shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=True))

A few things to note:

  • trust_remote_code=True is required because GLM-4 includes custom tokenizer and modeling code that Transformers loads from the model repository.
  • device_map="auto" lets Accelerate handle placement. On a single GPU it puts everything on that device; on multi-GPU systems it shards automatically.
  • apply_chat_template formats the messages according to GLM-4's expected instruction format — using raw string formatting instead risks confusing the model.

Loading with 4-bit Quantization

For GPUs with less VRAM, use BitsAndBytesConfig to load at 4-bit precision. This reduces memory consumption to roughly 5-6 GB while preserving most of the model's capability for conversational tasks.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

MODEL_ID = "THUDM/glm-4-9b-chat"

# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NormalFloat4 — best quality for LLM weights
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,     # secondary quantization for extra compression
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        inputs,
        max_new_tokens=256,
        do_sample=False,   # greedy decoding for code generation
    )

generated = output_ids[0][inputs.shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=True))

The nf4 quantization type with double quantization is the recommended configuration for inference quality. If you are preparing for QLoRA fine-tuning rather than just inference, this is also the correct starting configuration — peft integrates directly with models loaded this way.

To use 8-bit instead, replace load_in_4bit=True with load_in_8bit=True and remove the bnb_4bit_* parameters. Eight-bit gives slightly better quality at roughly double the VRAM cost compared to 4-bit.

Streaming Output

For interactive applications, waiting for the full output before displaying anything creates a poor user experience. Hugging Face Transformers supports streaming through TextIteratorStreamer.

from transformers import TextIteratorStreamer
from threading import Thread

streamer = TextIteratorStreamer(
    tokenizer,
    skip_prompt=True,
    skip_special_tokens=True,
)

generation_kwargs = dict(
    input_ids=inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    streamer=streamer,
)

# Run generation in a background thread so we can iterate the streamer in the main thread
thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

for token_text in streamer:
    print(token_text, end="", flush=True)

thread.join()
print()  # newline after streaming ends

The TextIteratorStreamer yields decoded text fragments as they are generated. This pattern works with both full-precision and quantized models without any changes to the streamer code.

Understanding the Relationship Between HF Models and GLM-4-Plus

It is worth being explicit about what these Hugging Face checkpoints are and are not.

The glm-4-9b-chat checkpoint is a 9-billion-parameter model. The production API model, glm-4-plus, is a Mixture-of-Experts architecture with 753B total parameters and 40B active parameters per forward pass — a fundamentally different scale. Public open weights for the full MoE model are not available as of writing.

The 9B checkpoint is part of the same research lineage and shares architectural DNA with the larger model, but performance on hard benchmarks will differ. The API version achieves 89% on GPQA Diamond and 62.1% on SWE-bench Pro. The 9B local variant is capable for a wide range of tasks — especially Chinese-English dialogue, coding assistance, and instruction following — but you should evaluate it on your specific use case rather than assuming benchmark parity.

If you need frontier-level quality and are comfortable with an API, try GLM 5.2 directly on glm5.app — it provides access to the full GLM-4-Plus model without any infrastructure setup.

Common Issues and Fixes

Out of memory on load: Use device_map="auto" and enable quantization. If VRAM is still insufficient, add offload_folder="offload" to from_pretrained to spill layers to disk (slow but functional for testing).

trust_remote_code warning: This is expected. GLM-4 includes custom code in the repository that Transformers needs to execute. Only set this flag for models from trusted sources — the THUDM organization is Zhipu AI's official Hugging Face account.

Output contains unexpected tokens: Make sure you are using apply_chat_template rather than manually constructing prompt strings. The model is sensitive to the exact formatting of turn boundaries.

Slow tokenization on first run: The tokenizer downloads and caches its vocabulary on first use. Subsequent calls are fast.

Next Steps

Once you have the model running locally, common next steps include:

  • Fine-tuning with QLoRA — load the model with load_in_4bit=True and add a PEFT adapter for task-specific training on your own data.
  • vLLM deployment — for serving the model to multiple concurrent users, vLLM provides significantly higher throughput than naive model.generate() calls.
  • Evaluation — run the model against your domain-specific test cases before committing to a deployment architecture.

For tasks where you do not need local control, explore GLM 5.2 capabilities on glm5.app — it is the fastest way to prototype against the full-scale model before deciding on a deployment strategy.

Sources

Start Using GLM 5 Today

Try GLM 5 free — reasoning, coding, agents, and image generation in one platform.

Try GLM 5 Free