ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All
Tools
AI-generated keywords:
Large Language Models
ChatGLM
GLM-4
Open-sourcing
Model capabilities
- Introducing the ChatGLM family of large language models:
- Ranging from GLM-130B to GLM-4 (All Tools)
- Significant advancements made by the team in understanding and developing these models over the past year and a half
- Latest ChatGLM models showcase remarkable progress:
- GLM-4 (0116, 0520), GLM-4-Air (0605), and GLM-4 All Tools
- Autonomously utilizing external tools and functions for handling complex tasks effectively
- Performance on par with or surpassing state-of-the-art models like GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro
- Particularly excelling in tasks related to the Chinese language
- Commitment to promoting accessibility and safety in Large Language Models (LLMs):
- Open release of model weights and techniques developed throughout the journey
- Over 10 million downloads on platforms like Hugging Face in 2023 alone
- Future focus on democratizing cutting-edge LLM technologies through open sourcing:
- Developing even more capable models based on gained knowledge
- Pushing boundaries of model capabilities towards teaching machines to think like humans
- Gratitude extended to contributors including data annotators, infrastructure operating staff members, collaborators, partners at Zhipu AI, Tsinghua University, Yuxuan Zhang, Wei Jia from Zhipu AI, teams at Hugging Face, ModelScope, WiseModel for their support in open-sourcing efforts
Authors:
Team GLM,
:,
Aohan Zeng,
Bin Xu,
Bowen Wang,
Chenhui Zhang,
Da Yin,
Dan Zhang,
Diego Rojas,
Guanyu Feng,
Hanlin Zhao,
Hanyu Lai,
Hao Yu,
Hongning Wang,
Jiadai Sun,
Jiajie Zhang,
Jiale Cheng,
Jiayi Gui,
Jie Tang,
Jing Zhang,
Jingyu Sun,
Juanzi Li,
Lei Zhao,
Lindong Wu,
Lucen Zhong,
Mingdao Liu,
Minlie Huang,
Peng Zhang,
Qinkai Zheng,
Rui Lu,
Shuaiqi Duan,
Shudan Zhang,
Shulin Cao,
Shuxun Yang,
Weng Lam Tam,
Wenyi Zhao,
Xiao Liu,
Xiao Xia,
Xiaohan Zhang,
Xiaotao Gu,
Xin Lv,
Xinghan Liu,
Xinyi Liu,
Xinyue Yang,
Xixuan Song,
Xunkai Zhang,
Yifan An,
Yifan Xu,
Yilin Niu,
Yuantao Yang,
Yueyan Li,
Yushi Bai,
Yuxiao Dong,
Zehan Qi,
Zhaoyu Wang,
Zhen Yang,
Zhengxiao Du,
Zhenyu Hou,
Zihan Wang
Abstract: We introduce ChatGLM, an evolving family of large language models that we have been developing over time. This report primarily focuses on the GLM-4 language series, which includes GLM-4, GLM-4-Air, and GLM-4-9B. They represent our most capable models that are trained with all the insights and lessons gained from the preceding three generations of ChatGLM. To date, the GLM-4 models are pre-trained on ten trillions of tokens mostly in Chinese and English, along with a small set of corpus from 24 languages, and aligned primarily for Chinese and English usage. The high-quality alignment is achieved via a multi-stage post-training process, which involves supervised fine-tuning and learning from human feedback. Evaluations show that GLM-4 1) closely rivals or outperforms GPT-4 in terms of general metrics such as MMLU, GSM8K, MATH, BBH, GPQA, and HumanEval, 2) gets close to GPT-4-Turbo in instruction following as measured by IFEval, 3) matches GPT-4 Turbo (128K) and Claude 3 for long context tasks, and 4) outperforms GPT-4 in Chinese alignments as measured by AlignBench. The GLM-4 All Tools model is further aligned to understand user intent and autonomously decide when and which tool(s) touse -- including web browser, Python interpreter, text-to-image model, and user-defined functions -- to effectively complete complex tasks. In practical applications, it matches and even surpasses GPT-4 All Tools in tasks like accessing online information via web browsing and solving math problems using Python interpreter. Over the course, we have open-sourced a series of models, including ChatGLM-6B (three generations), GLM-4-9B (128K, 1M), GLM-4V-9B, WebGLM, and CodeGeeX, attracting over 10 million downloads on Hugging face in the year 2023 alone. The open models can be accessed through https://github.com/THUDM and https://huggingface.co/THUDM.
Submitted to arXiv on 18 Jun. 2024
- Comprehensive Summary
- Key points
- Layman's Summary
- Blog article
Introducing the ChatGLM family of large language models: from GLM-130B to GLM-4 (All Tools). Our team has made significant advancements in understanding and developing these models over the past year and a half. With each generation, we have implemented more effective strategies for model pre-training and alignment. The latest ChatGLM models - including GLM-4 (0116, 0520), GLM-4-Air (0605), and GLM-4 All Tools - showcase remarkable progress in handling complex tasks by autonomously utilizing external tools and functions. These GLM-4 models have demonstrated performance on par with or even surpassing state-of-the-art models like GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro - particularly excelling in tasks related to the Chinese language. Our commitment to promoting accessibility and safety in Large Language Models (LLMs) is evident through the open release of model weights and techniques developed throughout our journey. The open-sourcing efforts of our models have been well-received, with over 10 million downloads on platforms like Hugging Face in 2023 alone. We are currently working on developing even more capable models based on the knowledge gained so far. Moving forward, our focus remains on democratizing cutting-edge LLM technologies through open sourcing while pushing the boundaries of model capabilities towards the goal of teaching machines to think like humans. We extend our gratitude to all data annotators, infrastructure operating staff members, collaborators, partners at Zhipu AI and Tsinghua University who have contributed to the success of ChatGLM. Special thanks to Yuxuan Zhang and Wei Jia from Zhipu AI as well as teams at Hugging Face, ModelScope, WiseModel for their support in open-sourcing efforts.
- - Introducing the ChatGLM family of large language models:
- - Ranging from GLM-130B to GLM-4 (All Tools)
- - Significant advancements made by the team in understanding and developing these models over the past year and a half
- - Latest ChatGLM models showcase remarkable progress:
- - GLM-4 (0116, 0520), GLM-4-Air (0605), and GLM-4 All Tools
- - Autonomously utilizing external tools and functions for handling complex tasks effectively
- - Performance on par with or surpassing state-of-the-art models like GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro
- - Particularly excelling in tasks related to the Chinese language
- - Commitment to promoting accessibility and safety in Large Language Models (LLMs):
- - Open release of model weights and techniques developed throughout the journey
- - Over 10 million downloads on platforms like Hugging Face in 2023 alone
- - Future focus on democratizing cutting-edge LLM technologies through open sourcing:
- - Developing even more capable models based on gained knowledge
- - Pushing boundaries of model capabilities towards teaching machines to think like humans
- - Gratitude extended to contributors including data annotators, infrastructure operating staff members, collaborators, partners at Zhipu AI, Tsinghua University, Yuxuan Zhang, Wei Jia from Zhipu AI, teams at Hugging Face, ModelScope, WiseModel for their support in open-sourcing efforts
SummaryThe ChatGLM family has different models, like GLM-130B to GLM-4, that have been improved a lot in the past year and a half. The latest models, such as GLM-4 and GLM-4-Air, can do complex tasks by themselves using external tools. These models are as good as or even better than other top models like GPT-4 Turbo. They are great at tasks related to the Chinese language. The team is sharing their model weights and techniques openly to make them accessible and safe.
Definitions- Large Language Models (LLMs): Advanced computer programs that can understand and generate human language.
- Autonomously: Acting independently or without direct human control.
- State-of-the-art: Refers to the most advanced or best technology currently available.
- Open sourcing: Making source code freely available for others to use, modify, and distribute.
- Democratizing: Making something accessible to everyone or more widely available.
Introducing the ChatGLM Family of Large Language Models: From GLM-130B to GLM-4 (All Tools)
Over the past year and a half, our team has made significant advancements in understanding and developing large language models (LLMs). Our latest achievement is the ChatGLM family of models, which includes GLM-4 (0116, 0520), GLM-4-Air (0605), and GLM-4 All Tools. These models showcase remarkable progress in handling complex tasks by autonomously utilizing external tools and functions.
Our journey towards creating these cutting-edge LLMs began with GLM-130B, our first model that was pre-trained on a massive amount of data. With each generation, we have implemented more effective strategies for model pre-training and alignment. This has resulted in our current state-of-the-art models - the ChatGLM family.
The performance of our ChatGLM models has been on par with or even surpassing other top-performing LLMs such as GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro. In particular, our models excel in tasks related to the Chinese language - a testament to our commitment to promoting accessibility and safety in LLMs.
One of the key features that sets ChatGLMs apart from other LLMs is their ability to autonomously utilize external tools and functions. This means that they can perform tasks beyond just generating text based on pre-existing knowledge but also use external resources to complete more complex tasks.
We are proud to say that we have open-sourced not only the weights of our models but also techniques developed throughout our journey towards creating them. This openness has been well-received by the community with over 10 million downloads on platforms like Hugging Face in 2023 alone.
Our dedication towards democratizing cutting-edge LLM technologies through open sourcing is evident in our efforts. We are constantly working on developing even more capable models based on the knowledge gained so far.
We would like to extend our gratitude to all data annotators, infrastructure operating staff members, collaborators, and partners at Zhipu AI and Tsinghua University who have contributed to the success of ChatGLM. Special thanks go out to Yuxuan Zhang and Wei Jia from Zhipu AI as well as teams at Hugging Face, ModelScope, WiseModel for their support in our open-sourcing efforts.
As we move forward, our focus remains on pushing the boundaries of LLM capabilities towards the ultimate goal of teaching machines to think like humans. With the ChatGLM family of models and our commitment towards openness and collaboration, we are confident that we can continue making significant advancements in this field.