# How is ChatGPT's behavior changing over time?

**Authors:** Lingjiao Chen, Matei Zaharia, James Zou  
**arXiv:** [2307.09009](https://arxiv.org/abs/2307.09009)  
**Date:** 2023-07-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.09009v1/

## Key points

- Lingjiao Chen, Matei Zaharia, and James Zou evaluate the performance and behavior of GPT-3.5 and GPT-4
- Study focuses on March 2023 and June 2023 versions of the models
- Four diverse tasks: solving math problems, answering sensitive/dangerous questions, generating code, and visual reasoning
- Both GPT-3.5 and GPT-4 show significant variations in performance and behavior over time
- GPT-4 (March 2023) has high accuracy in identifying prime numbers but declines drastically in June 2023
- GPT-3.5 (June 2023) outperforms its March 2023 version in identifying prime numbers
- GPT-4 becomes less willing to answer sensitive questions in June compared to March
- More formatting mistakes in code generation observed for both models during June compared to March
- Continuous monitoring of LLM quality is necessary as behavior can change within a short period of time
- Understanding how LLMs evolve is important for maintaining consistent performance levels across different tasks

## Summary

In their study titled "How is ChatGPT's behavior changing over time? ", Lingjiao Chen, Matei Zaharia, and James Zou evaluate the performance and behavior of GPT-3.5 and GPT-4, two widely used large language model (LLM) services. The researchers focus on the March 2023 and June 2023 versions of these models across four diverse tasks: solving math problems, answering sensitive/dangerous questions, generating code, and visual reasoning. The findings reveal that both GPT-3.5 and GPT-4 exhibit significant variations in performance and behavior over time. For instance, GPT-4 (March 2023) demonstrates a high accuracy of 97.6% in identifying prime numbers but experiences a drastic decline to only 2.4% accuracy in June 2023 for the same task. Interestingly, GPT-3.5 (June 2023) outperforms its March 2023 version in this particular task. Furthermore, the researchers observe that GPT-4 becomes less willing to answer sensitive questions in June compared to March. Both GPT-4 and GPT-3.5 also display more formatting mistakes in code generation during June as opposed to March. These findings highlight the need for continuous monitoring of LLM quality as they indicate that the behavior of these models can change substantially within a relatively short period of time. By evaluating different versions of LLMs on various tasks, researchers can gain insights into their performance variations over time and ensure their reliability for specific applications. This study emphasizes the importance of understanding how LLMs evolve and updating them transparently to maintain consistent performance levels across different tasks while ensuring reliable results for specific applications through continuous monitoring of LLM quality over time.