ChatGPT’s behavior appears to change over time, leading to a decline in the quality of some of its outputs.
TL;DR#
Methodology used: they evaluate the March 2023 and June 2023 versions of ChatGPT on 4 diverse tasks:
- solving math problems
- answering sensitive/dangerous questions
- generating code
- visual reasoning
They also analyze the behavior of ChatGPT in response to different prompts and inputs.
Key results: the performance and behavior of ChatGPT can vary greatly over time. For example, ChatGPT’s performance on math problems improved significantly between March and June 2023. However, its behavior is becoming more unpredictable as it is exposed to more data.
Original paper from Lingjiao Chen, Matei Zaharia, and James Zou (Stanford / UC Berkeley).