TIL: chatGPT is getting worse!

Tags
TIL
Published
July 20, 2023

ChatGPT's behavior appears to change over time, leading to a decline in the quality of its outputs.

TL;DR

👉 Methodology used: 

They evaluate the March 2023 and June 2023 versions of ChatGPT on 4 diverse tasks:

  1. solving math problems,
  2. answering sensitive/dangerous questions,
  3. generating code
  4. visual reasoning.

They also analyze the behavior of ChatGPT in response to different prompts and inputs.

👉 Key Results: 

They find out that the performance and behavior of ChatGPT can vary greatly over time. For example, ChatGPT's performance on math problems improved significantly between March and June 2023.

However, ChatGPT's behavior is becoming more unpredictable as it is exposed to more data.

Original paper from Lingjiao Chen† , Matei Zaharia‡ , James Zou†

(Stanford / UC Berkeley)