ChatGPT gives flawed answers to programmers

Share post:

A study from Purdue University has found that ChatGPT, a large language model chatbot developed by OpenAI, answered only 48% of programming questions correctly. The study also found that ChatGPT’s answers were often verbose and incorrect, but that many programmers still preferred its answers due to its pleasant, confident, and positive tone.

The study’s authors, Samia Kabir, David Udo-Imeh, Bonan Kou, and assistant professor Tianyi Zhang, say that ChatGPT’s incorrect answers were often due to its inability to understand the underlying context of the question being asked. They also say that ChatGPT’s verbose answers can make it difficult for programmers to identify errors.

The investigation encompassed posing 517 technical queries from Stack Overflow to ChatGPT, in addition to seeking responses from a select group of twelve volunteers. The evaluative metrics extended beyond mere correctness, encompassing factors like consistency, clarity, and brevity.

It also found that correct responses accounted for a modest 48%, nearly 40% of participants favored ChatGPT’s answers, attributing this preference to its comprehensive and eloquent language. Also, when ChatGPT erred outright, a 2 out of 12 participants still favored its responses.

The sources for this piece include an article in TechSpot.

Featured Tech Jobs


Related articles

Google Bard chats indexed in Google search

Users of Google Bard are reporting that their conversations with Bard are being indexed and appearing in Google...

Jony Ive and OpenAI working on AI hardware partnership

Jony Ive, the former design lead at Apple, is reportedly in talks with OpenAI CEO Sam Altman about...

AI models show promise in hurricane forecasting

New AI-based weather models are showing promise in hurricane forecasting. These AI weather models work by learning patterns...

Spotify launches AI voice translator for podcasts

Spotify is set to take podcasting to a new level with its new AI voice translation feature which...

Become a member

New, Relevant Tech Stories. Our article selection is done by industry professionals. Our writers summarize them to give you the key takeaways