ChatGPT gives flawed answers to programmers

Share post:

A study from Purdue University has found that ChatGPT, a large language model chatbot developed by OpenAI, answered only 48% of programming questions correctly. The study also found that ChatGPT’s answers were often verbose and incorrect, but that many programmers still preferred its answers due to its pleasant, confident, and positive tone.

The study’s authors, Samia Kabir, David Udo-Imeh, Bonan Kou, and assistant professor Tianyi Zhang, say that ChatGPT’s incorrect answers were often due to its inability to understand the underlying context of the question being asked. They also say that ChatGPT’s verbose answers can make it difficult for programmers to identify errors.

The investigation encompassed posing 517 technical queries from Stack Overflow to ChatGPT, in addition to seeking responses from a select group of twelve volunteers. The evaluative metrics extended beyond mere correctness, encompassing factors like consistency, clarity, and brevity.

It also found that correct responses accounted for a modest 48%, nearly 40% of participants favored ChatGPT’s answers, attributing this preference to its comprehensive and eloquent language. Also, when ChatGPT erred outright, a 2 out of 12 participants still favored its responses.

The sources for this piece include an article in TechSpot.

Featured Tech Jobs


Related articles

AI surpasses human benchmarks in most areas: Stanford report

Stanford University’s Institute for Human-Centered Artificial Intelligence (HAI) has published the seventh annual issue of its AI Index...

Microsoft and OpenAI partner to build a $100 Billion AI supercomputer “Stargate”

In a bold stride towards computational supremacy, Microsoft, in partnership with OpenAI, is reported to be laying the...

US Bill Aims to Unveil AI Training Data Sources Amid Copyright Concerns

In a significant move toward transparency, a bill was introduced in the US Congress on Tuesday by California...

AI presents an “extinction level threat” – US Gov’t Report: Hashtag Trending for Tuesday, March 12, 2024

A new US government report warns that AI presents an “extinction level threat to the human species. Elon Musk is outsourcing his Grok AI code. Hackers have breached the Cybersecurity and Infrastructure Security Agency in the US and a researcher shows how to steal a Tesla by leveraging a feature of the Tesla charging stations.

Become a member

New, Relevant Tech Stories. Our article selection is done by industry professionals. Our writers summarize them to give you the key takeaways