OpenAI unveils web crawler

Share post:

OpenAI has announced plans to use its web crawler, GPTBot, to assist publishers and site owners in controlling their content.

“Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII) or have text that violates our policies,” the company said in a post on its website.

Publishers have the option to prohibit content scraping. The technical document describes how to identify GPTBot in the HTTP request header using its user agent token and string. Web publishers can also affect GPTBot’s behavior proactively by including an entry in their web server’s robots.txt file.

This update gives the crawler explicit instructions, assuring conformity with the preferences of the publishers. A collection of robots.txt key/value pairs, for example, can limit GPTBot’s access to a certain portion of a website.

OpenAI asserts that permitting its bot to collect site data can enhance the caliber of AI models, without compromising sensitive information or raising legal concerns. The company emphasizes that the crawled web pages, filtered to exclude paywall-protected sources and personally identifiable information, serve to refine future AI models. They are called “web crawlers” because crawling is the term for automatically accessing a website and obtaining data using software.

Enabling GPTBot’s access to a site not only contributes to the refinement of AI models but also bolsters their overall efficacy and safety. By doing so, publishers assist OpenAI in advancing its models while alleviating the burden of development costs.

The sources for this piece include an article in TheRegister.


Related articles

Apple stock soars after announcement of AI offerings

Apple shares surged to a new record, closing with a 7.3% increase, driven by positive reactions from Wall...

Forget Recall, Windows is already tracking everything you do

Microsoft has been dealing with a PR nightmare since the announcement of Copilot+ PCs, particularly due to concerns...

Is Oracle killing off MySQL?

Yesterday we covered a story about how Oracle was now cracking down on licensing Java, which started as...

Lynn Conway, revolutionized semi-conductor design: Passes away at 85

Lynn Conway, a trailblazing engineer and influential figure in chip design, passed away at the age of 86....

Become a member

New, Relevant Tech Stories. Our article selection is done by industry professionals. Our writers summarize them to give you the key takeaways