Ever since ChatGPT arrived on the scene in November 2022, generative AI has been quite the talk of the town. It hasn’t been without certain issues as well, including plagiarism and where the data is coming from which ChatGPT, Bard and other tools generate. Publishers especially have been concerned over their data being scraped by AI tools. Google is now giving an option to publishers to control what data can be used by generative AI and other tools. In a blog post, Danielle Romain, VP, Trust, Google said “We’ve also heard from web publishers that they want greater choice and control over how their content is used for emerging generative AI use cases.”
New controls for publishers
Romain said that Google is introducing new control — Google-Extended — that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products. “By using Google-Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time,” she said. In other words, it is up to the publishers to give data to AI models that drive Bard, ChatGPT and others.
Google-Extended is standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs. Google and other companies use crawlers to get data to improve their tools. For instance, Google’s common crawlers are used for building Google’s search indices, perform other product specific crawls, and for analysis.
As per Google, making tools like Google-Extended available through robots.txt is an important step in providing transparency and control “that we believe all providers of AI models should make available.”
Romain said that as AI applications and tools expand, web publishers will face the increasing complexity of managing different uses at scale.