GDPR & Privacy

"It Was Public Anyway" Is Not a Legal Basis: The New Rules for Scraping the Web to Train AI

A practical guide for business owners, marketers and product teams. This article is for general information only and is not legal advice.

By Zirlandia Milkovic · 2026-10-17 · 9 min read

Illustration of a web crawler collecting public web pages for AI training, filtered by privacy safeguards and opt-out signals

Most generative AI models learn from text, images, and audio collected from the open web: blogs, forums, news sites, social media profiles, and public registers. Much of that material is about real, identifiable people.

For years, the informal assumption was simple: if it is publicly accessible, it is fair game. European regulators have now spelled out why that does not hold.

Why this is in the spotlight now

The EDPB's draft guidelines. Adopted on 7 July 2026, draft Guidelines 03/2026 on web scraping in the context of generative AI build on Opinion 28/2024 and apply the GDPR step by step to scraping for AI training. Consultation closes on 30 October 2026.
The ICO's report. On 8 October 2026, the UK Information Commissioner's Office closed its two-year supervision of major foundation model developers. Ten developers — Amazon, Anthropic, Apple, Cohere, DeepSeek, Google, Meta, Microsoft, OpenAI, and Stability AI — made, or committed to make, changes, including clearer transparency and better rights routes.
AI Act transparency. General-purpose AI providers must publish a training-content summary using a mandatory Commission template. According to the Commission, the AI Office can enforce this from 2 August 2026, with fines of up to 3% of worldwide turnover or €15 million, whichever is higher.

"Public" does not mean "free to use"

The GDPR applies to scraping whenever it involves personal data — extraction, cleaning, structuring, and storage, whether identification is direct or indirect.

Consent is "most probably not" workable: scrapers cannot realistically ask everyone first.
Publishing online is not consent: the EDPB states explicitly that the absence of a robots.txt file "does not amount to consent within the meaning of the GDPR."

The ICO agrees: calling data "publicly accessible information" is "unlikely to be sufficient" for transparency.

Who is responsible: scraper, developer, or both?

The EDPB says roles are assessed case by case. If a contractor scrapes to your documented instructions, it may be a processor and you the controller. If you buy a dataset, scraper and developer are each responsible for their own processing. If you jointly set the collection criteria, you are likely joint controllers. The ICO adds that developers must ensure their supply chain, including data brokers, meets transparency duties.

Legitimate interest: the three-step test

The ICO calls legitimate interest (Article 6(1)(f) GDPR) realistically the only lawful basis for training generative AI on scraped personal data. The EDPB says it is the basis private companies usually rely on. Both insist it must be earned:

A real, specific interest. Lawful, clearly articulated, and real and present. Broad interests such as "training our models" are unlikely to be enough, says the ICO, without specification.
Necessity. Narrowing collection criteria, rather than "scraping a wide part of the internet", may be crucial, says the EDPB. Pseudonymised or synthetic data may be less intrusive. The ICO expects an explanation of why licensed datasets are not suitable.
Balancing. The EDPB lists risks such as loss of control, a "chilling effect" on free expression, de-anonymisation of anonymous posters, and deepfakes. It also notes personal data "cannot be easily deleted" from a trained model.

Reasonable expectations depend on what websites signal. Posting on a platform that allows scraping and says so is one thing; posting where robots.txt and CAPTCHA block scrapers, and AI training is disallowed, is another.

The draft shows the line: collecting public voice recordings to sell a voice generation tool, with no safeguards against malicious reuse, "cannot rely" on legitimate interest. A text model trained only on freely accessible content people manifestly made public — respecting copyright opt-outs, with memorisation safeguards, easy rights requests, and a public source list — may generally pass.

Transparency when you cannot tell everyone

Article 14 GDPR requires informing people whose data you obtained indirectly. With millions scraped, individual notices are often impossible.

Article 14(5)(b) allows an exception for impossibility or disproportionate effort, but the EDPB says it "should not be routinely relied upon", and the ICO warns against a "blanket" exemption.

If you rely on it, you must at least publish the information, for example in a website notice: data categories, purposes and legal basis, people's rights, and a precise indication of sources, including the crawler's characteristics. As good practice, list domains and URLs scraped, searchable, with the collection period.

Scale matters: 20 years of forum posts with no direct identifiers may make individual notice disproportionate; fine-tuning on a closed group of 5,000 identifiable people means informing each person individually.

This overlaps with the AI Act template: providers must name crawlers, give the collection period, and list the top 10% of scraped domains (SMEs: top 5% or 1,000 domains, whichever is lower). One source inventory can serve both laws.

Minimisation, accuracy, and sensitive data

Data minimisation does not prohibit large-scale training. It prohibits collecting personal data you do not need — assessed before scraping starts: define precise criteria, consider synthetic data, exclude unnecessary categories (bank, location data), child-focused or structurally sensitive sites, and sites opposing scraping via robots.txt, ai.txt, login walls, or CAPTCHA. During and after collection, strip identifiers, pseudonymise, prefer reliable sources, timestamp collections, and spot-check samples.

Sensitive data: a narrow route, not an exemption

Special categories — health, political opinions, sexual orientation — need an Article 6 basis and an Article 9(2) exception. The EDPB proposes a narrow path for data collected incidentally and residually, despite efforts to avoid it, drawing on GC and Others (C-136/17, 24 September 2019): never intentional, hard to predict, search-engine-like, and with dissemination prevented through filters, deletion on identification, extraction testing, output filters, and monitoring. This is not a general exemption.

The ICO is more sceptical: "manifestly made public" is unlikely to cover scraped sensitive data without verifying intent — a position it says "limits the ability for developers to lawfully train their models using special category data".

Models can also contain personal data extractable from them (ICO; EDPB Opinion 28/2024). Web data can include email signatures, API keys, and passwords, so test for "singling out" and "linkability".

Where the law may be heading

The Commission's Digital Omnibus (November 2025) proposes GDPR amendments on AI training. According to a leaked Council compromise text of 3 September 2026 (document 12535/26), published by noyb, a recital would say processing personal data to develop and deploy AI "may be regarded as" a legitimate interest, unless rights override it, with limited allowance for incidentally collected sensitive data. noyb strongly opposes this.

The file is unsettled: EU ambassadors failed to agree a Council position on 7 October 2026, mainly over Data Act trade secrets; the item was removed from their 11 October agenda, with no new date set (MLex). Parliament had tabled roughly 1,840 amendments in August. As of 12 October 2026, none of this is law. Japan, by contrast, would allow consent-free use for statistics/AI organised as statistics creation, with draft disclosure rules published 1 October 2026, per the source article.

A practical checklist

Map your supply chain and roles; write a specific legitimate interest assessment per data type and stage.
Document necessity: why narrower, licensed, pseudonymised, or synthetic data will not do.
Respect opt-out signals (robots.txt, ai.txt, logins, CAPTCHA, "no AI training"); exclude high-risk sources by default.
Filter before and after collection; plan for sensitive data, prompt deletion, and output monitoring.
Publish a real training data notice with sources, crawler details, periods, categories, and an easy objection route.
Offer rights routes for non-users; test models for memorisation, singling out, and linkability; vet dataset vendors.
Align your AI Act training summary with your privacy notice.
Have your say: EDPB consultation closes 30 October 2026; the ICO's agentic AI call for evidence closes 20 November 2026.

A note for Brazilian readers

Brazil's LGPD (Law 13.709/2018) includes legitimate interest, but not in its Article 11 list for sensitive data. At FGV Direito Rio on 29 April 2026, ANPD director Iagê Miola said ANPD has no definitive position yet on legitimate interest for AI training, though contributors broadly accepted it could be valid with robust balancing and transparency. Brazilian companies serving people in the EU are covered by GDPR Article 3(2).

The bottom line

The open web is not a free-for-all, but it is not off-limits either. Scraping can be lawful when targeted, minimised, transparent, and backed by safeguards you can show work.