New SlopShape detector identifies AI-generated web content from structure alone, hits 98% accuracy

A new paper called SlopShape reports that AI-generated commercial web content can be reliably identified using structural patterns alone, without relying on word-level style cues, achieving 98.0% macro-F1 on held-out companies.

What's new

The paper, "SlopShape: Identifying AI-Generated Commercial Web Content," by Jochen Madler of Sitefire, replicates an earlier method called StoryScope but applies it to commercial web content specifically: 2,250 human-written blog posts compared against 11,250 AI-generated posts produced by five different models. Rather than scoring individual words or phrases, the detector applies a 214-feature instrument that captures the structural shape of a piece of content — how information is organized and presented — rather than its vocabulary.

The headline results:

• 98.0% macro-F1 on companies held out of the training set
• 98.1% accuracy retained even after the AI-generated posts were reworded
• 79.3% of AI posts correctly attributed to the specific model that generated them, against a 16.7% chance baseline across five candidate models

The paper's framing for why structure works where word-level detectors struggle: "A word-level score neither characterizes a text nor identifies which AI model wrote it." The authors argue the structural signal does both at once: "AI posts share a tidy, self-announcing shape, 79.3% are attributed to the correct source against a 16.7% chance rate."

Context

Most existing AI-text detectors work by scoring vocabulary, sentence-level statistics, or perplexity against a language model — approaches that are comparatively easy to defeat by rewording or paraphrasing AI output. SlopShape's central claim is that reworded AI content still keeps a detectable shape (headers, structure, information ordering) even when the wording changes, which is why its accuracy barely drops after paraphrasing.

Why it matters

As commercial web content increasingly mixes human and AI authorship, structural detectors like this one give platforms, search engines, and content moderators a harder-to-evade signal than wording-based checks — and the model-attribution result (correctly identifying which of five models wrote a given post nearly 80% of the time) points toward provenance tracking that survives editing, not just binary human-vs-AI classification.