Diffbot is operated by the company of the same name. Its crawler extracts structured data from web pages to build a large knowledge graph of entities and facts, which it licenses to AI, sales intelligence, and data analytics customers.
The consequence of blocking is that your organization, products, and pages stop being represented in Diffbot's graph, so any AI or data tool that relies on that graph loses your information. Businesses that want accurate representation across those downstream tools keep Diffbot allowed.
How to check and control it
To check, look in your robots.txt for a rule targeting this user-agent. To allow or block it:
User-agent: Diffbot
Disallow: / # blocks it — remove or set "Allow: /" to permitFor an instant check across your whole site, use SEO AEO Specialist's free AI crawler checker, and see the llms.txt & AI crawler guide.
FAQ
What is Diffbot's knowledge graph?
Diffbot's knowledge graph is a large structured database of entities, such as companies, people, and products, extracted from web pages. Many AI, sales, and analytics products query it. Blocking Diffbot removes your content from that graph, so those downstream tools no longer include your information.
Who uses data from Diffbot?
Diffbot licenses its knowledge graph to a range of customers building AI applications, sales intelligence platforms, and data analytics tools. When you block Diffbot, your pages are excluded from the graph, which in turn affects every product that draws on it to describe your organization.
See where you stand
SEO AEO Specialist runs a free AI-visibility audit and hands you the exact fixes. One-off report €9, unlimited €19/mo.
Run your free audit →