Best crawlers plugins for WordPress

Block AI Crawlers

Up in the last 7 days:66#4,875

by lastsplash (a11n)7 days ago

Adds robots.txt rules blocking AI crawlers and AI bots.

Robots.txt rewrite

Down in the last 7 days:13#5,833

by Eugen Bobrowski9 years ago

Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.

AI Content Signals

Up in the last 7 days:33#7,348

by Fernando Tellado2 months ago

Control how AI uses your content — search, AI answers or training — via robots.txt Content Signals, HTTP headers, meta tags and TDMRep.

CodingBunny LLMs.txt Generator

Down in the last 7 days:117#14,811

by CodingBunny1 month ago

Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.

Robots Meta Whiz

Up in the last 7 days:41#16,411

by Marat Nepomnyashy9 years ago

Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.

MetaRobots by SEO-Sign

Up in the last 7 days:45#16,501

by Artem Pilipets11 years ago

The easiest way to manage meta robots tag.

Serve Markdown

Up in the last 7 days:93#17,796

by Azey7 months ago

Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.

ParseLess

Down in the last 7 days:189#19,387

by phalkmin26 days ago

Serve AI the good version of your content, see which bots really read it, and cap what they cost you. Markdown for crawlers and agents — 95%+ less ban …

Log-Hero.com Plugin

Up in the last 7 days:596#21,589

by Log Hero6 days ago

See which search engines, AI crawlers and other bots visit your site, and what they find. The official plugin for log-hero.com.

CeeWP Robots.txt Manager

Down in the last 7 days:352#21,899

by CeeWP1 year ago

Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!

Temso AI

#25,957

by Temso AI2 months ago

Stream front-end request logs from your WordPress origin to Temso so AI-crawler and bot traffic shows up in your dashboard.

Seo Crawlytics Data Export & Delete

#29,675

by apasionados3 years ago

Exports all entries of Seo Crawlytics and gives you the option to delete the entries that are older than 1 month.

MSG LLMs TXT Manager

Down in the last 7 days:376#31,517

by Hugo1 month ago

Serve your content as clean Markdown to AI agents, publish llms.txt and llms-full.txt, and see which agents actually fetch what.

LJPL Armored robots.txt

Down in the last 7 days:354#31,851

by Lukasz Jasinski13 years ago

Add some directives to your robots.txt file to keep your site safer

OtterlyAI Agent Analytics

Up in the last 7 days:11,372#35,217

by OtterlyAI1 month ago

Reports every AI crawler and bot request on your site to OtterlyAI Agent Analytics. Runs server-side, in the background.

PufferSights

Down in the last 7 days:3,710#45,014

by senols1 month ago

Monitor 135+ known AI crawlers and agents with local analytics, AI referrals, llms.txt, and opt-in robots.txt policy tools.

Rawi MD

Down in the last 7 days:1,292#48,306

by Ahmed Abdelhamed2 months ago

Give ChatGPT, Claude and other AI crawlers a clean Markdown copy of your posts, pages and products. Nothing changes for your visitors.

Misha Manko AI Bot Access Checker

#50,211

by mishamanko1 month ago

See which AI bots can reach your site - ChatGPT, Claude, Perplexity, Google AI and more - and why the others are blocked.

Trakkr Crawler Tracking

Up in the last 7 days:20,419#51,017

by trakkr2 days ago

Record requests from known AI crawlers and view them in Trakkr. Optional AI Pages, off by default, can serve those crawlers through Trakkr.

Presswell Content Extractor

#53,517

by Presswell5 days ago

Serve AI-agent-friendly Markdown and an llms.txt index from your WordPress content.

PropertyRightsWeb™

Up in the last 7 days:17,729#55,415

by personaldigitalspaces19 days ago

License your WordPress content to AI companies: set your terms, publish them in a machine-readable standard, and get paid when your work is used.

AI Mention Visibility

Up in the last 7 days:6,535#56,618

by aimention2 months ago

See which AI crawlers (GPTBot, ClaudeBot, Perplexity and more) reach your site, and the visitors AI sends you. Verified, and free.

Botkibble

#57,206

by gregrandall8 months ago

Serves every published post and page as Markdown for AI agents and crawlers. No configuration, no API keys. Activate and it works.

Seenby AI Access Check

Up in the last 7 days:4,912#58,265

by citedby9 days ago

One button in wp-admin shows whether AI-search crawlers can read your site: robots.txt, llms.txt, sitemap and a free page check.

Referenced

#59,184

by Referenced AI4 days ago

See which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) read which pages of your site. No visitor data, no front-end changes.

Vinvin AI Visibility Tools

#59,928

by vinvin274 days ago

See what AI assistants can read on your site, and decide which AI crawlers may use it: a readiness checklist and purpose-based crawler controls.

StudioMeyer Agent Readiness

#61,392

by studiomeyerio2 months ago

Check what AI agents really get from your site, then publish the machine-readable files it is missing. Local only, no account, no cloud.

SenseGap

#62,397

by sensegap3 days ago

See which AI bots read your site: what they read, how often, and which ones are real.

PinMeOut

Up in the last 7 days:7,945#62,612

by tothiq1 month ago

Control which AI crawlers may read your site, see which ones actually do, and make your pages worth citing. Works alongside your SEO plugin.

AI Rights Protection by SimpleFeed

Up in the last 7 days:6,985#65,010

by Alex Khomichenko1 year ago

Protect your rights by reviewing and blocking AI bots from using your content without compensation.

Posterna

#65,584

by posterna4 days ago

See which AI agents and crawlers visit your site and whether they are who they say they are. Monitor only: it labels and counts, never blocks.

Clownsteins Rogue Agent Defender

#67,667

by Albert1 month ago

Protects the site from external AI attacks by blocking inference providers and selected crawlers.

Block old browser versions and suspicious browsers

Down in the last 7 days:1,891#67,703

by hupe1322 days ago

With the help of WhatIsMyBrowser the plugin detects old and bad browsers and denies them access. A special robots.txt denies crawling by bad bots.

CLEO by RegenAI

#68,923

by CLEO by RegenAI5 days ago

See which AI systems read your pages. Requires a free or paid Cleo account.

Yozigo GEO

#70,713

by yozigo1 month ago

Tracks visits and sends them to your Yozigo GEO dashboard for bot/AI-crawler and traffic-share analytics.

Frenemy

Down in the last 7 days:21,168#72,321

by frenemydev5 days ago

See every bot that reaches WordPress — search crawlers, AI agents, scrapers, impostors — classified and identity-verified, in one dashboard.

getReport

#73,278

by mariosmode13 days ago

See which AI crawlers visit (verified), which AI assistants send you visitors, and how quotable every post is. Plus one-click fixes with Undo. Free.

Supertab Connect

Down in the last 7 days:15,416#73,714

by getsupertab4 days ago

Connect your WordPress site to the Supertab Connect platform for RSL license serving and enabling crawler authentication protocol.

Arbling Beacon

#73,847

by arbling17 days ago

See which AI assistants send visitors to your site, and which AI crawlers read it, in your Arbling dashboard.