Best crawler plugins for WordPress

Email Address Encoder

Down in the last 7 days:2#406

by Till Krüss7 months ago

A lightweight plugin that protects email addresses from email-harvesting robots, by encoding them into decimal and hexadecimal entities.

Robots.txt Editor

Up in the last 7 days:1#1,916

by Processby5 years ago

Robots.txt for WordPress

No-Bot Registration

Down in the last 7 days:7#4,904

by Arnan de Gans10 months ago

Prevent bots from creating accounts by blacklisting domains and usernames and present people with a human friendly security question.

Robots.txt rewrite

Down in the last 7 days:16#5,837

by Eugen Bobrowski9 years ago

Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.

Marquee image crawler

Down in the last 7 days:41#8,320

by gopiplus3 years ago

Marquee image crawler is a continuous scrolling image plugin. This plugin crawls the images left or right.

Block Archive.org via WordPress robots.txt

Up in the last 7 days:7#9,422

by apasionados9 months ago

Blocks the archive.org bots through the WordPress virtual robots.txt file.

Bisteinoff SEO Robots.txt

Down in the last 7 days:55#9,661

by Denis Bisteinov10 months ago

An easy-to-use plugin that generates and configures a proper robots.txt file, essential for effective search engine optimization (SEO).

Mass Ping Tool for SEO

Down in the last 7 days:45#9,760

by Pagup29 days ago

Mass ping tool for SEO plugin is one of the fastest ways to get indexed by Google. After pinging, Google bots, crawlers will visit your website and in …

Block eCommerce Assets via robots.txt

Up in the last 7 days:2#16,789

by apasionados3 years ago

Blocks some WooCommerce assets and Search pages that should not be indexed through robots.txt

Block Chat GPT via robots.txt

Down in the last 7 days:250#16,893

by apasionados3 years ago

Blocks the ChatGPT-User bot that is used by plugins in ChatGPT to crawl websites through the WordPress virtual robots.txt file

bodi0`s Bots visits counter

Down in the last 7 days:216#17,133

by Budiony Damyanov2 years ago

Counts the visits from web spiders, crawlers and bots in your blog, with ability to get the blog rankings.

Warmer

Up in the last 7 days:721#17,197

by Atakan Oz5 months ago

Warm your WordPress cache with scheduled crawling, cache preloading, URL source control, logs, diagnostics, and WP-CLI.

Promium AI GEO

Up in the last 7 days:15,891#17,867

by PROMIUM1 month ago

Write a dedicated version of any post or page for AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, and choose which bots may read it.

Findkit

Down in the last 7 days:17#18,204

by Findkit Team2 months ago

Powerful cloud based search engine for your website. Search as you type, real-time index updates, and embeddable UI. Free for small websites.

HS3Dev AI Content Index for llms.txt

#19,813

by mirzahadi11 days ago

Generate llms.txt and llms-full.txt with full control over content, SEO integration, multilingual support, and AI crawler analytics.

Crawl Hunt

#21,651

by themeshopy6 days ago

AI-powered SEO audits and fixes for WordPress. Find SEO issues, fix them faster, and optimize your site for search engines and AI search.

Berq Markdown

#25,475

by BerqWP4 months ago

Increase your AI visibility. Berq Markdown serves every page as a clean Markdown file so AI assistants and LLM crawlers can read, cite, and recommend …

Azayem Bots Tracker

Down in the last 7 days:12#26,713

by botstracker10 months ago

A custom plugin to log search engine bot visits and analyze their crawl budget directly inside WordPress (no external services required).

Block Feed and Comments via robots.txt

Down in the last 7 days:90#26,887

by apasionados3 years ago

Blocks some Feed and Comment resources in order to avoid spam bots

Block Common Crawl via robots.txt

#27,849

by apasionados3 years ago

Blocks the Common Crawl bot (CCBot) that gathers data that is used to train AI algorithms like ChatGPT through the WordPress virtual robots.txt file

Boring Cache Warmup Lite

Up in the last 7 days:194#29,163

by chriskubisch2 months ago

A lightweight, privacy-friendly WordPress cache warmup tool. No external services, no drama.

Trackbase Analytics

#31,616

by trackbase25 days ago

Official Trackbase Analytics plugin for WordPress. Add your tracking ID and enable server-side crawler analytics in minutes.

GlobeSign Content Markdown Mirror

#32,259

by globesign4 months ago

Makes your WordPress site AI-friendly: generates llms.txt and clean, structured Markdown mirrors of every page and post.

GoBookMart Robots Editor

Up in the last 7 days:71#32,815

by Gobookmart11 months ago

A lightweight plugin to edit or create robots.txt directly from your WordPress dashboard. Developed by GoBookMart.

Rankal LLMs.txt

#33,967

by rankalseo2 months ago

Generate and serve a dynamic, cached /llms.txt AI context index. An add-on that works in partnership with Rankal SEO Optimizer.

Trimardeau SEO Log

Up in the last 7 days:5,061#38,525

by Nicolas TRIMARDEAU2 months ago

SEO & GEO bot analytics: see Googlebot, ChatGPT, Claude and every AI crawler at work on each post, page and term — full crawl history included.

JaspreetChahal’s wordpress bot detector lite

Up in the last 7 days:5,745#40,628

by Jaspreet Chahal11 years ago

This plugin detects a few bots (e.g. Google, Google ads, Alta vista etc) and sends an email based on interval set by you. you can choose what format e …

GoVISIBLE AI Bot Tracker

#45,495

by GoVISIBLE1 month ago

Track visits from AI bots like ChatGPT, GPTBot, Claude, Gemini, and Perplexity.

Robots.txt Extender

Up in the last 7 days:1,959#46,350

by bettoadami6 years ago

Dynamic robots.txt for Multisite! Change parameters, or don't, for each site of your network without losing the defaults from WordPress Includes.

Page Modified

Down in the last 7 days:8,561#47,288

by Chris Klosowski9 years ago

Display the results of your Page Modified crawls right within your WordPress Admin.

SEObot Monitor for Googlebot, Bingbot and search engine spiders

Down in the last 7 days:1,928#48,469

by Santiago Alonso5 years ago

With SEObot Monitor for Googlebot you will be able to dump the server logs to Google Analytics, easily and automatically. This will allow you to dete …

Okibako AI

Up in the last 7 days:5,665#50,110

by hokkaidoweb14 days ago

Records which AI crawlers actually fetch your pages, checks what they can read, and serves llms.txt — no database tables, no IPs stored.

Spuncin

Down in the last 7 days:57#50,404

by spuncin2 months ago

Blocks AI crawlers that collect your content to train models. Search engines and link previews are never touched.

Aiso AI Crawler Analytics

Down in the last 7 days:254#50,430

by bentannenbaum4 months ago

See which AI assistants — ChatGPT, Claude, Perplexity, Google and more — crawl your site, broken down by assistant, crawler, purpose and page.

BotGate

Up in the last 7 days:10,336#50,803

by Anil12 days ago

Choose which AI bots can read your content, and publish an llms.txt file for AI search tools.

LLMReader AI Bots

#53,006

by LLMReader11 days ago

See which pages AI crawlers read and how many people arrive from AI chat apps. Reports to your LLMReader account.

SeoNova Bot Radar

#53,106

by edison721 day ago

Bot traffic radar: who visits your site, what they are after, and what they cost you in money.

Bottoll

Up in the last 7 days:17,723#53,135

by bottoll1 month ago

Detect AI crawlers, apply access policies, and understand automated access to your WordPress content.

Zicy AI Access Monitor

#53,141

by zicygp5 months ago

Monitor known AI crawlers and AI assistant fetches, with daily rollups, CSV exports, weekly summaries, and page briefs.

OpenBotAuth

Up in the last 7 days:8,881#53,695

by hammadtq9 months ago

Verify AI crawlers with HTTP Signatures and enforce allow/deny/teaser policies; also serves llms.txt and AI-ready feeds.

Imedes AI Bot Verify

Up in the last 7 days:9,129#54,690

by imedes2 months ago

Verify supported AI bot identities, allow legitimate agents, and block spoofed claims.

AI Kansoku Lab Tracker

Down in the last 7 days:3,779#55,421

by AI Kansoku Lab5 days ago

Track AI crawler visits such as GPTBot, ClaudeBot, and PerplexityBot server-side and visualize them in WordPress.

WEO LLMS & AI Crawler Guide

Up in the last 7 days:9,017#57,666

by Md Atiar Rahman Ovi10 months ago

The professional architect for your llms.txt file. Control how AI agents (ChatGPT, Claude, Perplexity) consume your website content with a structured, …

Cleotic AI Traffic

Up in the last 7 days:7,786#58,393

by cleotic4 months ago

See AI crawler visits and AI assistant referrals on your site, with optional signed sync to Cleotic.

Virtual LLMS.TXT Generator

Up in the last 7 days:12,091#58,809

by Alex Oreshkin1 month ago

Publish a safe, configurable llms.txt map from public WordPress content, with an opt-in bounded full-text export.

Wizehawk Store Discovery Audit

Up in the last 7 days:9,587#60,490

by Wizehawk1 month ago

Local-first WooCommerce readiness checks for AI search discovery, crawler access, and structured product information.

Limpark Bot Monitor

Down in the last 7 days:8,433#62,456

by limpark2 months ago

Monitor search engine bots visiting your WordPress website in real time.

BotObservatory

Down in the last 7 days:15,418#65,205

by traveliaq2 months ago

Observe human and machine traffic, identify suspicious automation, and optionally quarantine aggressive scrapers.

Valvai

#65,379

by valvai1 year ago

Block, log and monetise AI crawlers directly from your WordPress dashboard.

Whizz MD

Up in the last 7 days:620#65,437

by Whizz People4 months ago

Serve every WordPress post and page as a clean .md file with full SEO frontmatter — discoverable by AI crawlers and search engines on the fly.