AVAILABLE FOR WEB, FLUTTER & QA PROJECTS
How to use this tool
1. Specify WordPress Site URL
Enter your domain URL (e.g. `https://yoursite.com`).
2. Configure Core WordPress Paths
Enable standard protections for `/wp-admin/` while keeping `/wp-admin/admin-ajax.php` allowed.
3. Block Query & Trackback Bloat
Optionally block faceted internal search (`/?s=`, `/*?*`) and trackback spam endpoints.
4. Copy robots.txt File
Copy the generated file and upload it to your web server root or inject it via an SEO plugin.
Formula or logic used
WordPress Crawler Rules Architecture
WordPress requires specific robots.txt exceptions so Googlebot can render themes and scripts without crawling restricted backend dashboards.
`Disallow: /wp-admin/`: Blocks private admin login and dashboard directories.`Allow: /wp-admin/admin-ajax.php`: Critical exception required for frontend AJAX components to load.`Disallow: /wp-includes/`: Blocks core PHP files while allowing CSS/JS scripts to render.`Disallow: /*?s=*`: Prevents Googlebot from indexing low-quality internal search result pages.`Sitemap: https://yoursite.com/sitemap_index.xml`: Explicitly points crawlers to Yoast, RankMath, or core sitemaps.
Examples
Example 1: Standard Secure WordPress Site
Standard production configuration preventing crawl waste while guaranteeing full visual rendering.
Example 2: WooCommerce Store Protection
Stops search bots from indexing private customer cart sessions.
Common use cases
Fresh WordPress Installation Setup
Deploy an SEO-optimized robots.txt replacing the minimal default virtual file.
WooCommerce Crawl Budget Conservation
Block cart, checkout, and account pages from consuming valuable bot requests.
Fixing 'Blocked by robots.txt' GSC Errors
Ensure theme CSS, fonts, and JavaScript assets are not accidentally blocked.
Internal Search URL De-indexing
Stop internal search spam links from flooding Google Search Console index reports.
Related tools
View All Tools Directory →WordPress Slug Generator
The WordPress Slug Generator converts headlines, article titles, and product names into clean, URL-safe WordPr...
WordPress Image Alt Text Generator
The WordPress Image Alt Text Generator crafts descriptive, accessible, and SEO-optimized alternative text (`al...
Robots.txt Generator
The Robots.txt Generator produces compliant, standards-based robots.txt directives for all major search engine...
Frequently asked questions
Why must /wp-admin/admin-ajax.php be allowed in robots.txt?
Many modern WordPress themes and plugins use `admin-ajax.php` to fetch content dynamically on public pages. If Googlebot cannot access this file, it may fail to render the page layout properly, harming mobile and ranking scores.
Where is the physical robots.txt file located in WordPress?
By default, WordPress generates a 'virtual' robots.txt dynamically. If you upload a physical `robots.txt` file to your root directory (e.g. `public_html/robots.txt`), it will override the virtual file.
Should I block /wp-content/ in robots.txt?
NO! Your themes, plugins, images, and CSS stylesheets reside in `/wp-content/`. Blocking this directory prevents search engines from rendering your website, leading to severe ranking drops.
Can I edit robots.txt through Yoast or RankMath?
Yes. Both Yoast SEO and RankMath provide built-in file editors (under Tools) that allow you to modify your robots.txt directly from your WordPress dashboard.
Let's build something
exceptional together
Work directly with Faisal Rafique to architect and deliver high-performance Next.js 15 platforms, 60fps Flutter mobile applications, and enterprise automated QA testing pipelines.