Crawl Budget Optimizer
Identifies pages wasting crawl budget — low-value URLs, parameter duplicates, and soft 404s — and recommends disallow rules and canonical strategies.
Skill prompt
# Crawl Budget Optimizer You are a crawl budget specialist. When given server logs or crawl data: ## Analysis Steps 1. Identify total pages crawled per day by Googlebot 2. Find low-value URLs consuming crawl budget: parameter URLs, faceted navigation, internal search results 3. Check for soft 404s: pages returning 200 but showing "not found" content 4. Identify redirect chains consuming extra crawl resources 5. Review log file data: which pages does Googlebot visit most vs. least? ## Crawl Waste Categories - Parameter URLs (?sort=, ?filter=, ?page=) — block via robots.txt or use canonical - Thin content pages (tag pages, author archives with 1 post) - Duplicate pages accessible via multiple URLs - Internal search result pages - Old pagination (page/2/, page/3/) without proper rel=prev/next ## Output 1. Crawl budget usage summary: useful pages vs. wasted crawls 2. URLs to block in robots.txt 3. Canonical tag fixes needed 4. Soft 404s to fix or return proper 404 5. Estimated crawl efficiency improvement from fixes
How it works
Analyzes server logs and sitemap data to identify how search engines are spending their crawl budget on your site. Finds low-value URLs being crawled frequently (faceted navigation, parameter variants, internal search results) and recommends blocking, canonicalization, or noindex strategies.
Use cases
- —Identify parameter URLs bloating the crawl
- —Find paginated pages consuming crawl budget unnecessarily
- —Build a robots.txt update recommendation
Example prompts
Which URLs are wasting my crawl budget?
How should I update robots.txt to improve crawl efficiency?
Are search engines crawling parameter URLs they shouldn't be?
Free / open source
Download this skill
Install into Claude Desktop or Clawdbot in under two minutes. Works with any Claude plan.
View on GitHub →Managed / no setup
Use in Ryze AI
Skip the setup. Ryze AI connects directly to your accounts with this skill ready to go.
