Can robots.txt protect sensitive content?

No. Use authentication and access controls. Robots.txt is publicly readable and advisory.

Will crawl-delay work for Googlebot?

Google does not support crawl-delay in robots.txt. Support differs between crawlers, so do not rely on it for Googlebot throttling.

Draft crawler rules and sitemap references.

Crawler requests, not access controls. Googlebot ignores crawl-delay. Review the draft before replacing your site’s existing robots.txt.

Presets:

User-agent

Rules

1Rule 1 directiveRule 1 path

Crawl Delay (seconds)

Sitemap URLs

Sitemap URL 1

User-agent: *
Disallow:

SEO Tools Local processing

Last reviewed: 2026-10-04 · Testing methodology

Choose a user-agent, add rules and review the text before placing it on your site.

Select the user agent and enter paths relative to your site. Avoid accidentally blocking resources needed to render public pages.

Use the sitemap served by your site. A sitemap declaration does not guarantee crawling or indexing.

Copy the draft to a review environment first. Check your existing rules and the target crawler’s supported directives.

Inputs are processed in your browser. Analytics and advertising can make separate network requests.

User-agent: * followed by Disallow: /private/ asks compliant crawlers not to request that path; it does not stop a browser from opening it.

Do not replace a working production robots.txt solely because this generator produced a syntactically plausible draft.

Can robots.txt protect sensitive content?: No. Use authentication and access controls. Robots.txt is publicly readable and advisory.
Will crawl-delay work for Googlebot?: Google does not support crawl-delay in robots.txt. Support differs between crawlers, so do not rely on it for Googlebot throttling.