Subtotal | $0.00 |
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~different/
User-agent: * Disallow: /
User-agent: * Disallow:
User-agent: *Example 2
Disallow: /cgi-bin/
User-agent: *
Disallow: /page_url
User-agent: *
Disallow: /page_url$
User-agent: Bot1
Disallow: /
User-agent: Opera 9
Disallow: User-agent: * Disallow: /
User-agent: *
Allow: /docs/file.jpeg
Disallow: /docs/
User-agent: *
Disallow: /docs/
Disallow: /images/
sitemap:http://www.domain.com/sitemap.xml
From an SEO perspective, this is not recommended but it’s up to you to decide. User-agent: *
Disallow: /profiles/
crawl-delay: 10
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /tag/
Disallow: /category/
Disallow: /tags/
Disallow: /category/
Disallow: /2010/
Disallow: /2011/
Disallow: /2012/
Disallow: /2013/
Need help? We're always here for you.