An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
But other WordPress blogs have lots of tags included in it.
Also, my XML file’s sitemap name looks like this:
How do I add other possible parameters in my robots.txt file? Is the XML file
sitemap.xml.gz name valid, and what is the difference between a
sitemap.xml.gzis a valid name. The
.gz extension just means that it’s been compressed (using
gzip compression), so that it’s smaller and served faster. Most search engine bots can read gzip’d compressed content.
You can edit the sitemap’s XML content just by decompressing
sitemap.xml.gz using a compression utility like
“Robots.txt” is separate from the sitemap.xml, and can be edited with a text editor (without decompressing it). There are many parameters that you can add to the “robots.txt” file, and depends on what you would like to achieve. For more information on possible parameters, Google “robots.txt”, or check this link: Robots.txt