Configure Robots.txt

📘

NOTE

Configure Robots.txt will only appear if the plugin is enabled.

Access from the Admin Console

Location: Site > Optimization > Configure Robots.txt

Overview

Here you can set up your site to prevent robots from indexing the web and direct search results of your AnswerHub site.

In the Content field, enter the Sitemap, Disallow, and User-Agent callouts as follows:

1. Sitemap: Enter the location of your sitemap.xml file.
2. User-Agent: Enter the specific bots to deny access to.

📘

NOTE

If there is more than one, create as many User-Agent callouts as necessary, separated by a double space.

• Under each instance of User-Agent, you will find the word Disallow.

3. Disallow: Enter the directory you want to deny the specified bot access to (for example, /users/, /comments, /search, etc.)

📘

NOTE

If there is more than one, create as many Disallow callouts as necessary with each one on a separate line.

4. Click the SAVE button.
   A successful save will result in a green bar at the top of the page with a message that reads "Robots.txt file successfully saved."