Adding additional HTTP header to the real servers HTTP response.
We will use the ServerIrons ability to rewrite HTTP responses here. We are going to add an X-Robots-Tag in the HTTP header of real server responses.
X-Robots-Tag is part of the Robot Exclusion Protocol. The X-Robots-Tag contains robot instructions and the tag is part of the HTTP header. The X-Robots-Tag can contain various instructions for robots. Examples are:
NO-INDEX: do not show this page in search results
NOFOLLOW: disallow bots to crawl links on this page
NOARCHIVE: disallow prevent archiving
NOSNIPPET: disallow displaying a summary in search result pages
There are some other TAGs - some TAGs are for specific bots only and some others are more general.
We will use CSW to rewrite HTTP response and we are going to insert the following X-Robots-Tag into the HTTP response header:
csw-policy "pol" default forward 1 default rewrite response-insert header"X-Robots-Tag: noindex, nofollow" ! server real rs101 192.168.9.101 port http port http url "HEAD /" port http group-id 1 1 ! ! server virtual vs222 192.168.8.222 port http port http csw-policy "pol" port http csw port http keep-alive bind http rs101 http
Tips / Caveats
NOTE: Ensure keep-alive is enabled for the service you are suing response-rewrite for. Response-rewrite needs it to work properly!