Application Delivery (ADX)

How to insert HTTP headers into HTTP responses (http response rewrite)

by on ‎06-08-2009 12:40 AM - edited on ‎10-31-2013 03:33 PM by bcm1 (1,142 Views)

Summary

 

Adding additional HTTP header to the real servers HTTP response.

Specifics

We will use the ServerIrons ability to rewrite HTTP responses here. We are going to add an X-Robots-Tag in the HTTP header of real server responses.

 

X-Robots-Tag is part of the Robot Exclusion Protocol. The X-Robots-Tag contains robot instructions and the tag is part of the HTTP header. The X-Robots-Tag can contain various instructions for robots. Examples are:

 

NO-INDEX: do not show this page in search results

NOFOLLOW: disallow bots to crawl links on this page

NOARCHIVE: disallow prevent archiving

NOSNIPPET: disallow displaying a summary in search result pages

...

 

There are some other TAGs - some TAGs are for specific bots only and some others are more general.

 

We will use CSW to rewrite HTTP response and we are going to insert the following X-Robots-Tag into the HTTP response header:

 

"X-Robots-Tag: noindex, nofollow"

 

Original real server response:

HTTP/1.1 200 OK\r\n
Server: Microsoft-IIS/5.0
\r\n
Content-Location: http://192.168.8.222/index.html\r\n
Date: Tue, 09 Jun 2009 09:25:52 GMT\r\n
Content-Type: text/html\r\n
Accept-Ranges: bytes\r\n
Last-Modified: Thu, 02 Apr 2009 08:36:18 GMT\r\n
ETag: "02d92176eb3c91:8f8"\r\n
Content-Length: 10\r\n
\r\n

Response after processing:

HTTP/1.1 200 OK\r\n
X-Robots-Tag: noindex, nofollow\r\n
Server: Microsoft-IIS/5.0\r\n
Content-Location: http://192.168.8.222/index.html\r\n
Date: Tue, 09 Jun 2009 09:22:52 GMT\r\n
Content-Type: text/html\r\n
Accept-Ranges: bytes\r\n
Last-Modified: Thu, 02 Apr 2009 08:36:18 GMT\r\n
ETag: "02d92176eb3c91:8f8"\r\n
Content-Length: 10\r\n
\r\n

Topology Diagram

none

Sample Code/Configuration

 

csw-policy "pol"
default forward 1
default rewrite response-insert header
"X-Robots-Tag: noindex, nofollow"
!
server real rs101 192.168.9.101
port http
port http url "HEAD /"
port http group-id  1 1
!
!
server virtual vs222 192.168.8.222
port http
port http csw-policy "pol"
port http csw
port http keep-alive
bind http rs101 http

 

Tips /  Caveats

NOTE: Ensure keep-alive is enabled for the service you are suing response-rewrite for. Response-rewrite needs it to work properly!

Further Reading

-

Contributors