Application Delivery (ADX)

Reply
Occasional Contributor
Posts: 17
Registered: ‎11-10-2010

How can I troubleshoot TCS service faster?

Dear All,

I believe many engineer have the same problem about troubleshooting TCS service. There are a lot of cache server in the world and different setting as well. Cache Engine I know in the world following list below. It's impossible to know setting all of product unless handing-on.

  • BlueCoat/CacheFlow

  • NetApp

  • IronPort

  • iCache

  • SwiftCache

  • ARA Network

  • CacheMARA

  • Squid

  • etc.

Basically, TCS configuration is quite simple and uses only a few commands to show statistic in mpconsole and rconsole likes “show cache-group “ to see statistic like below:

show_cache-group_rconsole.JPG

In my experience, if I see status Client status show Failed (instead of “active”), proxy service is probably stopped or didn’t configure “port forwarding” in iptable (squid). Alternatively, if cache server state is 2 (failed), it means server can be pinged or gone down or blocked by iptable or many reason. By the way, incorrect route-map setting in case of spoofing feature is also classic misconfiguration.

Do you guys have any idea or better step for troubleshooting TCS service or any document for this? Please give me advice or share your idea. I’ll thank you so much

Best Regards,

Tony

Occasional Contributor
Posts: 17
Registered: ‎11-10-2010

Re: How can I troubleshoot TCS service faster?

Since I posted this topic I haven’t found the best way to troubleshoot TCS so far. Let me share how I troubleshoot TCS (even though it's not the best). In the example, I have 2 squid servers (2.6), 1 ServerIron ADX1000 (v12200a) and Cisco Multilayer Switch. All devices are set gateway to ServerIron as In-line deployment (without PBR now).

Note: 10.33.59.12 (Client), 10.33.56.2 (Squid2) and 10.33.58.11 (Webserver)

1) Using “show cache-group” command to display TCS information

telnet@SI-ADX1000#showcache-group

Cache-group 1 has 2 members Admin-status = Enabled Active = 0
Hash_info: Dest_mask = 255.255.255.0 Src_mask = 0.0.0.0

Cache Server Name                Admin-status Hash-distribution  L7-Hash-Buckets
squid2                           6            0                  0          
squid1                           6            0                  0         

HTTP Traffic  From <-> to  Web-Caches

Name: squid2          IP: 10.33.56.2       State: 6   Groups =   1

                                     Host->Web-cache      Web-cache->Host  
           State   CurCon TotCon     Packets   Octets     Packets   Octets 
Web-Server active  0      0          7         3362       8         2007   
Client     active  0      61         11        1895       8         3620
   
Total              0      61         18        5257       16        5627  

Name: squid1          IP: 10.33.56.1       State: 6   Groups =   1

                                     Host->Web-cache      Web-cache->Host  
           State   CurCon TotCon     Packets   Octets     Packets   Octets 
Web-Server active  0      0          147       79811      161       41198  
Client     active  0      69         165       36065      235       238976 
Total              0      69         312       115876     396       280174


HTTP Uncached traffic

If not, traffic won’t be distributed to any server or incomplete flow like example below:

telnet@SI-ADX1000#showcache-group

Cache-group 1 has 1 members Admin-status = Enabled Active = 0
Hash_info: Dest_mask = 255.255.255.0 Src_mask = 0.0.0.0

Cache Server Name                Admin-status Hash-distribution  L7-Hash-Buckets
squid2                           6            0                  0         

HTTP Traffic  From <-> to  Web-Caches

Name: squid2         IP: 10.33.56.2       State: 6   Groups =   1

                                     Host->Web-cache      Web-cache->Host  
           State   CurCon TotCon     Packets   Octets     Packets   Octets 
Web-Server active  0      0          0         0          0         0                    <- No conncetion between Webserver and Cache server (return/redirect)
Client     active  1      168        60        9865       61        3782            <- Only Cache server and Client.
Total              1      168        60        9865       61        3782  


HTTP Uncached traffic

2) Using “tcpdump” command to display packet comes to Squid

Squid:

# tcpdump -i eth0 -n port 80

04:03:57.162592 IP 10.33.59.12.sna-cs > 10.33.58.11.http: P 4277319266:4277319655(389) ack 2324865133 win 63342

04:03:57.163098 IP 10.33.56.2.38128 > 10.33.58.11.http: S 2354134492:2354134492(0) win 5840 <mss 1460,sackOK,timestamp 25976585 0,nop,wscale 6>

04:03:57.163569 IP 10.33.58.11.http > 10.33.56.2.38128: S 2091862811:2091862811(0) ack 2354134493 win 5792 <mss 1460,sackOK,timestamp 13715797 25976585,nop,wscale 6>

3) Using “tail” command to display access-log on squid/webserver

Squid:

# tail -f /var/log/squid/access.log

1293137337.256      5 10.33.59.12 TCP_CLIENT_REFRESH_MISS/403 5378 GET http://10.33.58.11/ - DIRECT/10.33.58.11 text/html

1293137337.294      3 10.33.59.12 TCP_CLIENT_REFRESH_MISS/200 2719 GET http://10.33.58.11/icons/apache_pb.gifDIRECT/10.33.58.11 image/gif

1293137337.304      9 10.33.59.12 TCP_CLIENT_REFRESH_MISS/200 1606 GET http://10.33.58.11/icons/powered_by_rh.png - DIRECT/10.33.58.11 image/png

...

Webserver:

# tail -f /var/log/httpd/access_log

10.33.56.2 - - "GET / HTTP/1.0" 403 5043 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbNDV/5.9.1.14019)"

10.33.56.2 - - "GET /icons/apache_pb.gif HTTP/1.0" 200 2326 "http://10.33.58.11/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbNDV/5.9.1.14019)"

4) summary

  • Without spoofing-support command src-ip address will be cache-ip

  • ServerIron doesn't care what port cache server uses (port 3128 in this example). ServerIron uses only port 80 for L4-health-check.

  • Squid requires 2 configurations -- transparent (squid.conf) and Port redirect (iptables).

Finally, I'm still looking forward to you guys idea.

Note: running-config.txt is attached.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.