Application Delivery (ADX)

Debug filter or how to get packet captures at the ServerIron

by on ‎10-20-2009 04:01 PM (6,713 Views)

Summary

 

Debug filter is an utility available at the ServerIron ADX and as well at older JetCore based platforms. It is a utility that captures packets at the ServerIron itself based on user-defined filters. The captured packets are stored in a capture buffer and it is possible to view the packets on-screen or to transfer them to a TFTP server to have a look at the offline. The ServerIron ADX offers offers the possibility to store captures in PCAP format. This simplifies the work dramatically due to the fact that users are able to use PCAP based tools to work with the packet captures taken at the ServerIron.

 

Using the packet capture utility

 

You have to do the following to use the utility:

 

  • enter utility
  • configure capture buffer
  • specify packet size to capture
  • specify filters
  • apply filters
  • start capturing process
  • stop capturing process
  • view captured packets

 

Enter Utility

 

To enter the debug filter utility do the following:

 

ServerIron> ena
ServerIron# debug filter
ServerIron(debug-filter)#

 

Configuring Capture Buffer

 

You have to specify the size of the capture buffer in kilobytes. The maximum size of the buffer is 1024 kilobytes talking about the ServerIron 4G Series/GT-C/GT-E and the ServerIron 350/450/850. The new ServerIron ADX is able to support up to 256 megabyte as buffer space for debug filter.

 

To set the buffer size do the following:

 

ServerIron(debug-filter)# buffer-size 1024

 

Syntax: buffer-size <kilobytes>

 

To display the buffer size:

 

ServerIron(debug-filter)# show buffer-size
Capture buffer size: 1048567 bytes

 

Specify packet size to capture

 

You can specify the number of bytes from a captured packet that are getting stored in the capture buffer. It is as well possible to store the entire packet.

 

To set the amount of bytes to store do the following:

 

ServerIron(debug-filter)# packet-size 128

 

Syntax: packet-size <bytes>|whole

 

The whole parameter specifies that the entire packet needs to be stored in the capture buffer.

 

To show the currently configured packet size:

 

ServerIron(debug-filter)# show packet-size
Max bytes stored from a filtered pkt: 128

 

Specify filter(s)

 

You specify the packets to store in the capture buffer by configuring one or more filter IDs. A filter ID consists of a set of filters that specify the attributes of packets to be stored in the capture buffer. You can configure up to 16 filter IDs.

 

Within a filter ID, you can specify filters for Layer 2 - 4 information in a packet. In addition, you can set up filters to capture packets that contain a specified pattern within the packet.

 

By default, a filter ID is configured to match any packet. Within a filter ID, all the filters must match a received packet in order for the packet to be captured. The filters not explicitly configured have "don't care" values, which are ignored during the matching process.

 

To specify the filter with ID 1:

 

ServerIron(debug-filter)# specify 1
ServerIron(debug-filter-spec-1)#

 

Syntax: specify <filter-id>

 

You are able to specify filter settings at the filter ID configuration level. It is as well possible to display the current settings.

 

Ethernet filter settings

 

CLI Command
Filter type
mac bcast Ethernet broadcast packets
mac dest <mac-address> Packets with the specified destination MAC address
mac mcast Ethernet multicast packets
mac src <mac-address> Packets with the specified source MAC address
mac type <type-in-hex> Packets of the specified Layer 3 type

 

IP filter settings

 

CLI Command
Filter type
ip bcast IP broadcast packets
ip dest <ip-address> Packets with the specified destination IP address
ip mcast IP multicast packets
ip protocol <protocol-in-hex> Packets with the specified Layer 4 protocol
ip src <ip-address> Packets with the specified source IP address

 

TCP filter settings

 

CLI Command
Filter type
tcp src <port-number> Packets with the specified source TCP port
tcp dest <port-number> Packets with the specified destination TCP port
tcp syn TCP packets with the SYN flag on
tcp reset TCP packets with the RST flag on
tcp fin TCP packets with the FIN flag on
tcp ack TCP packets with the ACK flag on
tcp push TCP packets with the PSH flag on
tcp urgent TCP packets with the URG flag on

 

UDP filter settings

 

CLI Command
Filter type
udp src <port-number> Packets with the specified source UDP port
udp dest <port-number> Packets with the specified destination UDP port

 

Pattern matching filter settings

 

You can set up a filter to capture packets that contain a pattern of a specified length, starting from a given offset from the beginning of the packet. Example:

 

ServerIron(debug-filter-spec-1)# pattern 24 2 1023

 

Syntax: pattern <offset> <length> <pattern-in-hex>

 

The <offset> is the number of bytes from the start of the packet.
The <length> is the length of the pattern in bytes. You can specify between 1 - 32 bytes.
The <pattern-in-hex> is the pattern to match. The length of the pattern must be equal to the number of bytes specified with the <length> parameter.

 

Example:

 

Create a filter to look for every packet from source IP 192.168.8.1 going to destination IP 192.168.8.222 - destination port 80:

 

ServerIron(debug-filter-spec-1)#ip src 192.168.8.1
ServerIron(debug-filter-spec-1)#ip dest 192.168.8.222
ServerIron(debug-filter-spec-1)#tcp dest 80

 

To show the currently applied settings for the filter:

 

ServerIron(debug-filter-spec-1)# show

 

Filter-ID: 1

 

        MAC filters:
                Src  MAC : ANY
                Dest MAC : ANY
                MAC Type : ANY
        IP filters:
                Src  IP  : 192.168.8.1
                Dest IP  : 192.168.8.222
                Protocol : ANY
        TCP filters:
                Src  port: ANY
                Dest port: 80
                Flags    : None
        UDP filters:
                Src  port: ANY
                Dest port: ANY
        HTTP filters:
                Url      : ANY
                Cookie   : ANY
        Pattern filters:
                Pattern  : ANY

 

Syntax: show

 

Use reset at filter ID configuration level to restore the filter IDs default settings (match ALL):

 

ServerIron(debug-filter-spec-1)# reset

 

Syntax: reset

 

Use exit to leave the filter ID configuration level.

 

ServerIron(debug-filter-spec-1)# exit

ServerIron(debug-filter)#

 

Syntax: exit

 

It is as well possible to display the setting for a filter ID at the "debug filter" level. This looks like:

 

ServerIron(debug-filter)#show 1

 

Filter-ID: 1

 

        MAC filters:
                Src  MAC : ANY
                Dest MAC : ANY
                MAC Type : ANY
        IP filters:
                Src  IP  : 192.168.8.1
                Dest IP  : 192.168.8.222
                Protocol : ANY
        TCP filters:
                Src  port: ANY
                Dest port: 80
                Flags    : None
        UDP filters:
                Src  port: ANY
                Dest port: ANY
        HTTP filters:
                Url      : ANY
                Cookie   : ANY
        Pattern filters:
                Pattern  : ANY

 

 

Syntax: show <filter-id>

 

 

Apply filter(s)

 

It is possible to define multiple filters like shown above - a filter is a set of matching criteria to select packets. A filter takes effect when you apply it. A filter ID should be applied globally or on an individual port. You can apply a filter ID so that filters inbound traffic only, outbound traffic only, or both.

 

The following is going to apply filter 1 globally for in- and outbound traffic:

 

ServerIron(debug-filter)# apply 1

 

Syntax: apply <filter-id>

 

You can apply multiple filter IDs and specify an and/or relationship between them. For example, to apply filter IDs 1 and 2, enter the following command. Packets that match the filters in both filter IDs are stored in the capture buffer.

 

ServerIron(debug-filter)# apply "1 and 2"

 

This is useful to create more complex filtering rules. A single filter ID is not able to cover a communication from host A to host B because it is necessary to cover the "flow" from host A to host B and the "flow" from host B to host A.

 

Example:

 

Create a filter logic to look for every packet of the communication in between the client with IP 192.168.8.1 and the virtual server with IP 192.168.8.222 @ port 80:

 

ServerIron(debug-filter)# sp 1

ServerIron(debug-filter-spec-1)#ip src 192.168.8.1
ServerIron(debug-filter-spec-1)#ip dest 192.168.8.222
ServerIron(debug-filter-spec-1)#tcp dest 80

ServerIron(debug-filter-spec-1)#exit

ServerIron(debug-filter)# sp 2

ServerIron(debug-filter-spec-2)#ip dest 192.168.8.1
ServerIron(debug-filter-spec-2)#ip src 192.168.8.222
ServerIron(debug-filter-spec-2)#tcp src 80

ServerIron(debug-filter-spec-2)#exit

ServerIron(debug-filter)# apply "1 or 2"

 

The filter ID 1 is going to hit for all packets from the client to the virtual server and filter ID 2 is going to hit for the return traffic. One of them need to be true to be sure it is part of the communication we are looking for.

 

It is as well possible to create more complex filter expressions as shown below:

 

To apply filter IDs 1, 2, and 3 so that packets must match the filters in 1 and match the filters in either 3 or 4, enter the following command:

 

ServerIron(debug-filter)# apply "(1 and (3 or 4))"

 

To view the currently applied expressions:

 

ServerIron(debug-filter-all-all)# show apply
Filter ID apply expression: ( 1 and ( 3 or 4 ) )

 

Syntax: show apply

 

 

Start Capturing Process

 

You are able to start the capturing process as soon as you have done all the steps mentioned above. Once you start the packet capture utility, filtered packets are stored in the capture buffer and are available for viewing until you restart the utility.

 

To start the packet capture utility, enter the following command:

 

ServerIron(debug-filter)# start

 

Syntax: start

 

The packet capturing process will run until the configured buffer is full or until it is getting stopped manually.

 

Stop Capturing Process

 

You can stop the packet capture utility with the following command:

 

ServerIron(debug-filter)# stop

Number of packets captured: 126

 

Syntax: stop

 

View Captured Packets

 

First of all you have to select one of the places packets are getting captured at. JetCore based devices do offer two locations:

 

1. MP = Management Processor

2. BP = Barrel Processor

 

Use the view command to select the processor you would like to choose:

 

To select BP 2 of the WSM module in slot 1:

 

ServerIron(debug-filter)# view bp 1 2
ServerIron(debug-filter-1-2)#

 

To select the MP:

 

ServerIron(debug-filter)# view mp
ServerIron(debug-filter-mp)#

 


Syntax: view [mp | bp <slot-number> <cpu-number>]

 

You do have multiple options now - you might want to use the summary command to see a summary of all packets captured at the chosen location:

 

Syntax: summary

 

It is as well possible to have a close look at a single packet using the hex-dump or ascii-dump command:

 

Syntax: ascii-dump <packet-number>

Syntax: hex-dump <packet-number>

 

The example below includes some example outputs which might help to understand how to use the commands.

 

"debug filter" example

 

Test network:

 

client (192.168.8.100) ---- port 15 (192.168.8.1) ServerIron ADX (192.168.9.1) port 3 --- (192.168.9.101) real server

 

Virtual server @ ADX: 192.168.8.222 offering HTTP service

 

ADX real and virtual server configuration:

 

server real rs101 192.168.9.101
port http
port http url "HEAD /"
!
!
server virtual vs222 192.168.8.222
port http
bind http rs101 http

 

 

Task: get a packet capture of an HTTP request coming from the client IP 192.168.8.100 to the virtual server 192.168.8.222 including the packets going to the real server 192.168.9.101.

 

The source IP of the client request is 192.168.8.100 and the destination TCP port is 80. The backend traffic is as well using the client IP 192.168.8.100 because we do not have source-nat configured. The real servers service port is 80 as well. The return/reply traffic is going to use 192.168.8.100 as destination IP and it is using the source TCP port 80.

 

Filters to configure:

 

1. from source IP 192.168.8.100 to destination port 80 TCP

2. to destination IP 192.168.8.100 from source port 80 TCP

 

Interesting traffic will hit the the first OR second filter.

 

CLI commands/outputs and comments:

 

Enter debug filter utility:

telnet@ServerIronADX 1000>ena
No password has been assigned yet...
telnet@ServerIronADX 1000#debug filter

Configure buffer-size and packet-size to store in buffer:
telnet@ServerIronADX 1000(debug-filter)#buffer-size 4096
telnet@ServerIronADX 1000(debug-filter)#packet-size whole

Create filter 1 and filter 2:
telnet@ServerIronADX 1000(debug-filter)#sp 1
telnet@ServerIronADX 1000(debug-filter-spec-1)#reset
telnet@ServerIronADX 1000(debug-filter-spec-1)#ip src 192.168.8.100
telnet@ServerIronADX 1000(debug-filter-spec-1)#tcp dest 80
telnet@ServerIronADX 1000(debug-filter-spec-1)#exit
telnet@ServerIronADX 1000(debug-filter)#sp 2
telnet@ServerIronADX 1000(debug-filter-spec-2)#reset
telnet@ServerIronADX 1000(debug-filter-spec-2)#ip dest 192.168.8.100
telnet@ServerIronADX 1000(debug-filter-spec-2)#tcp src 80
telnet@ServerIronADX 1000(debug-filter-spec-2)#exit

Apply filters using OR as operator:
telnet@ServerIronADX 1000(debug-filter)#apply 1or2

Start capturing process:
telnet@ServerIronADX 1000(debug-filter)#start

SENT REQUEST FROM TEST CLIENT 192.168.8.100 NOW

Stop capturing process:
telnet@ServerIronADX 1000(debug-filter)#stop

Check ALL BPs to find the BP responsible for the test client:
telnet@ServerIronADX 1000(debug-filter)#view bp 1 1
telnet@ServerIronADX 1000(debug-filter-1-1)#sum
telnet@ServerIronADX 1000(debug-filter-1-1)#view bp 1 2
telnet@ServerIronADX 1000(debug-filter-1-2)#sum
telnet@ServerIronADX 1000(debug-filter-1-2)#view bp 1 3
telnet@ServerIronADX 1000(debug-filter-1-3)#sum

Nothing so far - check the next and last processor talking about the ADX1016-4:
telnet@ServerIronADX 1000(debug-filter-1-3)#view bp 1 4
telnet@ServerIronADX 1000(debug-filter-1-4)#sum

Number of packets captured: 20

 

pkt:1 IN len:112  TCP :3881 ->80    Seq:4047880146 Ack            SYN
pkt:2 OUTlen:138  TCP :3881 ->80    Seq:4047880146 Ack            SYN
pkt:3 IN len:112  TCP :80   ->3881  Seq:2434401979 Ack:4047880147 SYN ACK
pkt:4 OUTlen:138  TCP :80   ->3881  Seq:2434401979 Ack:4047880147 SYN ACK
pkt:5 IN len:106  TCP :3881 ->80    Seq:4047880147 Ack:2434401980 ACK
pkt:6 OUTlen:132  TCP :3881 ->80    Seq:4047880147 Ack:2434401980 ACK
pkt:7 IN len:152  TCP :3881 ->80    Seq:4047880147 Ack:2434401980 ACK PSH
pkt:8 OUTlen:178  TCP :3881 ->80    Seq:4047880147 Ack:2434401980 ACK PSH
pkt:9 IN len:106  TCP :80   ->3881  Seq:2434401980 Ack:4047880199 ACK
pkt:10 OUTlen:132  TCP :80   ->3881  Seq:2434401980 Ack:4047880199 ACK
pkt:11 IN len:423  TCP :80   ->3881  Seq:2434401980 Ack:4047880199 ACK PSH
pkt:12 OUTlen:449  TCP :80   ->3881  Seq:2434401980 Ack:4047880199 ACK PSH
pkt:13 IN len:106  TCP :3881 ->80    Seq:4047880199 Ack:2434402303 ACK
pkt:14 OUTlen:132  TCP :3881 ->80    Seq:4047880199 Ack:2434402303 ACK
pkt:15 IN len:106  TCP :3881 ->80    Seq:4047880199 Ack:2434402303 ACK FIN
pkt:16 OUTlen:132  TCP :3881 ->80    Seq:4047880199 Ack:2434402303 ACK FIN
pkt:17 IN len:106  TCP :80   ->3881  Seq:2434402303 Ack:4047880200 ACK FIN
pkt:18 OUTlen:132  TCP :80   ->3881  Seq:2434402303 Ack:4047880200 ACK FIN
pkt:19 IN len:106  TCP :3881 ->80    Seq:4047880200 Ack:2434402304 ACK
pkt:20 OUTlen:132  TCP :3881 ->80    Seq:4047880200 Ack:2434402304 ACK

 

The test client hits BP 1 4 and the summary is showing 20 packets related to the test request. Packet #1 is the incoming TCP SYN packet, packet #2 is the same SYN packet but it is behind the ServerIron (going to the real server). Each packet is visibile twice because it exists before and after the ServerIron (before SLB and after SLB).

 

The client HTTP request is part of packet #7 (after the 3-way handshake):

 

telnet@ServerIronADX 1000(debug-filter-1-4)#ascii 7

 

Packet 7 captured at Oct 26 17:00:24 ; Packet size is 152(0x0098) bytes
        In port: 15
        fpga optimized: No

 

Ethernet Version II
        Address: 0024.81f7.2f8f ---> 021b.ed3c.cb60
        Ethernet II Protocol Type: IP
Internet Protocol
        Version(MSB 4 bits): 4
        Header length(LSB 4 bits): 5 (32-bit word)
        Service Type: 0x00
        Total length: 92 (Octets)
        Fragment ID: 7207
        Flags summary: 0x40
                0... .... = Reserved
                .1.. .... = Do not fragment
                ..0. .... = Last fragment
                Fragment offset(LSB 13 bits): 0 (0x00)
        Time to live: 128 seconds/hops
        IP protocol type: TCP (0x06)
        Checksum: 0x4be2
        IP address: 192.168.8.100 ---> 192.168.8.222
        No option
Transmission Control Protocol
        Port 3881 ---> 80
        Sequence Number: 4047880147
        Acknowledgement Number: 2434401980
        Header Length(MSB 4 bits): 5 (32-bit word)
        Reserved(LSB 4 bits): 0
        Code: 0x18
                RES: 0... ....
                CON: .0.. ....
                URG: ..0. ....
                ACK: ...1 ....
                PSH: .... 1...
                RST: .... .0..
                SYN: .... ..0.
                FIN: .... ...0
        Window: 64000
        Checksum: 0xe90f
        Urgent Pointer: 0x0000
Data:
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a | GET / HTTP/1.1..
0010: 48 6f 73 74 3a 20 31 39 32 2e 31 36 38 2e 38 2e | Host: 192.168.8.
0020: 32 32 32 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a | 222..Accept: */*
0030: 0d 0a 0d 0a 00 00 00 00 00 00 00 00 00 00       | ..............

 

The packet arrived via port 15 (client facing) and the IP header is showing the following:

 

source IP: 192.168.8.100

destination IP: 192.168.8.222

 

This is a packet coming from the client to the virtual server. Packet #8 is the same packet BUT on the way to the real server (after SLB):

 

telnet@ServerIronADX 1000(debug-filter-1-4)#ascii 8

 

Packet 8 captured at Oct 26 17:00:24 ; Packet size is 178(0x00b2) bytes
        Out port: 3
        fpga optimized: No

 

Ethernet Version II
        Address: 001b.ed3c.cb62 ---> 021b.ed3c.cb60
        Ethernet II Protocol Type: IP
Internet Protocol
        Version(MSB 4 bits): 4
        Header length(LSB 4 bits): 5 (32-bit word)
        Service Type: 0x00
        Total length: 92 (Octets)
        Fragment ID: 7207
        Flags summary: 0x40
                0... .... = Reserved
                .1.. .... = Do not fragment
                ..0. .... = Last fragment
                Fragment offset(LSB 13 bits): 0 (0x00)
        Time to live: 128 seconds/hops
        IP protocol type: TCP (0x06)
        Checksum: 0x4be2
        IP address: 192.168.8.100 ---> 192.168.9.101
        No option
Transmission Control Protocol
        Port 3881 ---> 80
        Sequence Number: 4047880147
        Acknowledgement Number: 2434401980
        Header Length(MSB 4 bits): 5 (32-bit word)
        Reserved(LSB 4 bits): 0
        Code: 0x18
                RES: 0... ....
                CON: .0.. ....
                URG: ..0. ....
                ACK: ...1 ....
                PSH: .... 1...
                RST: .... .0..
                SYN: .... ..0.
                FIN: .... ...0
        Window: 64000
        Checksum: 0xe90f
        Urgent Pointer: 0x0000
Data:
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a | GET / HTTP/1.1..
0010: 48 6f 73 74 3a 20 31 39 32 2e 31 36 38 2e 38 2e | Host: 192.168.8.
0020: 32 32 32 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a | 222..Accept: */*
0030: 0d 0a 0d 0a 00 00 00 00 00 00 00 00 00 00 00 00 | ................
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
0050: 00 00 00 00 00 00 00 00                         | ........

 

The packet is leaving via port 3 (real server facing) and the IP header is showing the following:

 

source IP: 192.168.8.100

destination IP: 192.168.9.101

 

This is a packet coming from the client going to the real server.

 

Tips and Tricks / Latest News

 

- It is a good practice to RESET a filter ID as soon as you want to define the filter. Filter ID settings do survive until the unit is getting rebooted. A reset ensures the filter is back at the default which implies "match ALL":

 

telnet@ServerIronADX 1000(debug-filter)#sp 2
telnet@ServerIronADX 1000(debug-filter-spec-2)#reset

 

- The ADX is able to store traces in PCAP format - have a look at the following post to get some more details: debug filter and pcap format

 

- The ADX offers up to 256 MB of capturing space - JetCore is limited to 1 MB

 

- The ADX at different location like MP (mgmt processor), BP (barrel processor), SFM (switch fabric module), LC (line card) - JetCore offers MP and BP only