Wednesday, 11 December 2013

Troubleshooting asymmetric routing issues in Cisco WAE

•See for Current Active Pass-Through Flows in “sh statistics connection ”
•There are multiple reasons why the traffic may be in pass through mode. It may be because of asymmetric routing, or it may be because of policy or classifier, or may be the traffic is not being redirected through WCCP
•Then go and see pass through traffic in
#sh statistics pass-through | i As (In Cisco WAE)
•Asymmetric                              1284                  88268196
•Which will show you active and closed connection(1284 active,88268196 closed)
•If we found active, then the next step is go and check for what connection this is active.
sh statistics connection pass-through | i As
•And that’s BINGO!!!!...

•Following the above steps you will be able to observe that for the given source and destination your WAE is working or not and if not then is there any issue regarding asymmetric routing.

Example of an overload condition(On Cisco WAE)

Its very easy to identify your WAE is overloaded or not. Just have to run few commands and observe the outputs which are highlighted as below.

#sh statistics con(On Cisco WAE)
•
•Current Active Optimized Flows:                      5960-->nearly equal to connection limit
•   Current Active Optimized TCP Plus Flows:          5609
•   Current Active Optimized TCP Only Flows:          349
•   Current Active Optimized TCP Preposition Flows:   0
•Current Active Auto-Discovery Flows:                 23
•Current Reserved Flows:                              40
•Current Active Pass-Through Flows:                   1260
•Historical Flows:                                    585
•
#sh tfo detail(On Cisco WAE)
•
•   Policy Engine Config Item            Value              
•   -------------------------            -----              
•   State                                Registered         
•   Default Action                       Use Policy         

•   Connection Limit                     6000

•Policy Engine Statistics       
•   -------------------------      
•   Session timeouts: 0,  Total timeouts: 0
•   Last keepalive received 00.4 Secs ago
•   Last registration occurred 417:10:10:06.7 Days:Hours:Mins:Secs ago
•   Hits:               1693419791, Update Released:              886466302
•   Active Connections:       5960, Completed Connections:        476081344
•   Drops:                       0, Pre-Resource Counter:         21
•   Rejected Connection Counts Due To: (Total: 342344669)
•      Not Registered      :          0,  Keepalive Timeout   :          0
•      No License          :          0,  Load Level          :          0
•      Connection Limit    :  342344669,  Rate Limit          :          0
•      Minimum TFO         :          0,  Resource Manager    :          0
•      Global Config       :          0,  TFO Overload        :          0
•      Server-Side         :          0,  DM Deny             :          0
•      No DM Accept        :          0

TFO Troubleshooting(Cisco WAE)

•TFO-Transport Flow Optimization; a set of optimizations applied to intercepted traffic by the WAAS devices  
•The number of TCP connections, their status, and disposition can give an indication of the health of the WAAS system in a specific location.

•The show statistics tfo detailcommand provides an indication of the volume, status, and disposition of connections between a particular WAAS device and other devices in the network.

#sh statistics tfo detail(On Cisco WAE)
•  Total number of connections                          : 81350884
•  No. of active connections                            : 578
•  No. of pending (to be accepted) connections          : 0
•  No. of bypass connections                            : 292144
•  No. of normal closed conns                           : 78445114
•  No. of reset connections                             : 2905192
•     Socket write failure                              : 1704
•     Socket read failure                               : 2
•     WAN socket close while waiting to write           : 967
•     AO socket close while waiting to write            : 147326
•     WAN socket error close while waiting to read      : 0
•     AO socket error close while waiting to read       : 1249665
•WAN socket unexpected close while waiting to read : 77254
•     Exceeded maximum number of supported connections  : 0
•     Buffer allocation or manipulation failed          : 0
•     Peer received reset from end host                 : 1427862
•     DRE connection state out of sync                  : 0
•     Memory allocation failed for buffer heads         : 0
•     Unoptimized packet received on optimized side     : 412
• 
•   Policy Engine Statistics       
•   -------------------------      
•   Session timeouts: 0,  Total timeouts: 0
•   Last keepalive received 00.7 Secs ago
•   Last registration occurred 84:09:23:33.0 Days:Hours:Mins:Secs ago
•   Hits:                424524224, Update Released:              305682847
•   Active Connections:        687, Completed Connections:        161054837
•   Drops:                       0, Pre-Resource Counter:        347
•   Rejected Connection Counts Due To: (Total: 0)
•      Not Registered      :          0,  Keepalive Timeout   :          0
•      No License          :          0,  Load Level          :          0
•      Connection Limit    :          0,  Rate Limit          :          0
•      Minimum TFO         :          0,  Resource Manager    :          0
•      Global Config       :          0,  TFO Overload        :          0
•      Server-Side         :          0,  DM Deny             :          0
•      No DM Accept        :          0
•
•   Auto-Discovery Statistics      
•   -------------------------      
•   Total Connections queued for accept:  81350826
•   Connections queuing failures:         0
•   Socket pairs queued for accept:       81350826
•   Socket pairs queuing failures:        0
•   AO discovery successful:              0
•   AO discovery failure:                 0



•The No. of active connections field reports the number of connections that are currently being optimized.
•No. of pending (to be accepted) connections-Number of TCP connections that will be optimized but are currently in the setup stage.
•No. of connections closed normally-Number of optimized connections closed without any issues using TCP FIN.
•No. of connections closed with error-Number of optimized connection closed with some issues or using TCP RST.
•DRE decode failure-DRE internal error while decoding data. (Should not happen.)
•DRE encode failure-DRE internal error while encoding data. (Should not happen.)
•Connection init failure-Failed to setup the connection although auto-discovery finished successfully..
•Peer received reset from end host-TCP RST sent by the server or client. (Can be normal behavior and does not necessarily indicate a problem.)
•In the Policy Engine Statistics section of the output, the Rejected Connection Counts section show various reasons why connections have been rejected. The Connection Limit counter reports the number of times that a connection has been rejected because the maximum number of optimized connections has been exceeded. If you see a high number here, you should look into overload conditions.


#sh statistics connection(On Cisco WAE)
•
•Current Active Optimized Flows:                      660
•   Current Active Optimized TCP Plus Flows:          647
•   Current Active Optimized TCP Only Flows:          14
•   Current Active Optimized TCP Preposition Flows:   0
•Current Active Auto-Discovery Flows:                 320
•Current Reserved Flows:                              80
•Current Active Pass-Through Flows:                   4659
•Historical Flows:                                    687
•
•
•D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio
•A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
•
•ConnID        Source IP:Port          Dest IP:Port            PeerID Accel RR  
•983071      172.31.95.8:3732    155.64.28.160:8014 00:21:5e:76:8b:88 TDL   00.0%
•983196   172.31.113.13:56581    155.64.28.160:8014 00:21:5e:76:8b:88 TDL   00.0%
•327923   172.21.88.239:60790      155.64.28.32:135               N/A E     00.0%

•Current Active Optimized TCP Plus Flows-they are all being handled with TFO/DRE/LZ optimization
•Current Active Optimized TCP Only Flows:-flows that are optimized by TFO only.
•Current Active Auto-Discovery Flows:-displays flows that have not been fully set up to become optimized flows or pass-through flows.
•The Current Active Pass-Through Flows counter shows connections that the device has determined to be pass-through or where the device did not see the SYN, SYN ACK, ACK setup. These flows will not be counted as optimized flows. For pass-through flows, a device should be able to handle up to 10 times the number of optimized flows for which it is rated.
•The sum of the following three counters tells you how close the WAE device is to its connection limit:
•Current Active Optimized Flows
•Current Active Auto-Discovery Flows
•Current Reserved Flows (available only in 4.1.5 and later)
•If this sum is equal to or greater than the connection limit, the device is in an overload condition.

Identify routing loops(On Cisco WAE)

•If a WAE detects its own ID returned in the TCP options field, a redirection loop has occurred and results in the following syslog message:
•%WAAS-SYS-3-900000: 137.34.79.11:1192 - 137.34.77.196:139 - opt_syn_rcv: Routing Loop detected - Packet has our own devid. Packet dropped.
•You can search the syslog.txt file for instances of this error by using the find command as follows
#find-pattern match "Routing Loop" syslog.txt (On Cisco WAE)
•As per cisco -If you are doing outbound redirection on the router, as traffic leaves the router it will get redirected back to the WAE, which will reroute the packet out the router, causing a routing loop
•But if we are not using outbound redirection so we have to study why we are facing this(As far as my practical knowledge says avoid using outbound redirection , it increases your troubleshoting part if loop occurs )

•I have put this here just to ensure that there should not be routing loop errors while troubleshooting, it’s a serious problem.

Troubleshooting WCCP on the WAE

•Begin troubleshooting on the WAE by using the show wccp services command. You want to see both services 61 and 62 configured, as follows
sh wccp services(On Cisco WAE)
Services configured on this Wide Area Engine
        TCP Promiscuous 61
        TCP Promiscuous 62
Next check the WCCP status by using the show wccp status command. You want to see that WCCP version 2 is enabled and active as follows:
#sh wccp status(On Cisco WAE)
WCCP version 2 is enabled and currently active
sh wccp routers(On Cisco WAE)
Router Information for Service: TCP Promiscuous 61
        Routers Seeing this Wide Area Engine(2)
        Router Id       Sent To
        x.x.x.x            y.y.y.y
        x1.x1.x1.x1   y1.y1.y1.y1

sh wccp gre(On Cisco WAE)
•Transparent GRE packets received:              0<-----Increments for WCCP GRE redirection
•Transparent non-GRE packets received:          2467369718<-----Increments for WCCP L2                                                                                                                            redirection
•Transparent non-GRE non-WCCP packets received: 0<-----Increments for ACE or PBR redirection
•Total packets accepted:                        892102614<-----Accepted for optimization; peer WAE                                                                                                                        found
•Invalid packets received:                      7
•Packets received with invalid service:         0
•Packets received on a disabled service:        0
•Packets received too small:                    0
•Packets dropped due to zero TTL:               0
•Packets dropped due to bad buckets:            0
•Packets dropped due to no redirect address:    0
•Packets dropped due to loopback redirect:      0
•Pass-through pkts on non-owned bucket:         0
•Connections bypassed due to load:              0
•Packets sent back to router:                   348
•GRE packets sent to router (not bypass):       0
•Packets sent to another WAE:                   0
•GRE fragments redirected:                      140487
•GRE encapsulated fragments received:           0
•Packets failed encapsulated reassembly:        0
•Packets failed GRE encapsulation:              0
•Packets dropped due to invalid fwd method:     0
•Packets dropped due to insufficient memory:    0
•Packets bypassed, no pending connection:       0
•Connections bypassed during wccp shutdown:     0
•Connections bypassed due to bypass-list lookup:0
•Conditionally Accepted connections:            0
•Conditionally Bypassed connections:            0
•L2 Bypass packets destined for loopback:       0
•Packets w/WCCP GRE received too small:         0
•Packets dropped due to received on loopback:   0
•Packets dropped due to IP access-list deny:    0
•Packets fragmented for bypass:                 23
•Packets fragmented for egress:                 0
•Packet pullups needed:                         28610
•Packets dropped due to no route found:         0



•If WCCP redirection is working, either of the first two counters should be increment.
•The Transparent non-GRE packets received counter increments for packets that are redirected using the WCCP Layer 2 redirect forwarding method.
•The Transparent non-GRE non-WCCP packets received counter increments for packets that are redirected by a non-WCCP interception method (such as ACE or PBR).
•The Total packets accepted counter indicates packets that are accepted for optimization because auto-discovery found a peer WAE.

Cisco WAAS (Basic concepts)

Watch the complete Video of this section -


WAAS is a symmetric solution requiring a WAAS component on both sides of the conversation. Peer-to-peer. WAN traffic between WAAS components is highly optimized while still visible to QoS/Management applications

WAAS devices auto-discover each other without any user intervention allowing optimization not only to the data center but between WAAS enabled branches



The WAAS administrator can classify which WAN traffic should be optimized based on criteria such as protocol (port number), source and/or destination IP address and/or network. 

WAAS Overview Application Acceleration Transparency
Packet network transparency (L3/L4 headers) allows application acceleration components to maintain compliance with existing network features

•WAAS Overview TFO Auto-Discovery – TCP SYN
•When the client sends a TCP-SYN packet, WAE1 will apply TCP options to identify itself and specify the optimizations that it would like to apply
•The modified TCP-SYN packet is then forwarded to the server, and intercepted on the other side 



•WAAS Overview TFO Auto-Discovery – TCP SYN (Cont.)
•Once WAE2 receives the TCP-SYN packet with the options marked, it then knows WAE1’s details and desire to optimize this connection
•The TCP-SYN packet is then forwarded to the server 

•WAAS Overview TFO Auto-Discovery – TCP SYN-ACK
•When the server responds with the TCP SYN-ACK, WAE2 then marks TCP options to acknowledge optimization and to identify itself to WAE1
The marked TCP SYN-ACK packet is then forwarded towards the client and intercepted on the other side

•WAAS Overview TFO Auto-Discovery – TCP SYN-ACK (Cont.)
•When WAE1 receives the TCP SYN-ACK with the optimization confirmation and details about WAE2, the defined policy (or negotiated optimizations) can then be acknowledged
•The TCP SYN-ACK packet is then forwarded to the client 
•WAAS Overview TFO Auto-Discovery – TCP ACK
•After the SYN-ACK is received, the TCP proxy is initiated for the connection, and WAE1 sends a TCP ACK to WAE2 to acknowledge optimizations
•WAE2 can then send a TCP ACK to Server B

Client A sends a TCP ACK to WAE1