Wednesday 11 December 2013

Troubleshooting asymmetric routing issues in Cisco WAE

See for Current Active Pass-Through Flows in “sh statistics connection ”
There are multiple reasons why the traffic may be in pass through mode. It may be because of asymmetric routing, or it may be because of policy or classifier, or may be the traffic is not being redirected through WCCP
Then go and see pass through traffic in
#sh statistics pass-through | i As (In Cisco WAE)
Asymmetric                              1284                  88268196
Which will show you active and closed connection(1284 active,88268196 closed)
If we found active, then the next step is go and check for what connection this is active.
sh statistics connection pass-through | i As
And that’s BINGO!!!!...

Following the above steps you will be able to observe that for the given source and destination your WAE is working or not and if not then is there any issue regarding asymmetric routing.

Example of an overload condition(On Cisco WAE)

Its very easy to identify your WAE is overloaded or not. Just have to run few commands and observe the outputs which are highlighted as below.

#sh statistics con(On Cisco WAE)
Current Active Optimized Flows:                      5960-->nearly equal to connection limit
   Current Active Optimized TCP Plus Flows:          5609
   Current Active Optimized TCP Only Flows:          349
   Current Active Optimized TCP Preposition Flows:   0
Current Active Auto-Discovery Flows:                 23
Current Reserved Flows:                              40
Current Active Pass-Through Flows:                   1260
Historical Flows:                                    585
#sh tfo detail(On Cisco WAE)
   Policy Engine Config Item            Value              
   -------------------------            -----              
   State                                Registered         
   Default Action                       Use Policy         

   Connection Limit                     6000

Policy Engine Statistics       
   -------------------------      
   Session timeouts: 0,  Total timeouts: 0
   Last keepalive received 00.4 Secs ago
   Last registration occurred 417:10:10:06.7 Days:Hours:Mins:Secs ago
   Hits:               1693419791, Update Released:              886466302
   Active Connections:       5960, Completed Connections:        476081344
   Drops:                       0, Pre-Resource Counter:         21
   Rejected Connection Counts Due To: (Total: 342344669)
      Not Registered      :          0,  Keepalive Timeout   :          0
      No License          :          0,  Load Level          :          0
      Connection Limit    :  342344669,  Rate Limit          :          0
      Minimum TFO         :          0,  Resource Manager    :          0
      Global Config       :          0,  TFO Overload        :          0
      Server-Side         :          0,  DM Deny             :          0
      No DM Accept        :          0

TFO Troubleshooting(Cisco WAE)

TFO-Transport Flow Optimization; a set of optimizations applied to intercepted traffic by the WAAS devices  
The number of TCP connections, their status, and disposition can give an indication of the health of the WAAS system in a specific location.

The show statistics tfo detailcommand provides an indication of the volume, status, and disposition of connections between a particular WAAS device and other devices in the network.

#sh statistics tfo detail(On Cisco WAE)
  Total number of connections                          : 81350884
  No. of active connections                            : 578
  No. of pending (to be accepted) connections          : 0
  No. of bypass connections                            : 292144
  No. of normal closed conns                           : 78445114
  No. of reset connections                             : 2905192
     Socket write failure                              : 1704
     Socket read failure                               : 2
     WAN socket close while waiting to write           : 967
     AO socket close while waiting to write            : 147326
     WAN socket error close while waiting to read      : 0
     AO socket error close while waiting to read       : 1249665
WAN socket unexpected close while waiting to read : 77254
     Exceeded maximum number of supported connections  : 0
     Buffer allocation or manipulation failed          : 0
     Peer received reset from end host                 : 1427862
     DRE connection state out of sync                  : 0
     Memory allocation failed for buffer heads         : 0
     Unoptimized packet received on optimized side     : 412
 
   Policy Engine Statistics       
   -------------------------      
   Session timeouts: 0,  Total timeouts: 0
   Last keepalive received 00.7 Secs ago
   Last registration occurred 84:09:23:33.0 Days:Hours:Mins:Secs ago
   Hits:                424524224, Update Released:              305682847
   Active Connections:        687, Completed Connections:        161054837
   Drops:                       0, Pre-Resource Counter:        347
   Rejected Connection Counts Due To: (Total: 0)
      Not Registered      :          0,  Keepalive Timeout   :          0
      No License          :          0,  Load Level          :          0
      Connection Limit    :          0,  Rate Limit          :          0
      Minimum TFO         :          0,  Resource Manager    :          0
      Global Config       :          0,  TFO Overload        :          0
      Server-Side         :          0,  DM Deny             :          0
      No DM Accept        :          0
   Auto-Discovery Statistics      
   -------------------------      
   Total Connections queued for accept:  81350826
   Connections queuing failures:         0
   Socket pairs queued for accept:       81350826
   Socket pairs queuing failures:        0
   AO discovery successful:              0
   AO discovery failure:                 0



The No. of active connections field reports the number of connections that are currently being optimized.
No. of pending (to be accepted) connections-Number of TCP connections that will be optimized but are currently in the setup stage.
No. of connections closed normally-Number of optimized connections closed without any issues using TCP FIN.
No. of connections closed with error-Number of optimized connection closed with some issues or using TCP RST.
DRE decode failure-DRE internal error while decoding data. (Should not happen.)
DRE encode failure-DRE internal error while encoding data. (Should not happen.)
Connection init failure-Failed to setup the connection although auto-discovery finished successfully..
Peer received reset from end host-TCP RST sent by the server or client. (Can be normal behavior and does not necessarily indicate a problem.)
In the Policy Engine Statistics section of the output, the Rejected Connection Counts section show various reasons why connections have been rejected. The Connection Limit counter reports the number of times that a connection has been rejected because the maximum number of optimized connections has been exceeded. If you see a high number here, you should look into overload conditions.


#sh statistics connection(On Cisco WAE)
Current Active Optimized Flows:                      660
   Current Active Optimized TCP Plus Flows:          647
   Current Active Optimized TCP Only Flows:          14
   Current Active Optimized TCP Preposition Flows:   0
Current Active Auto-Discovery Flows:                 320
Current Reserved Flows:                              80
Current Active Pass-Through Flows:                   4659
Historical Flows:                                    687
D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
ConnID        Source IP:Port          Dest IP:Port            PeerID Accel RR  
983071      172.31.95.8:3732    155.64.28.160:8014 00:21:5e:76:8b:88 TDL   00.0%
983196   172.31.113.13:56581    155.64.28.160:8014 00:21:5e:76:8b:88 TDL   00.0%
327923   172.21.88.239:60790      155.64.28.32:135               N/A E     00.0%

Current Active Optimized TCP Plus Flows-they are all being handled with TFO/DRE/LZ optimization
Current Active Optimized TCP Only Flows:-flows that are optimized by TFO only.
Current Active Auto-Discovery Flows:-displays flows that have not been fully set up to become optimized flows or pass-through flows.
The Current Active Pass-Through Flows counter shows connections that the device has determined to be pass-through or where the device did not see the SYN, SYN ACK, ACK setup. These flows will not be counted as optimized flows. For pass-through flows, a device should be able to handle up to 10 times the number of optimized flows for which it is rated.
The sum of the following three counters tells you how close the WAE device is to its connection limit:
Current Active Optimized Flows
Current Active Auto-Discovery Flows
Current Reserved Flows (available only in 4.1.5 and later)
If this sum is equal to or greater than the connection limit, the device is in an overload condition.

Identify routing loops(On Cisco WAE)

If a WAE detects its own ID returned in the TCP options field, a redirection loop has occurred and results in the following syslog message:
%WAAS-SYS-3-900000: 137.34.79.11:1192 - 137.34.77.196:139 - opt_syn_rcv: Routing Loop detected - Packet has our own devid. Packet dropped.
You can search the syslog.txt file for instances of this error by using the find command as follows
#find-pattern match "Routing Loop" syslog.txt (On Cisco WAE)
As per cisco -If you are doing outbound redirection on the router, as traffic leaves the router it will get redirected back to the WAE, which will reroute the packet out the router, causing a routing loop
But if we are not using outbound redirection so we have to study why we are facing this(As far as my practical knowledge says avoid using outbound redirection , it increases your troubleshoting part if loop occurs )

I have put this here just to ensure that there should not be routing loop errors while troubleshooting, it’s a serious problem.

Troubleshooting WCCP on the WAE

Begin troubleshooting on the WAE by using the show wccp services command. You want to see both services 61 and 62 configured, as follows
sh wccp services(On Cisco WAE)
Services configured on this Wide Area Engine
        TCP Promiscuous 61
        TCP Promiscuous 62
Next check the WCCP status by using the show wccp status command. You want to see that WCCP version 2 is enabled and active as follows:
#sh wccp status(On Cisco WAE)
WCCP version 2 is enabled and currently active
sh wccp routers(On Cisco WAE)
Router Information for Service: TCP Promiscuous 61
        Routers Seeing this Wide Area Engine(2)
        Router Id       Sent To
        x.x.x.x            y.y.y.y
        x1.x1.x1.x1   y1.y1.y1.y1

sh wccp gre(On Cisco WAE)
Transparent GRE packets received:              0<-----Increments for WCCP GRE redirection
Transparent non-GRE packets received:          2467369718<-----Increments for WCCP L2                                                                                                                            redirection
Transparent non-GRE non-WCCP packets received: 0<-----Increments for ACE or PBR redirection
Total packets accepted:                        892102614<-----Accepted for optimization; peer WAE                                                                                                                        found
Invalid packets received:                      7
Packets received with invalid service:         0
Packets received on a disabled service:        0
Packets received too small:                    0
Packets dropped due to zero TTL:               0
Packets dropped due to bad buckets:            0
Packets dropped due to no redirect address:    0
Packets dropped due to loopback redirect:      0
Pass-through pkts on non-owned bucket:         0
Connections bypassed due to load:              0
Packets sent back to router:                   348
GRE packets sent to router (not bypass):       0
Packets sent to another WAE:                   0
GRE fragments redirected:                      140487
GRE encapsulated fragments received:           0
Packets failed encapsulated reassembly:        0
Packets failed GRE encapsulation:              0
Packets dropped due to invalid fwd method:     0
Packets dropped due to insufficient memory:    0
Packets bypassed, no pending connection:       0
Connections bypassed during wccp shutdown:     0
Connections bypassed due to bypass-list lookup:0
Conditionally Accepted connections:            0
Conditionally Bypassed connections:            0
L2 Bypass packets destined for loopback:       0
Packets w/WCCP GRE received too small:         0
Packets dropped due to received on loopback:   0
Packets dropped due to IP access-list deny:    0
Packets fragmented for bypass:                 23
Packets fragmented for egress:                 0
Packet pullups needed:                         28610
Packets dropped due to no route found:         0



If WCCP redirection is working, either of the first two counters should be increment.
The Transparent non-GRE packets received counter increments for packets that are redirected using the WCCP Layer 2 redirect forwarding method.
The Transparent non-GRE non-WCCP packets received counter increments for packets that are redirected by a non-WCCP interception method (such as ACE or PBR).
The Total packets accepted counter indicates packets that are accepted for optimization because auto-discovery found a peer WAE.

Cisco WAAS (Basic concepts)

Watch the complete Video of this section -


WAAS is a symmetric solution requiring a WAAS component on both sides of the conversation. Peer-to-peer. WAN traffic between WAAS components is highly optimized while still visible to QoS/Management applications

WAAS devices auto-discover each other without any user intervention allowing optimization not only to the data center but between WAAS enabled branches



The WAAS administrator can classify which WAN traffic should be optimized based on criteria such as protocol (port number), source and/or destination IP address and/or network. 

WAAS Overview Application Acceleration Transparency
Packet network transparency (L3/L4 headers) allows application acceleration components to maintain compliance with existing network features

WAAS Overview TFO Auto-Discovery – TCP SYN
When the client sends a TCP-SYN packet, WAE1 will apply TCP options to identify itself and specify the optimizations that it would like to apply
The modified TCP-SYN packet is then forwarded to the server, and intercepted on the other side 



WAAS Overview TFO Auto-Discovery – TCP SYN (Cont.)
Once WAE2 receives the TCP-SYN packet with the options marked, it then knows WAE1’s details and desire to optimize this connection
The TCP-SYN packet is then forwarded to the server 

WAAS Overview TFO Auto-Discovery – TCP SYN-ACK
When the server responds with the TCP SYN-ACK, WAE2 then marks TCP options to acknowledge optimization and to identify itself to WAE1
The marked TCP SYN-ACK packet is then forwarded towards the client and intercepted on the other side

WAAS Overview TFO Auto-Discovery – TCP SYN-ACK (Cont.)
When WAE1 receives the TCP SYN-ACK with the optimization confirmation and details about WAE2, the defined policy (or negotiated optimizations) can then be acknowledged
The TCP SYN-ACK packet is then forwarded to the client 
WAAS Overview TFO Auto-Discovery – TCP ACK
After the SYN-ACK is received, the TCP proxy is initiated for the connection, and WAE1 sends a TCP ACK to WAE2 to acknowledge optimizations
WAE2 can then send a TCP ACK to Server B

Client A sends a TCP ACK to WAE1