Isn't it time to open a TAC case when you have nothing to account for the
problems you are seeing. Clearly if you take it off and everything returns
to normal then you have already narrowed it down to the source of the
problem.
Regards,
Tyson Scott - CCIE #13513 R&S, Security, and SP
Technical Instructor - IPexpert, Inc.
Mailto: tscott_at_ipexpert.com
Telephone: +1.810.326.1444, ext. 208
Live Assistance, Please visit: www.ipexpert.com/chat
eFax: +1.810.454.0130
-----Original Message-----
From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf Of
Joshua
Sent: Friday, January 15, 2010 2:08 AM
To: ccielab_at_groupstudy.com
Subject: Flexible Netflow causes Cisco 2821 CPU Hogging
Hi Guys,
One of our remote site Cisco 2821 is running IPSec VPN in a hub-and-spokes
topology environment. SolarWinds netflow analyzer is running on a server at
hub location. For some reasons, as long as flexible netflow applied on Cisco
2821 interfaces, CPU utilization reached closed 100%. But "sh proc cpu"
shows nothing.
Below is information from "show xxx". Please help!
Thanks,
Joshua
-sh ver
-sh run
-sh proc cpu | e 0.00
- sh int stat
-sh inter switching
-sh int | in proto|queue|rate|err
==================Remote Router sh run============
System image file is "flash:c2800nm-advsecurityk9-mz.124-24.T2.bin"
Remote_Office#sh run
...
!
flow exporter 701-0174
destination 10.10.50.206
source GigabitEthernet0/1
output-features
transport udp 2055
export-protocol netflow-v5
!
!
flow monitor flow-monitor
record netflow-original
exporter 701-0174
cache timeout active 1
!
ip source-route
!
!
ip cef
no ip dhcp use vrf connected
ip dhcp excluded-address 10.9.2.69
ip dhcp excluded-address 10.9.2.192 10.9.2.254
ip dhcp excluded-address 10.9.2.1 10.9.2.31
!
ip dhcp pool PROD
network 10.9.2.0 255.255.255.0
default-router 10.9.2.2
dns-server 10.9.2.69 10.10.5.155
domain-name abc.net
!
!
no ip domain lookup
ip domain name yourdomain.com
!
multilink bundle-name authenticated
!
!
!
!
username cisco privilege 15 secret 5 $1$U95M$il6Xa8ObGGTerhddWe27y1
!
!
crypto isakmp policy 10
encr 3des
hash md5
authentication pre-share
crypto isakmp key EK2CLRS2 address 120.239.178.3
crypto isakmp key EK2CLRS2 address 172.164.230.218
crypto isakmp key EK2CLRS2 address 167.133.22.142
crypto isakmp key EK2CLRS2 address 171.6.24.75
crypto isakmp key EK2CLRS2 address 163.239.217.98
crypto isakmp key EK2CLRS2 address 165.115.64.18
!
!
crypto ipsec transform-set myset esp-3des esp-md5-hmac
!
crypto map mymap 10 ipsec-isakmp
set peer 124.239.178.3
set transform-set myset
match address 150
crypto map mymap 20 ipsec-isakmp
set peer 172.164.230.218
set transform-set myset
match address 155
crypto map mymap 30 ipsec-isakmp
set peer 167.133.22.142
set transform-set myset
match address 156
crypto map mymap 40 ipsec-isakmp
set peer 171.6.24.75
set transform-set myset
match address 165
crypto map mymap 50 ipsec-isakmp
set peer 163.239.217.98
set transform-set myset
match address 175
crypto map mymap 60 ipsec-isakmp
set peer 165.115.64.18
set transform-set myset
match address 185
!
archive
log config
hidekeys
!
!
interface GigabitEthernet0/0
description Conntect to Internet via T1
ip address 165.126.217.2 255.255.255.224
ip flow monitor flow-monitor input
ip nat outside
no ip virtual-reassembly
duplex auto
speed auto
no cdp enable
crypto map mymap
!
interface GigabitEthernet0/1
description Inside
ip address 10.9.2.2 255.255.255.0
ip access-group 120 in
ip accounting output-packets
ip flow monitor flow-monitor input
ip nat inside
ip virtual-reassembly
duplex auto
speed auto
!
interface Serial0/3/0
no ip address
shutdown
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 165.126.217.1
no ip http server
ip http access-class 23
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
!
ip nat inside source route-map internet interface GigabitEthernet0/0
overload
!
logging source-interface GigabitEthernet0/1
logging 10.10.50.206
access-list 120 permit tcp any host 10.10.50.132 eq 2967
access-list 120 permit udp any host 10.10.50.132 eq 2967
access-list 120 deny tcp any any eq 2967
access-list 120 deny udp any any eq 2967
access-list 120 permit ip any any
access-list 150 permit ip 10.9.2.0 0.0.0.255 10.10.0.0 0.0.127.255
access-list 155 permit ip 10.9.2.0 0.0.0.255 10.9.18.0 0.0.0.255
access-list 156 permit ip 10.9.2.0 0.0.0.255 10.9.24.0 0.0.0.255
access-list 160 deny ip 10.9.2.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 160 deny ip 10.9.2.0 0.0.0.255 10.9.18.0 0.0.0.255
access-list 160 deny ip 10.9.2.0 0.0.0.255 10.9.24.0 0.0.0.255
access-list 160 deny ip 10.9.2.0 0.0.0.255 10.10.132.0 0.0.0.255
access-list 160 deny ip 10.9.2.0 0.0.0.255 10.10.136.0 0.0.0.255
access-list 160 deny ip 10.9.2.0 0.0.0.255 10.9.30.0 0.0.0.255
access-list 160 permit ip 10.9.2.0 0.0.0.255 any
access-list 165 permit ip 10.9.2.0 0.0.0.255 10.10.132.0 0.0.0.255
access-list 175 permit ip 10.9.2.0 0.0.0.255 10.9.30.0 0.0.0.255
access-list 185 permit ip 10.9.2.0 0.0.0.255 10.10.136.0 0.0.0.255
!
!
!
route-map internet permit 10
match ip address 160
!
!
snmp-server community ledcorsnmp RO
snmp-server enable traps tty
snmp-server enable traps frame-relay multilink bundle-mismatch
...
==============================================
Remote_Office#sho proc cpu | e 0.00
CPU utilization for five seconds: 98%/95%; one minute: 92%; five minutes:
86%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
2 659264 181712 3628 2.16% 1.06% 1.24% 0 Load Meter
41 1094688 916216 1194 0.08% 0.04% 0.05% 0 Per-Second
Jobs
82 534812 906685 589 0.08% 0.06% 0.06% 0 Kontrol
Common
H
111 1974804 1380744 1430 0.17% 0.10% 0.08% 0 IP Input
171 1292788 221834927 5 0.34% 0.35% 0.36% 0 HQF Shaper
Backg
===================================================
Remote_Office#sho int stat
GigabitEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 387367 58716537 241702 26394423
Route cache 34135062 2871254989 29089399 3775098211
Total 34522423 2929963330 29331096 3801492634
GigabitEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 355571 41417261 233322 46737972
Route cache 29076432 2396714220 34048500 1240926749
Total 29432003 2438131481 34281822 1287664721
Interface Serial0/3/0 is disabled
NVI0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 0 0
Route cache 0 0 0 0
Total 0 0 0 0
===========================================
Remote_Office#sho interface switching
GigabitEthernet0/0 Conntect to Internet via T1
Throttle count 2
Drops RP 59 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 585256 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 775766 107911263 619547 78146041
Cache misses 0 - - -
Fast 79945420 3102930096 68075262 3761958554
Auton/SSE 0 0 0 0
Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 1518 116886
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 490959 29457540 1524 91440
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 90790 5447400
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
GigabitEthernet0/1 Inside
Throttle count 15
Drops RP 2384 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 142713 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 685757 84939238 358465 68724586
Cache misses 0 - - -
Fast 67552473 339769035 79588748 3624304061
Auton/SSE 0 0 0 0
Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 1518 116886
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
===============================================
SanDiegoOffice#sh flow mo
Flow Monitor flow-monitor:
Description: User defined
Flow Record: netflow-original
Flow Exporter: 701-0174
Cache:
Type: normal
Status: allocated
Size: 4096 entries / 327700 bytes
Inactive Timeout: 15 secs
Active Timeout: 1 secs
Update Timeout: 1800 secs
================================================
Remote_Office#sho int | i proto|queue|rate|err
GigabitEthernet0/0 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1617000 bits/sec, 215 packets/sec
5 minute output rate 1007000 bits/sec, 175 packets/sec
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
29334368 packets output, 3802888567 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
GigabitEthernet0/1 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 931000 bits/sec, 173 packets/sec
5 minute output rate 1527000 bits/sec, 212 packets/sec
161 input errors, 0 CRC, 0 frame, 0 overrun, 161 ignored
34286645 packets output, 1293090925 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
57181 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
Serial0/3/0 is administratively down, line protocol is down
Hardware is GT96K with integrated T1 CSU/DSU
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
NVI0 is up, line protocol is up
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
Blogs and organic groups at http://www.ccie.net
Received on Fri Jan 15 2010 - 09:13:41 ART
This archive was generated by hypermail 2.2.0 : Thu Feb 04 2010 - 20:28:41 ART