Cisco Internetwork Troubleshooting Correcting the Problem at the Transport and Application Layers © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT Commands Used to Correct Transport Layer Problems access-list {access-list-number} {deny | permit} {tcp | udp} source source-wildcard destination destination- wildcard [log] router(config-if)# Defines an extended access list. ip access-list {standard | extended} {access-list-name} router(config-if)# Defines a standard or extended named access list. ip access-group {access-list-number | access-list-name} router(config)# Applies an extended access list.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Correcting an Extended Access List Problem at the Transport Layer Next Animation Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>enable Columbia#conf terminal Enter configuration commands, one per line. End with CNTL/Z. Columbia(config)#ip access-list extended Traffic Columbia(config-ext-nacl)#permit tcp any eq telnet Columbia(config-ext-nacl)#exit Columbia# Dec 19 16:16:02: %SYS-5-CONFIG_I: Configured from console by console Columbia#show access-lists Traffic Extended IP access list Traffic permit icmp any any (15 matches) permit tcp any eq ftp-data permit tcp any eq ftp permit tcp any eq www permit udp any eq tftp permit tcp any eq telnet Columbia# Correcting an Extended Access List Problem at the Transport Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW>telnet Baltimore Trying Baltimore ( )... Open BaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBase Baltimore an ACME Distribution Workgroup Router -- Baseline -- BaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBase User Access Verification Password: Baltimore> Verifying the Correction to the Misconfigured Access List

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Correcting a Problem at the Transport Layer Next Animation Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#conf t SanFran(config-if)#interface fastethernet 0/0 SanFran(config-if)#ip route-cache flow SanFran(config-if)#interface fastethernet 0/1 SanFran(config-if)#ip route-cache flow SanFran(config-if)#^Z SanFran# Configuring IP Cache Flow Switching on SanFran and Oakland Oakland#conf t Oakland(config-if)#interface fastethernet 0/1 Oakland(config-if)#ip route-cache flow Oakland(config-if)#interface fastethernet 0/0 Oakland(config-if)#ip route-cache flow Oakland(config-if)#^Z Oakland#

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show version Cisco Internetwork Operating System Software IOS (tm) C2600 Software (C2600-IO3-M), Version 12.2(10a), RELEASE SOFTWARE (fc1) Copyright (c) by cisco Systems, Inc. Compiled Tue 21-May-02 13:57 by pwade Image text-base: 0x , data-base: 0x80A11A68 ROM: System Bootstrap, Version 12.1(3r)T2, RELEASE SOFTWARE (fc1) SanFran uptime is 35 minutes System returned to ROM by reload System image file is "flash:c2600-io3-mz a.bin" cisco 2621 (MPC860) processor (revision 0x200) with 28672K/4096K bytes of memory. Processor board ID JAD051605U8 ( ) M860 processor: part number 0, mask 49 Bridging software. X.25 software, Version Ethernet/IEEE interface(s) 2 FastEthernet/IEEE interface(s) 2 Serial network interface(s) 32K bytes of non-volatile configuration memory. 8192K bytes of processor board System flash (Read/Write) Configuration register is 0x2102 SanFran# Viewing the Cisco IOS Version on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT Reviewing Cisco IOS Release Status on the Cisco Feature Navigator

© 2005 Cisco Systems, Inc. All rights reserved. CIT Finding Features by Cisco IOS Image Name on the Cisco Feature Navigator

© 2005 Cisco Systems, Inc. All rights reserved. CIT Reviewing Cisco IOS Release Status on the Cisco Feature Navigator

© 2005 Cisco Systems, Inc. All rights reserved. CIT Looking for Software Advisories for a Specific Image

© 2005 Cisco Systems, Inc. All rights reserved. CIT Reviewing the Software Advisories for a Specific Image

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show ip cache flow IP packet size distribution (53 total packets): IP Flow Switching Cache, bytes 3 active, 4093 inactive, 5 added 105 ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) Flows /Sec /Flow /Pkt /Sec /Flow /Flow UDP-other ICMP Total: SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa0/ Local Fa0/ Null Fa0/ Null SanFran# Reviewing IP Cache Flow on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT Oakland#show ip cache flow IP packet size distribution ( total packets): IP Flow Switching Cache, bytes 1049 active, 3047 inactive, added ager polls, 0 flow alloc failures last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) Flows /Sec /Flow /Pkt /Sec /Flow /Flow TCP-Telnet TCP-WWW TCP-SMTP TCP-other UDP-DNS UDP-NTP UDP-other ICMP IP-other Total: Reviewing IP Cache Flow on Oakland

© 2005 Cisco Systems, Inc. All rights reserved. CIT Oakland#show ip cache flow... SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ B 007B 1 Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Reviewing IP Cache Flow on Oakland (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Oakland#show ip cache flow | include 0800 Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Fa0/ Reviewing IP Cache Flow on Oakland (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT snmp-server enable {traps | informs} router(config)# Enables SNMP traps or informs. snmp-server community [rw | ro] {access-list number} router(config)# Configures a community string to act like a password to regulate read-write and read-only access to the agent on the router. snmp-server host router(config)# Configures the recipient of an SNMP trap operation. Commands Used to Correct Network Management Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT ntp server {ip-address} router(config)# Configures the NTP server. ntp peer {ip-address} router(config)# Configures the NTP peer. ntp source {type number} router(config)# Configures the interface for the NTP source address. Commands Used to Correct Network Management Problems (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT no snmp-server router(config)# Disables SNMP agent operation. service timestamps log datetime localtime router(config)# Configures the system to time-stamp logging messages. service timestamps debug datetime localtime router(config)# Configures the system to time-stamp debugging messages. Commands Used to Correct Network Management Problems (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ip helper-address router(config-if)# Forwards UDP broadcasts, including BOOTP, received on an interface. [no] service dhcp router(config)# Enables and disables DHCP server and relay functionality on the router. Commands Used to Correct DHCP Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT Next Animation Click for Animation Example: Correcting a TFTP Problem at the Application Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 5 > IP_ADDRESS= rommon 6 > IP_SUBNET_MASK= rommon 7 > DEFAULT_GATEWAY= monitor: command "DEFAULT_GATEWAY=" not found rommon 8 > DEFAULT_GATEWAY= rommon 9 > TFTP_SERVER= rommon 10 > TFTP_FILE=c1700-sv8y-mz YL.bin rommon 11 > Correcting a TFTP Problem at the Application Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 11 > tftpdnld IP_ADDRESS: IP_SUBNET_MASK: DEFAULT_GATEWAY: TFTP_SERVER: TFTP_FILE: flash:/c1700-sv8y-mz YL.BIN Invoke this command for disaster recovery only. WARNING: all existing data in all partitions on flash will be lost! Do you wish to continue? y/n: [n]: y Receiving c1700-sv8y-mz YL.BIN from !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! File reception completed. Copying file c1700-sv8y-mz YL.BIN to flash. Erasing flash at 0x62fe0000 Programming location rommon 12 > Invoking the TFTP Server

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 12 > boot program load complete, entry point: 0x , size: 0x98d494 Self decompressing the image : ############################################################################### ############################################################################### ########################### [OK]. Cisco Internetwork Operating System Software IOS (tm) C1700 Software (C1700-SV8Y-M), Version 12.2(8)YL, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) Synched to technology version 12.2(10.3)T1 TAC Support: Copyright (c) by cisco Systems, Inc. Compiled Wed 17-Jul-02 14:04 by ealyon Image text-base: 0x , data-base: 0x8122D408. Press RETURN to get started! Booting Up the Router to Restore the Cisco IOS Image

© 2005 Cisco Systems, Inc. All rights reserved. CIT Example: Correcting a Problem at the Application Layer Animations Done Click for Animation

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston# Dec 21 9:30:25.353: IPSEC(key_engine): request timer fired: count = 1, (identity) local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4) Dec 21 9:30:25.353: IPSEC(sa_request):, (key eng. msg.) OUTBOUND local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4), protocol= ESP, transform= esp-des esp-sha-hmac, lifedur= 3600s and kb, spi= 0x71B65BF8( ), conn_id= 0, keysize= 0, flags= 0x400C Kingston# Dec 21 9:30:55.355: IPSEC(key_engine): request timer fired: count = 2, (identity) local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4) Kingston# Dec 21 9:31:09: %CRYPTO-4-RECVD_PKT_NOT_IPSEC: Rec'd packet not an IPSEC packet. (ip) dest_addr= , src_addr= , prot= 1 Dec 21 9:31:10.753: IPSEC(sa_request):, (key eng. msg.) OUTBOUND local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4), protocol= ESP, transform= esp-des esp-sha-hmac, lifedur= 3600s and kb, spi= 0x22C15DFB( ), conn_id= 0, keysize= 0, flags= 0x400C Kingston# Reviewing Debug Output on Kingston

© 2005 Cisco Systems, Inc. All rights reserved. CIT Toronto# Dec 21 9:31:11.704: IPSEC(validate_proposal_request): proposal part #1, (key eng. msg.) INBOUND local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4), protocol= ESP, transform= esp-des esp-sha-hmac, lifedur= 0s and 0kb, spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x4 Dec 21 9:31:11.708: IPSEC(validate_transform_proposal): proxy identities not supported Dec 21 9:31:11: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Quick mode failed with peer at Toronto# Reviewing Debug Output on Toronto

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston#show crypto map Crypto Map "test" 10 ipsec-isakmp Peer = Extended IP access list 133 access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip Current peer: Security association lifetime: kilobytes/3600 seconds PFS (Y/N): N Transform sets={ auth2, } Interfaces using crypto map test: Serial1/0 Kingston# Reviewing the Crypto Map on Kingston

© 2005 Cisco Systems, Inc. All rights reserved. CIT Toronto#show crypto map Crypto Map "test" 10 ipsec-isakmp Peer = Extended IP access list 133 access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip Current peer: Security association lifetime: kilobytes/3600 seconds PFS (Y/N): N Transform sets={ auth2, } Interfaces using crypto map test: Serial1/0 Toronto# Reviewing the Crypto Map on Toronto

© 2005 Cisco Systems, Inc. All rights reserved. CIT Toronto#conf t Enter configuration commands, one per line. End with CNTL/Z. Toronto(config)#no access-list 133 Toronto(config)#access-list 133 permit ip Toronto(config)#access-list 133 permit ip Toronto(config)#access-list 133 permit ip Toronto(config)#access-list 133 permit ip Toronto(config)#access-list ip Toronto(config)#access-list ip Toronto(config)#access-list ip Toronto(config)#access-list ip Toronto(config)#exit Toronto# Toronto#show access-list 133 Extended IP access list 133 permit ip permit ip permit ip permit ip permit ip permit ip permit ip permit ip Toronto# Correcting the Crypto Map Access List on Toronto

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston_SW#ping cit_server Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:..!!! Success rate is 60 percent (3/5), round-trip min/avg/max = 72/72/72 ms Kingston_SW#ping cit_server Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 72/72/76 ms Kingston_SW# Testing Connectivity from the Kingston Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT Cisco Systems Cisco Systems TAC Internetwork Troubleshooting Handbook Cisco Systems technologies reference Support Resources for Correcting Transport and Application Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT Calling Cisco TAC Have a network diagram of your network, or affected portion of your network, ready. Make sure all IP addresses and their associated network masks or prefix lengths are listed. Have any information that you gathered thus far while troubleshooting available for the engineer. If the problem appears to be with only a few routers (fewer than four), capture the output from the show tech command on these routers.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Procedure for Correcting Transport and Application Layer Problems 1 Verify that you have a valid saved configuration for any device on which you intend to modify the configuration. 3 Evaluate and document the results of each change that you make. 4 Verify that the changes you made actually fixed the problem without introducing any new problems. 5 Continue making changes until the problem appears to be solved. 6 If necessary, get input from outside resources. 2 Make initial configuration changes. 7 Once the problem is resolved, document the solution.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Summary Troubleshooters can use the appropriate commands to make configuration changes to correct problems with TCP and UDP at the transport layer. Troubleshooters can use the appropriate commands to make configuration changes to correct problems with network management protocols at the application layer. Some transport and application layer support resources are as follows: –Cisco Systems TAC –Internetwork Troubleshooting Handbook –Cisco Systems technologies reference Following a systematic procedure increases the chances that you will successfully and effectively correct an isolated problem at the transport or application layer.

Completed Troubleshooting Logs © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! a) example shown for POD2 ! router ospf 202 area 0 authentication message-digest no area 2 authentication message-digest ! ! b) example shown for POD2 no ip access-list extended CIT ip access-list standard CIT remark Include the other pods as /16 networks permit permit permit permit permit ! ! c) ! router bgp neighbor distribute-list CIT in neighbor distribute-list CIT in Troubleshooting LogTrouble Ticket G Core Router/Switch a) The wrong area running authentication b) Bogus access-list CIT c) Mistyped access list in distribute list (ClT for CIT)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Distribution Router ! a) line con 0 EXEC ! ! b) route-map USE_FAST permit 20 set ip next-hop no set interface Serial1/1 ! !c) no ip access-list extended END_USERS ip access-list extended END_USERS remark Allow PC End Users permit ip any permit ip any ! Troubleshooting LogTrouble Ticket G Distribution Router a) Cannot connect to console (no exec) b) MISSING ICMP, telnet goes slow path c) Use of physical interface on route map

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router ! a) line con 0 speed 9600 ! need to connect via Telnet to fix ! b) no ip access-list extended Traffic ip access-list extended Traffic remark Allow ICMP, TCP outbound, FTP & WWW permit icmp any permit tcp any eq telnet permit tcp any eq ftp-data permit tcp any eq ftp permit tcp any eq www permit udp any eq tftp Troubleshooting LogTrouble Ticket G Access Router a) Cannot connect to console (line speed) b) MISSING www statement, ICMP denies END users c) SEE NEXT FIGURE

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router ! c) ip dhcp excluded-address ip dhcp excluded-address ip dhcp excluded-address no ip dhcp excluded-address no ip dhcp excluded-address no ip dhcp excluded-address ! c) DHCP does not provide addresses Troubleshooting LogTrouble Ticket G Access Router (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Switch Nothing neededNo issues Troubleshooting LogTrouble Ticket G Access Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! a) no banner motd hostname Tampa service prompt config ! b) logging console ! c) reload cancel ! d) router bgp neighbor update-source Loopback0 neighbor update-source Loopback0 vlan 27 no shut vlan 28 no shut ! Troubleshooting LogTrouble Ticket H Core Router/Switch a) Wrong banner/host name missing service prompt b) No console messages c) Reload in xxx d) Cannot reach BGP neighbors e) SEE NEXT FIGURE

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Core Router/Switch ! e) interface Vlan27 no ip ospf message-digest-key 27 md5 acme ip ospf message-digest-key 27 md5 acme ! interface Vlan28 no ip ospf message-digest-key 27 md5 acme ip ospf message-digest-key 28 md5 ACME ! Troubleshooting LogTrouble Ticket H Core Router/Switch (Cont.) e) MD5 keys messed up (extra space, wrong places)

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Distribution Router ! a) no banner motd hostname Orlando ! ! b) no prompt %%sInvalid%sinput%sdetected%s ! ! c) router eigrp 101 no eigrp stub ! ! d) router ospf 101 no distribute-list Access_Routes in distribute-list Access_Routes out ! ! e) interface serial 1/0 no mtu 64 ! interface serial 1/1 no mtu 64 ! Troubleshooting LogTrouble Ticket H Distribution Router a) Wrong banner and host name b) Wrong prompt c) No need for EIGRP stub d) Distribute list in OSPF going wrong way e) Small MTU breaks serial links for EIGRP

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router ! a) hostname Daytona ! b) no route-map USE_FAST ! route-map USE_FAST permit 10 match ip address Admin set interface Serial1/1.1 ! route-map USE_FAST deny 20 match ip address End_Users ! ! c) interface serial 1/0 no frame-relay lmi-type ansi ! interface serial 1/1 no frame-relay lmi-type ansi ! Troubleshooting LogTrouble Ticket H Access Router a) Wrong banner and host name b) Route map permit/deny swapped, sends ARP from out ser 1/1.1 c) Wrong ANSI type on frame relay d) SEE NEXT FIGURE

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Router ! d) interface serial 1/1.1 ip access-group Traffic out no ip access-group Traffic in ! interface serial 1/0.1 ip access-group Traffic out no ip access-group Traffic in ! Troubleshooting LogTrouble Ticket H Access Router (Cont.) d) Access-group applied wrong way on serial links

© 2005 Cisco Systems, Inc. All rights reserved. CIT ProblemSolution Access Switch ! a) interface Vlan901 no shut ! ! b) interface FastEthernet0/1 no switchport access vlan 2 switchport trunk native vlan 901 switchport mode trunk ! Troubleshooting LogTrouble Ticket H Access Switch a) SVI shutdown b) No trunk on VLAN 1