Cisco Internetwork Troubleshooting Isolating the Problem at the Transport and Application Layers © 2005 Cisco Systems, Inc. All rights reserved. CIT

© 2005 Cisco Systems, Inc. All rights reserved. CIT Common Symptoms of Transport Layer Problems No connectivity on the link as seen from the application layer A lack of connectivity and unreachable resources when the physical, data link, and network layers are functional Network functional but operating either consistently or intermittently at …

© 2005 Cisco Systems, Inc. All rights reserved. CIT Possible Symptoms of Transport Layer Problems Error messages supplied by the application using the transport protocol Console messagesSystem log file messagesManagement system alarmsComplaints from users that the network is slow

© 2005 Cisco Systems, Inc. All rights reserved. CIT Common Symptoms of Application Layer Problems Unreachable or unusable resources when the physical, data link, network, and transport layers are functional Operation of a network service or application does not meet the normal expectations of a user

© 2005 Cisco Systems, Inc. All rights reserved. CIT Possible Symptoms of Application Layer Problems Error messages from the afflicted applicationConsole messagesSystem log file messagesManagement system alarms Users complain that the network or the particular application that they are working with is sluggish or slower than usual

© 2005 Cisco Systems, Inc. All rights reserved. CIT netstat [-a] [-r] [-n] [-s] Windows command with options to show the routing table, connections and ports, and per-protocol statistics. nbtstat -A Displays the NetBIOS name table of a remote host at a specified IP address. C:\> Commands to Isolate Transport Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip access-lists router> Displays the contents of all IP access lists. telnet host [port] Tests the functionality of any TCP port. router> show queueing Lists queueing strategies for all or selected interfaces. router> Cisco IOS Commands to Isolate Transport Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip cache flow router> Displays a summary of the NetFlow switching statistics. show policy-map router> Displays the configuration of all classes for a specified service policy map or all classes for all existing policy maps. Cisco IOS Commands to Isolate Transport Layer Problems (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Next Animation Click for Animation Example: Isolating an Extended Access List Problem at the Transport Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW>telnet Columbia Trying Columbia ( )... Open BaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBase Columbia an ACME Distribution Workgroup Router -- Baseline -- BaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBase User Access Verification Password: Columbia> Isolating an Extended Access List Problem at the Transport Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>exit [Connection to Columbia closed by foreign host] Columbia_SW>telnet Baltimore Trying Baltimore ( )... % Destination unreachable; gateway or host down Trying Baltimore ( )... % Destination unreachable; gateway or host down Trying Baltimore ( )... % Destination unreachable; gateway or host down Trying Baltimore ( )... % Destination unreachable; gateway or host down Columbia_SW> Attempting to Telnet to Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia_SW>ping Baltimore Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 16/18/20 ms Columbia_SW> Checking Connectivity Between Columbia_SW and Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>telnet Baltimore Trying Baltimore ( )... Open BaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBase Baltimore an ACME Distribution Workgroup Router -- Baseline -- BaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBaseBase User Access Verification Password: Baltimore> Verifying Telnet Access Between Columbia and Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Baltimore>show logging Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns) Console logging: level debugging, 40 messages logged Monitor logging: level debugging, 0 messages logged Buffer logging: level debugging, 19 messages logged Logging Exception size (4096 bytes) Count and timestamp logging messages: disabled Trap logging: level informational, 46 message lines logged Logging to , 5 message lines logged Log Buffer (65536 bytes): Dec 13 06:02:25: %CONTROLLER-5-UPDOWN: Controller T1 0/1, changed state to administratively down Dec 13 06:02:27: %LINK-3-UPDOWN: Interface Serial0/0:0, changed state to up Dec 13 06:02:27: %LINK-3-UPDOWN: Interface Serial0/0:1, changed state to up Dec 13 06:02:28: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up Dec 13 06:02:28: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:1, changed state to up Dec 13 06:02:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to down Dec 13 06:02:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:1, changed state to down. Checking for Configuration Changes on Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Dec 13 06:02:56: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up Dec 13 06:02:56: %LINK-3-UPDOWN: Interface Serial1/0, changed state to up Dec 13 06:02:56: %DUAL-5-NBRCHANGE: IP-EIGRP 202: Neighbor (FastEthernet0/0) is up: new adjacency Dec 13 06:02:56: %LINK-3-UPDOWN: Interface Serial1/1, changed state to up Dec 13 06:02:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up Dec 13 06:02:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up Dec 13 06:03:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/1, changed state to up Dec 13 06:04:34: %SYS-5-CONFIG_I: Configured from console by console Dec 13 15:50:44: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:0, changed state to up Dec 13 15:50:44: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0:1, changed state to up Dec 13 15:51:34: %DUAL-5-NBRCHANGE: IP-EIGRP 202: Neighbor (Serial0/0:0) is up: new adjacency Dec 13 15:51:34: %DUAL-5-NBRCHANGE: IP-EIGRP 202: Neighbor (Serial0/0:1) is up: new adjacency Baltimore> Baltimore>show clock 14:55: EST Thu Dec Baltimore> Checking for Configuration Changes on Baltimore (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Baltimore>exit [Connection to baltimore closed by foreign host] Columbia> Columbia>show logging Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns) Console logging: level debugging, 115 messages logged Monitor logging: level debugging, 0 messages logged Buffer logging: level debugging, 155 messages logged Logging Exception size (4096 bytes) Count and timestamp logging messages: disabled Trap logging: level informational, 186 message lines logged Logging to , 139 message lines logged Log Buffer (65536 bytes): Dec 19 12:19:15: %SYS-5-CONFIG_I: Configured from console by vty0 ( ) Dec 19 13:03:06: %SYS-5-CONFIG_I: Configured from console by vty0 ( ) Dec 19 13:21:07: %SYS-5-CONFIG_I: Configured from console by vty0 ( ) Columbia> Columbia>show clock 15:53: EST Thu Dec Columbia> Checking for Configuration Changes on Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show access-lists Standard IP access list 21 permit permit permit permit permit , wildcard bits (18 matches) Standard IP access list Admin permit , wildcard bits (95 matches) permit , wildcard bits Standard IP access list END_USERS permit , wildcard bits permit , wildcard bits Extended IP access list Traffic permit icmp any any (15 matches) permit tcp any eq ftp-data permit tcp any eq ftp permit tcp any eq www permit udp any eq tftp Columbia> Checking the Access List Configuration on Columbia

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show ip route. Gateway of last resort is to network D EX /16 [170/ ] via , 2d00h, Serial0/0:0 D EX /16 [170/ ] via , 2d00h, Serial0/0: /16 is variably subnetted, 13 subnets, 2 masks D /26 [90/ ] via , 6d00h, Serial0/0:0 D /26 [90/ ] via , 6d00h, Serial0/0:0 C /26 is directly connected, Serial0/0:1 C /26 is directly connected, Serial1/1 D EX /16 [170/ ] via , 2d02h, Serial0/0:0 D /26 [90/ ] via , 6d00h, Serial0/0:0 C /26 is directly connected, FastEthernet0/0.2 C /26 is directly connected, FastEthernet0/0.3 C /26 is directly connected, FastEthernet0/0.1 C /26 is directly connected, Serial0/0:0 C /26 is directly connected, Serial1/0 C /26 is directly connected, FastEthernet0/0.4 C /26 is directly connected, Loopback0 D EX /16 [170/ ] via , 2d00h, Serial0/0:0 D EX /16 [170/ ] via , 2d00h, Serial0/0:0 D EX /16 [170/ ] via , 2d00h, Serial0/0:0 D EX /24 [170/ ] via , 2d00h, Serial0/0:0 D*EX /0 [170/ ] via , 6d00h, Serial0/0:0 Columbia> Identifying the Interface That Is Forwarding Traffic to Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia>show ip interface serial 0/0:0 Serial0/0:0 is up, line protocol is up Internet address is /26 Broadcast address is Address determined by setup command MTU is 1500 bytes Helper address is not set Directed broadcast forwarding is disabled Multicast reserved groups joined: Outgoing access list is Traffic Inbound access list is not set. Columbia>enable Verifying That the Access List is Applied to the Interface for Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT Columbia#show running-config | begin ip access-list extended Traffic ip access-list extended Traffic remark Allow ICMP, Telnet outbound, FTP & WWW permit icmp any any permit tcp any eq ftp-data permit tcp any eq ftp permit tcp any eq www permit udp any eq tftp ! logging source-interface Loopback0 logging access-list 21 permit access-list 21 permit access-list 21 remark Also allow Lenexa and Elmhurst to Telnet in access-list 21 permit access-list 21 permit access-list 21 remark Allow this workgroup to Telnet in access-list 21 permit ! route-map USE_FAST permit 10 match ip address Admin set interface Serial0/0:1 !... Columbia# Viewing the Access List in the Running Configuration

© 2005 Cisco Systems, Inc. All rights reserved. CIT Next Animation Click for Animation Example: Isolating a Problem at the Transport Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show version Cisco Internetwork Operating System Software IOS (tm) C2600 Software (C2600-IO3-M), Version 12.2(10a), RELEASE SOFTWARE (fc1) Copyright (c) by cisco Systems, Inc. Compiled Tue 21-May-02 13:57 by pwade Image text-base: 0x , data-base: 0x80A11A68 ROM: System Bootstrap, Version 12.1(3r)T2, RELEASE SOFTWARE (fc1) SanFran uptime is 28 minutes System returned to ROM by reload System image file is "flash:c2600-io3-mz a.bin" cisco 2621 (MPC860) processor (revision 0x200) with 28672K/4096K bytes of memory. Processor board ID JAD051605U8 ( ) M860 processor: part number 0, mask 49 Bridging software. X.25 software, Version Ethernet/IEEE interface(s) 2 FastEthernet/IEEE interface(s) 2 Serial network interface(s) 32K bytes of non-volatile configuration memory. 8192K bytes of processor board System flash (Read/Write) Configuration register is 0x2102 SanFran> Viewing the Router Components on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show process cpu CPU utilization for five seconds: 27%/8%; one minute: 38%; five minutes: 24% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% 0 Chunk Manager % 0.00% 0.00% 0 Load Meter % 0.19% 0.05% 66 Virtual Exec % 0.00% 0.00% 0 DHCPD Timer % 0.03% 0.00% 0 Check heaps % 0.00% 0.00% 0 Chunk Manager % 0.00% 0.00% 0 Pool Manager % 0.00% 0.00% 0 Timers % 0.00% 0.00% 0 Serial Backgroun % 0.00% 0.00% 0 ALARM_TRIGGER_SC % 0.00% 0.00% 0 Environmental mo % 0.00% 0.00% 0 ARP Input % 0.00% 0.00% 0 DDR Timers % 0.00% 0.00% 0 Dialer event % 0.00% 0.00% 0 Entity MIB API % 0.00% 0.00% 0 SERIAL A'detect % 0.00% 0.00% 0 Critical Bkgnd % 0.00% 0.00% 0 Net Background % 0.00% 0.00% 0 Logger % 0.00% 0.00% 0 TTY Background % 0.00% 0.00% 0 Per-Second Jobs % 0.00% 0.00% 0 Hawkeye Backgrou... Viewing the Process CPU on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show process cpu CPU utilization for five seconds: 27%/8%; one minute: 38%; five minutes: 24% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% 0 Net Input % 0.00% 0.00% 0 Compute load avg % 0.02% 0.00% 0 Per-minute Jobs % 0.00% 0.00% 0 Service-module a % 0.00% 0.00% 0 AAA Dictionary R % 22.55% 14.96% 0 IP Input % 0.00% 0.00% 0 CDP Protocol % 0.00% 0.00% 0 X.25 Encaps Mana % 0.00% 0.00% 0 Asy FS Helper % 0.00% 0.00% 0 PPP IP Add Route % 0.00% 0.00% 0 IP Background % 0.00% 0.00% 0 IP RIB Update % 0.00% 0.00% 0 Adj Manager % 0.00% 0.00% 0 TCP Timer % 0.00% 0.00% 0 TCP Protocols % 0.00% 0.00% 0 Probe Input % 0.00% 0.00% 0 RARP Input % 0.00% 0.00% 0 HTTP Timer % 0.00% 0.00% 0 Socket Timers % 0.00% 0.00% 0 DHCPD Receive % 0.00% 0.00% 0 IP Cache Ager % 0.00% 0.00% 0 COPS... Viewing the Process CPU on SanFran (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show process memory Total: , Used: , Free: PID TTY Allocated Freed Holding Getbufs Retbufs Process *Init* *Sched* *Dead* Chunk Manager Load Meter Virtual Exec DHCPD Timer Check heaps Chunk Manager Pool Manager Timers Serial Backgroun ALARM_TRIGGER_SC Environmental mo ARP Input DDR Timers Dialer event Entity MIB API SERIAL A'detect Critical Bkgnd Net Background Logger TTY Background Per-Second Jobs... Viewing the Process Memory on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show process memory Total: , Used: , Free: PID TTY Allocated Freed Holding Getbufs Retbufs Process Hawkeye Backgrou Net Input Compute load avg Per-minute Jobs Service-module a AAA Dictionary R IP Input CDP Protocol X.25 Encaps Mana Asy FS Helper PPP IP Add Route IP Background IP RIB Update Adj Manager TCP Timer TCP Protocols Probe Input RARP Input HTTP Timer Socket Timers DHCPD Receive IP Cache Ager... Viewing the Process Memory on SanFran (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show process cpu CPU utilization for five seconds: 44%/15%; one minute: 44%; five minutes: 36% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% 0 Chunk Manager % 0.00% 0.00% 0 Load Meter % 0.11% 0.07% 66 Virtual Exec % 0.00% 0.00% 0 DHCPD Timer % 0.03% 0.00% 0 Check heaps % 0.00% 0.00% 0 Chunk Manager % 0.00% 0.00% 0 Pool Manager % 0.00% 0.00% 0 Timers % 0.00% 0.00% 0 Serial Backgroun % 0.00% 0.00% 0 ALARM_TRIGGER_SC % 0.00% 0.00% 0 Environmental mo % 0.02% 0.01% 0 ARP Input % 0.00% 0.00% 0 DDR Timers % 0.00% 0.00% 0 Dialer event % 0.00% 0.00% 0 Entity MIB API % 0.00% 0.00% 0 SERIAL A'detect % 0.00% 0.00% 0 Critical Bkgnd % 0.00% 0.00% 0 Net Background % 0.00% 0.00% 0 Logger % 0.00% 0.00% 0 TTY Background % 0.00% 0.00% 0 Per-Second Jobs % 0.00% 0.00% 0 Hawkeye Backgrou... Rechecking the Process CPU

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show process cpu CPU utilization for five seconds: 44%/15%; one minute: 44%; five minutes: 36% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% 0 Net Input % 0.00% 0.00% 0 Compute load avg % 0.01% 0.00% 0 Per-minute Jobs % 0.00% 0.00% 0 Service-module a % 0.00% 0.00% 0 AAA Dictionary R % 26.28% 22.69% 0 IP Input % 0.00% 0.00% 0 CDP Protocol % 0.00% 0.00% 0 X.25 Encaps Mana % 0.00% 0.00% 0 Asy FS Helper % 0.00% 0.00% 0 PPP IP Add Route % 0.00% 0.00% 0 IP Background % 0.00% 0.00% 0 IP RIB Update % 0.00% 0.00% 0 Adj Manager % 0.00% 0.00% 0 TCP Timer % 0.00% 0.00% 0 TCP Protocols % 0.00% 0.00% 0 Probe Input % 0.00% 0.00% 0 RARP Input % 0.00% 0.00% 0 HTTP Timer % 0.00% 0.00% 0 Socket Timers... Rechecking the Process CPU (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#sh proc mem Total: , Used: , Free: PID TTY Allocated Freed Holding Getbufs Retbufs Process *Init* *Sched* *Dead* Chunk Manager Load Meter Virtual Exec DHCPD Timer Check heaps Chunk Manager Pool Manager Timers Serial Backgroun ALARM_TRIGGER_SC Environmental mo ARP Input DDR Timers Dialer event Entity MIB API SERIAL A'detect Critical Bkgnd Net Background Logger... Rechecking Memory Utilization

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#sh proc mem Total: , Used: , Free: PID TTY Allocated Freed Holding Getbufs Retbufs Process IP Flow Backgrou Net Input Compute load avg Per-minute Jobs Service-module a AAA Dictionary R IP Input CDP Protocol X.25 Encaps Mana Asy FS Helper PPP IP Add Route IP Background IP RIB Update Adj Manager TCP Timer TCP Protocols... Rechecking Memory Utilization (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Sending Results to Cisco Systems Output Interpreter

© 2005 Cisco Systems, Inc. All rights reserved. CIT SHOW PROCESS MEMORY NOTIFICATIONS (if any) INFO: The output of 'show process memory' only shows the memory associated with the processor and does not identify other memory such as I/O, Fast, VM, etc. To receive a statistical analysis on these types of memory, submit the first page of output from the 'show memory' command to Output Interpreter. NOTE: The types of memory vary depending on router platform and installed modules. ERROR: Processor memory utilization is %. This is considered to be very high. Processor memory or main memory stores the running configuration and routing tables. The Cisco IOS software executes from main memory. The amount of processor memory required by the router is affected by the Cisco IOS version used, the size of the network and by the access list configurations. TRY THIS: Consider using Cisco's Memory Calculator to determine the required memory for your configuration. Or use the IOS Upgrade Planner to find the required memory for the current IOS software version. Also, ensure that an optimal IOS versionhas been chosen. For example, if an image is being used that supports 'IPSEC 56'and no IPSec features are being utilized (or will be utilized), these features are not needed and a smaller image can be loaded. Memory Calculator IOS Upgrade Planner Review Results from Output Interpreter

© 2005 Cisco Systems, Inc. All rights reserved. CIT SHOW PROCESS MEMORY NOTIFICATIONS (if any) INFO:... ERROR: The following processes are currently holding more than 1 MB of memory: 'IP Input' (Holding bytes) This is considered to be high and can indicate a memory leak. INFO: A memory leak occurs when a process requests or allocates memory and then forgets to free (de- allocate) the memory when it is finished with that task. As a result, the memory block is reserved until the router is reloaded. Over time,more and more memory blocks are allocated by that process until there is no free memory available. TRY THIS: Analyze the 'show process memory' output for this router over a period of time (for example, every few hours or days depending on whether you have a fast or slow leak). Check to see if memory utilization for the affected process(es) continues to increase and the amount of freed memory remains the same. The rate at which free memory disappears depends on how often the event occurs that leads to the leak. A memory leak is a complex condition sometimes requiring an IOS upgrade to correct. If the above is in fact occurring, and you are uncertain about how to proceed, use the Case Open Tool to contact the Cisco TAC for further assistance. Case Open Tool... Review Results from Output Interpretor (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT SanFran#show ip cache flow IP packet size distribution (0 total packets): IP Flow Switching Cache, 0 bytes 0 active, 0 inactive, 0 added 0 ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) Flows /Sec /Flow /Pkt /Sec /Flow /Flow SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts SanFran# Reviewing IP Cache Flow on SanFran

© 2005 Cisco Systems, Inc. All rights reserved. CIT Oakland#show ip cache flow IP packet size distribution (0 total packets): IP Flow Switching Cache, 0 bytes 0 active, 0 inactive, 0 added 0 ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds last clearing of statistics never Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec) Flows /Sec /Flow /Pkt /Sec /Flow /Flow SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Oakland# Reviewing IP Cache Flow on Oakland

© 2005 Cisco Systems, Inc. All rights reserved. CIT show host router> Displays the default domain name, the style of name lookup service, a list of name server hosts, and the cached list of host names and addresses. cat /etc/resolv.conf Displays the identity of the name server from hosts running UNIX. terminal% nslookup {domain name} Displays the identity of the name server being used. C:\> General Commands for Isolating Application Layer Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT telnet {ip-address | host} 25 Tests SMTP protocol functionality. telnet {ip-address | host} 143 Tests IMAP protocol functionality. telnet {ip-address | host} 110 Tests POP protocol functionality. Commands Used to Isolate Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT debug snmp requests router# Displays information about every SNMP request made by the SNMP manager. debug ntp events router# Displays events related to the operation of NTP. show snmp Displays the status of SNMP communications. router> Commands Used to Isolate Network Management Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT copy tftp flash router# Tests functionality by invoking the TFTP application. telnet {ip-address | host} 21 Tests FTP protocol functionality. debug tftp Displays activity related to the operation of TFTP. router# Commands Used to Isolate File Management Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT telnet {ip-address | hostname} [/source-interface] Tests functionality of the Telnet application. debug telnet Displays events during the negotiation process of a Telnet connection. router# Commands Used to Isolate Telnet Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT show ip dhcp binding router> Displays address bindings on a DHCP server. show dhcp lease router> Shows DHCP addresses leased from a server. debug ip dhcp server [events | packets] router# Reports DHCP server events, such as address assignments and database updates, and also packet activity. Commands Used to Isolate DHCP Problems

© 2005 Cisco Systems, Inc. All rights reserved. CIT Next Animation Click for Animation Example: Isolating a TFTP Problem at the Application Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 14 > ? alias set and display aliases command boot boot up an external process break set/show/clear the breakpoint confreg configuration register utility cont continue executing a downloaded image context display the context of a loaded image cookie display contents of cookie PROM in hex dev list the device table dir list files in file system dis display instruction stream dnld serial download a program module frame print out a selected stack frame help monitor builtin command help history monitor command history meminfo main memory information repeat repeat a monitor command reset system reset set display the monitor variables stack produce a stack trace sync write monitor environment to NVRAM sysret print out info from last system return tftpdnld tftp image download unalias unset an alias unset unset a monitor variable xmodem x/ymodem image download rommon 15 > Isolating a TFTP Problem at the Application Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 15 > boot loadprog: bad file magic number: 0x0 boot: cannot load "flash:" rommon 16 > Attempting to Boot the Router in ROMMON Mode

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 16 > reset System Bootstrap, Version 12.2(4r)XL, RELEASE SOFTWARE (fc1) TAC Support: Copyright (c) 2001 by cisco Systems, Inc. C1700 platform with Kbytes of main memory loadprog: bad file magic number: 0x0 boot: cannot load "flash:" System Bootstrap, Version 12.2(4r)XL, RELEASE SOFTWARE (fc1) TAC Support: Copyright (c) 2001 by cisco Systems, Inc. C1700 platform with Kbytes of main memory loadprog: bad file magic number: 0x0 boot: cannot load "flash:" System Bootstrap, Version 12.2(4r)XL, RELEASE SOFTWARE (fc1) TAC Support: Copyright (c) 2001 by cisco Systems, Inc. C1700 platform with Kbytes of main memory rommon 1 > Attempting to Reset the Router in ROMMON Mode

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 1 > dir usage: dir rommon 2 > dir flash: File size Checksum File name 5858 bytes (0x16e2) 0x699a base.cfg rommon 2 > Attempting to Locate the Cisco IOS Image in Flash Memory

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 2 > ? alias set and display aliases command boot boot up an external process break set/show/clear the breakpoint confreg configuration register utility cont continue executing a downloaded image context display the context of a loaded image cookie display contents of cookie PROM in hex dev list the device table dir list files in file system dis display instruction stream dnld serial download a program module frame print out a selected stack frame help monitor builtin command help history monitor command history meminfo main memory information repeat repeat a monitor command reset system reset set display the monitor variables stack produce a stack trace sync write monitor environment to NVRAM sysret print out info from last system return tftpdnld tftp image download unalias unset an alias unset unset a monitor variable xmodem x/ymodem image download rommon 3 > Investigating ROMMON Commands Related to TFTP

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 3 > tftpdnld Missing or illegal ip address for variable IP_ADDRESS Illegal IP address. usage: tftpdnld [-r] Use this command for disaster recovery only to recover an image via TFTP. Monitor variables are used to set up parameters for the transfer. (Syntax: "VARIABLE_NAME=value" and use "set" to show current variables.) "ctrl-c" or "break" stops the transfer before flash erase begins. The following variables are REQUIRED to be set for tftpdnld: IP_ADDRESS: The IP address for this unit IP_SUBNET_MASK: The subnet mask for this unit DEFAULT_GATEWAY: The default gateway for this unit TFTP_SERVER: The IP address of the server to fetch from TFTP_FILE: The filename to fetch The following variables are OPTIONAL: TFTP_VERBOSE: Print setting. 0=quiet, 1=progress(default), 2=verbose TFTP_RETRY_COUNT: Retry count for ARP and TFTP (default=7) TFTP_TIMEOUT: Overall timeout of operation in seconds (default=7200) TFTP_CHECKSUM: Perform checksum test on image, 0=no, 1=yes (default=1) Command line options: -r: do not write flash, load to DRAM only and launch image rommon 4 > Attempting to Download the Cisco IOS Image from Baltimore

© 2005 Cisco Systems, Inc. All rights reserved. CIT rommon 4 > set PS1=rommon ! > TFTP_CHECKSUM=1 BOOT= BSI=0 SAVE_2_RTS=04:52:24 EST Thu Dec RET_2_RTS=13:04:41 EST Thu Dec RET_2_RUTC= ?=0 rommon 5 > Checking the Variables for the ROMMON set Command

© 2005 Cisco Systems, Inc. All rights reserved. CIT Baltimore#show running-config | include tftp tftp-server flash:c1700-sv8y-mz YL.bin Baltimore# Verifying That Baltimore Is Configured as a TFTP Server

© 2005 Cisco Systems, Inc. All rights reserved. CIT Animations Done Click for Animation Example: Isolating a Problem at the Application Layer

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston#ping cit_server Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/56 ms Kingston# Checking Connectivity from Kingston

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston_SW#ping cit_server Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:..... Success rate is 0 percent (0/5) Kingston_SW# Checking Connectivity from the Kingston Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston# Dec 21 09:28:57: %CRYPTO-4-RECVD_PKT_NOT_IPSEC: Rec'd packet not an IPSEC packet. Kingston#show crypto map Crypto Map "test" 10 ipsec-isakmp Peer = Extended IP access list 133 access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip Current peer: Security association lifetime: kilobytes/3600 seconds PFS (Y/N): N Transform sets={ auth2, } Interfaces using crypto map test: Serial1/0 Kingston# Reviewing the Crypto Map Configuration on Kingston

© 2005 Cisco Systems, Inc. All rights reserved. CIT Toronto#show crypto map Crypto Map "test" 10 ipsec-isakmp Peer = Extended IP access list 133 access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip access-list 133 permit ip Current peer: Security association lifetime: kilobytes/3600 seconds PFS (Y/N): N Transform sets={ auth2, } Interfaces using crypto map test: Serial1/0 Toronto# Reviewing the Crypto Map Configuration on Toronto

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston#show crypto ipsec sa interface: Serial1/0 Crypto map tag: test, local addr local ident (addr/mask/prot/port): ( / /0/0) remote ident (addr/mask/prot/port): ( / /0/0) current_peer: PERMIT, flags={origin_is_acl,} #pkts encaps: 0, #pkts encrypt: 0, #pkts digest 0 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: , remote crypto endpt.: path mtu 1500, media mtu 1500 current outbound spi: 0 inbound esp sas: inbound ah sas: inbound pcp sas: outbound esp sas: outbound ah sas:... Reviewing the Crypto Configuration on Kingston

© 2005 Cisco Systems, Inc. All rights reserved. CIT outbound pcp sas: local ident (addr/mask/prot/port): ( / /0/0) remote ident (addr/mask/prot/port): ( / /0/0) current_peer: PERMIT, flags={origin_is_acl,} #pkts encaps: 0, #pkts encrypt: 0, #pkts digest 0 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: , remote crypto endpt.: path mtu 1500, media mtu 1500 current outbound spi: 0 inbound esp sas: inbound ah sas: inbound pcp sas: outbound esp sas: outbound ah sas: outbound pcp sas:... Reviewing the Crypto Configuration on Kingston (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT local ident (addr/mask/prot/port): ( / /0/0) remote ident (addr/mask/prot/port): ( / /0/0) current_peer: PERMIT, flags={origin_is_acl,} #pkts encaps: 0, #pkts encrypt: 0, #pkts digest 0 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify 0 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: , remote crypto endpt.: path mtu 1500, media mtu 1500 current outbound spi: 0 inbound esp sas: inbound ah sas: inbound pcp sas: outbound esp sas: outbound ah sas: outbound pcp sas:... Reviewing the Crypto Configuration on Kingston (Cont.)

© 2005 Cisco Systems, Inc. All rights reserved. CIT Configuring Crypto IPSec Debugging Kingston#debug crypto ipsec Crypto IPSEC debugging is on Kingston# Toronto#debug crypto ipsec Crypto IPSEC debugging is on Toronto#

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston_SW#ping cit_server Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:..... Success rate is 0 percent (0/5) Kingston_SW# Testing Connectivity from the Kingston Switch

© 2005 Cisco Systems, Inc. All rights reserved. CIT Kingston# Dec 21 9:30:25.353: IPSEC(key_engine): request timer fired: count = 1, (identity) local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4) Dec 21 9:30:25.353: IPSEC(sa_request):, (key eng. msg.) OUTBOUND local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4), protocol= ESP, transform= esp-des esp-sha-hmac, lifedur= 3600s and kb, spi= 0x71B65BF8( ), conn_id= 0, keysize= 0, flags= 0x400C Kingston# Dec 21 9:30:55.355: IPSEC(key_engine): request timer fired: count = 2, (identity) local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4) Kingston# Dec 21 9:31:09: %CRYPTO-4-RECVD_PKT_NOT_IPSEC: Rec'd packet not an IPSEC packet. (ip) dest_addr= , src_addr= , prot= 1 Dec 21 9:31:10.753: IPSEC(sa_request):, (key eng. msg.) OUTBOUND local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4), protocol= ESP, transform= esp-des esp-sha-hmac, lifedur= 3600s and kb, spi= 0x22C15DFB( ), conn_id= 0, keysize= 0, flags= 0x400C Kingston# Reviewing Debug Output on Kingston

© 2005 Cisco Systems, Inc. All rights reserved. CIT Toronto# Dec 21 9:31:11.704: IPSEC(validate_proposal_request): proposal part #1, (key eng. msg.) INBOUND local= , remote= , local_proxy= / /0/0 (type=4), remote_proxy= / /0/0 (type=4), protocol= ESP, transform= esp-des esp-sha-hmac, lifedur= 0s and 0kb, spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x4 Dec 21 9:31:11.708: IPSEC(validate_transform_proposal): proxy identities not supported Dec 21 9:31:11: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Quick mode failed with peer at Toronto# Toronto#undebug all All possible debugging has been turned off Toronto# Reviewing Debug Output on Toronto

© 2005 Cisco Systems, Inc. All rights reserved. CIT Guidelines for Isolating Problems at the Transport and Application Layers First establish whether IP connectivity exists between the source and the destination. Test the sending and receiving functions separately. Check the RFCs to obtain detailed information about a malfunctioning transport layer protocol.

© 2005 Cisco Systems, Inc. All rights reserved. CIT Summary A problem at the transport layer results in a lack of connectivity and unreachable resources when the physical, data link, and network layers are functional. A problem at the application layer results in unreachable or unusable resources when the physical, data link, network, and transport layers are functional. The output of the commands at the transport layer are noteworthy because they highlight problems in the interface between the network and transport layers. The output of the commands at the transport layer are noteworthy because they highlight problems in the interface between the transport and application layers. Using an effective and systematic technique allows you to successfully isolate a problem at the transport or application layer.