1 Support Resources for Troubleshooting 1
1 Support Resources for Troubleshooting 1
Types and Puroses of Troubleshooting Tools 2
Media Test Equipment 3
Network Monitors 4
Protocol Analyzers 5
About the Contributors 5
Exercise 1-1: Using Troubleshooting Tools 6
Network Management Systems 7
Management Information Bases 8
Simple Nerwork Management Protocol 9
Acknowledgments 9
Remote Monitoring 10
Resource Manager Essentials 12
Cisco Works 12
Cisco Works for Switched Internetworks 13
NetSys 15
Exercise 1-2: Network Management 17
Cisco Connection Online 17
Infrastructure 18
Documentation 18
Marketplace 19
Software Center 19
Bug Toolkit 19
Troubleshooting Engine 20
Stack Decoder 20
Open Forum 21
Case Management Toolkit 21
Cisco Technical Assistance Center 22
Exercise 1-3: CCO 24
Two-Minute Drill 26
Q A Self Test 28
Lab Question 31
Self Test Answers 32
Preface 33
Lab Answer 35
2 Using Troubleshooting Methods 37
Introduction 37
2 Using Troubleshooting Methods 37
The Problem-Solving Model 38
Exercise 2-1: Generic Troubleshooting Lab: Yippee.net 40
Why Use a Methodology? 45
Flowchart the Troubleshooting Process 46
Defining the Problem 47
Defining the Normal Operational Baseline 52
Defining What Could Possibly Go Wrong in the Network 53
Defining Exactly What Is Wrong 54
Exercise 2-2: Problem Definition 54
Gathering Facts 57
Defining What Still Works Despite the Reported Problem 63
Comparing Findings to the Network Baseline 65
Exercise 2-3: Gathering the Facts 69
Considering Possibilities Based on Facts 73
Comparing What Still Works to What Could Go Wrogn 73
Eliminating Possible Points of Failure 73
Creating a List of Possible Trouble Areas 74
Exercise 2-4: Considering the Possibilities 75
Creating an Action Plan 76
Creating a Plan on How to Test Each Possible Problem 76
Sequencing the Possible Problems, With the Most Likely First 79
Exercise 2-5: Creating an Action Plan 81
Implementing the Action Plan 84
Executing Each Action Plan 84
Within an Action Plan, Change Only One Variable at a Time 85
Testing Each Plan to Check for Multiple Failures 86
Observing the Results 86
After Each Variable Change, Ensure That the Network Is Still Operational 87
Correcting the Individual Problem, Ensure That the Network Is Still Operational 87
After Each Action Plan, Ensure That the Network Is Still Operational 87
Exercise 2-6: Executing the Action Plan 88
Iterating the Process 90
If There Are Multiple Action Plans, Execute Each Independently 90
Exercise 2-7: Iterating the Process 91
Documenting the Facts 93
Documenting Each Change Made to the Network 94
Exercise 2-8: Documenting the Facts 94
Two-Minute Drill 96
Q A Self Test 98
Lab Question 104
Self Test Answers 105
Lab Answer 111
3 LAN Troubleshooting Problem Areas 113
3 LAN Troubleshooting Problem Areas 113
Physical Layer Responsibilities 114
Exercise 3-1: Testing Connectivity with Cable Pin-outs 117
Data-Link Layer Responsibilities 118
MAC Layer 119
Exercise 3-2: Decoding Broadcast Packets Using a Protocol Analyzer 121
Logical Link Layer 123
Ethernet/IEEE802.3 126
Exercise 3-3: Collision Increase Due to a Large File Transfer on a Shared Medium 127
CSMA/CD Protocol 128
Ethernet Statistics for Troubleshooting 130
Token Ring/IEEE 802.5 138
Understanding MAC Communications 139
Troubleshooting Token Ring Networks 143
Exercise 3-4: Determining the Type of Address in Use 146
FDDI 147
Exercise 3-5: Examining the Status of Neighbors in an FDDI Network 149
Troubleshooting LANS 150
Ethernet 152
Exercise 3-6: Identifying the Source of High Network Activity 154
Token Ring 156
Fiber Distributed Data Interface 158
Two-Minute Drill 163
Q A Self Test 165
Lab Question 170
SelfTest Answers 171
Lab Answer 175
4 Identifying Troubleshooting Targets 177
4 Identifying Troubleshooting Targets 177
Understanding Data Link Troubleshooting Targets 179
Frame Relay 179
HDLC and SDLC 180
Point-to-Point Protocol 180
Fiber Distributed Data Interface and Copper Distributed Data Interface 181
Ethernet 182
The Possible Interfaces of a Cisco Router 182
Differentiating Between LAN and WAN Interfaces 183
Troubleshooting Physical and Data Link Characteristics 184
Cable Testers 184
Network Monitors 185
Protocol Analyzers 185
Exercise 4-1: Using Network Monitor to Capture Bridge Traffic 186
Physical Checking 187
The CLEAR COUNTERS Command 187
The SHOW INTERFACES Command 188
The SHOW CONTROLLERS Command 189
The SHOW CDP NEIGHBORS Command 190
Clearing the Interface Counters 190
Defining the Interface Packet Counters 191
The CLEAR COUNTERS Command 191
Exercise 4-2: Using the CLEAR COUNTERS Command to Aid in Establishing a Baseline for Interface Counters 195
Troubleshooting Commands 196
The SHOW INTERFACES Command 196
A Basic Template for All SHOW INTERFACES Commands 197
The SHOW INTERFACES ETHERNET Command 199
The SHOW INTERFACES TOKENRING Command 204
The SHOW INTERFACES FDDI Command 206
Upstream and Downstream Neighbors 208
The SHOW INTERFACES ATM Command 209
Cisco Discovery Protocol 213
The SHOW CDP NEIGHBORS Command 213
Data-Link Layer Exercises 215
Various Data-Link Sniffer Traces 215
Two-Minute Drill 218
Q A Self Test 220
Lab Question 225
Self Test Answers 226
Lab Answer 229
5 Applying Cisco Troubleshooting Tools 231
5 Applying Cisco Troubleshooting Tools 231
Cisco IOS Software Troubleshooting Tools 232
Exercise 5-1: Assessing the Status of an Ethernet Interface 233
Understanding the Output of Diagnostic Commands 235
The SHOW INTERFACES Command 235
PING 236
The TRACEROUTE Command 237
Exercise 5-2: Understanding PING Output 240
The Routing Process 240
Routing Protocols 241
The Route Processor and the Routing Table 242
Exercise 5-3: The Routing Process 243
The Switching Process 243
Exercise 5-4: The Switching Process 245
Switching Initialization 245
Process Switching 245
Fast Switching 246
Optimum Switching 247
Other Switching Methods 247
Exercise 5-5: Switching Processes 248
Tracing Packet Flow 249
Exercise 5-6: Silicon Switching 250
What the Route Processor Handles Directly 251
Exercise 5-7: Process Switching 252
The DEBUG Utility 252
DEBUG Performance Considerations 253
Service Time Stamps 255
Exercise 5-8: Debug Restrictions 256
Error Message Logging 256
Logging Levels 257
Console 258
Virtual Terminal 259
Buffered 259
The SHOW LOGGING Command 260
Syslog 260
Exercise 5-9: Error Message Logging 262
The SHOW VERSIONG Command 262
Exercise 5-10: Using the SHOW VERSION Command 264
Buffers and Queues 265
I/O Buffer Error Conditions 265
The SHOW BUFFERS Command 267
The SHOW MEMORY Command 269
The SHOW PROCESSES Command 269
The SHOW CONTROLLERS CXBUS Command 271
The SHOW STACKS Command 272
Core Dumps 273
Exercise 5-11: Buffer Errors 274
Two-Minute Drill 275
Q A Self Test 277
Lab Question 282
Self Test Answers 283
Lab Answer 286
6 Common Routing Protocols 287
6 Common Routing Protocols 287
Periodic Broadcast-Based Protocols 288
Routing Information Protocol 289
Troubleshooting RIP 294
Cisco s Interior Gateway Routing Protocol 298
Troubleshooting IGRP 300
IPX RIP and Service Advertisement Protocol 302
Troubleshooting IPX RIP 303
Apple Talk Routing Table Maintenance Protocol 309
Troubleshooting Apple Talk Networks 310
Event-Driven Protocols 314
Open Shortest Path First 315
Troubleshooting OSPF 318
Cisco s Enhanced IGRP 323
Troubleshooting EIGRP 324
Exercise 6-1: Configuring EIGRP 333
Border Gateway Protocol 334
Troubleshooting BGP 337
Two-Minute Drill 343
Q A Self Test 346
Lab Question 350
Self Test Answers 351
Lab Answer 353
7 Diagnosing and Correcting IP Problems 355
7 Diagnosing and Correcting IP Problems 355
TCP Connection Sequence 358
Address Resolution Protocol 360
Routing Protocol Traffic 360
Three-Way Handshake 362
Cisco IOS Software Tools and Commands 363
Exercise 7-1: Watching ARP in Operation 366
The CLEAR ARP-CACHE Command 367
The SHOW IP INTERFACE Command 367
The SHOW IP PROTOCOLS Command 370
The SHOW IP ROUTE Command 372
The SHOW IP TRAFFIC Command 373
The SHOW IP ACCESS-LISTS Command 374
The SHOW IP OSPF Command 375
The SHOW IP OSPF DATABASE Command 377
The SHOW IP OSPF INTERFACE Command 379
The SHOW IP OSPF NEIGHBOR Command 381
The SHOW IP OSPF VIRTUAL-LINKS Command 382
The SHOW IP EIGRP INTERFACES Command 382
The SHOW IP EIGRP NEIGHBORS Command 383
The SHOW IP EIGRP TRAFFIC Command 384
The SHOW IP EIGRP TOPOLOGY Command 386
The DEBUG IP RIP Command 387
The DEBUG IP IGRP EVENTS Command 388
The DEBUG IP IGRP TRANSACTIONS Command 389
The DEBUG IP EIGRP Command 390
The DEBUG ARP Command 390
The DEBUG IP RACKET
The DEBUG IP OSPF EVENTS Command 392
The DEBUG IP OSPF ADJACENCY Command 392
The DEBUG IP OSPF PACKET Command 393
PING and TRACE 393
The Standard IP PING Command 394
The Extended IP PING Command 395
Exercise 7-2: Using Extended PING 396
The STANDARD IP TRACE Command 397
The Extended IP TRACE Command 398
Problem Isolation in IP Networks 399
The Local Host 401
The Local Gateway 402
Remote Routers 405
Remote Hosts 409
Problem Isolation in Windows 95 and 98 Networks 410
The Local Host 412
Exercise 7-3: Isolating NIC Failures in Windows 95 and 98 412
The Local Gateway 416
IP Configuration 420
NT Server 421
IP Symptoms, Problems, and Action Plans 424
Host Cannot Access Other Hosts on Different Subnets 424
Host Cannot Access Other Subnets 425
Host Can Access Some Hosts But Not Others 426
Host Cannot Access Remote Hosts via Redundant Path 427
Host Can Access Some Services But Not Others 427
Router Sees Duplicate Routing Updates 428
Certain Protocols Routed While Others Are Not 428
Host/Router Cannot Reach Certain Subnets 429
No Routing with Redistribution 430
Windows 95 and 98 Symptoms, Problems, and Action Plans 431
Host Unable to Reach Target Address 431
Data-Link and Network Layer Exercises 432
Various Data Link and IP Network Layer Sniffer Traces 433
Two-Minute Drill 440
Q A Self Test 442
Lab Question 447
Self Test Answers 448
Lab Answer 451
8 Diagnosing and Correcting IPX Problems 453
8 Diagnosing and Correcting IPX Problems 453
IPX Connection Sequence 455
Routing Protocol Traffic 455
Service Advertisement Traffic 459
Exercise 8-1: SAP Identification 461
Get Nearest Server 462
NCP Request/Response 462
Cisco IOS Software Tools and Commands 463
The SHOW IPX INTERFACE Command 465
The SHOW IPX ROUTE Command 466
The SHOW IPX TRAFFIC Command 467
The SHOW IPX ACCESS-LIST Command 468
The SHOW IPX NLSP DATABASE Command 470
The SHOW IPX NLSP NEIGHBORS Command 472
The SHOW IPX EIGRP INTERFACES Command 473
The SHOW IPX EIGRP NEIGHBORS Command 474
The SHOW IPX EIGRP TRAFFIC Command 475
The SHOW IPX EIGRP TOPOLOGY Command 475
The SHOW IPX SERVERS Command 476
The DEBUG IPX ROUTING Command 477
The DEBUG IPX EIGRP Command 479
The DEBUG IPX EIGRP EVENTS Command 480
The DEBUG IPX EIGRP NEIGHBOR Command 481
The DEBUG IPX PACKET Command 482
The DEBUG IPX SAP Command 483
Exercise 8-2: SAP DEBUGs 485
PING 487
The Standard IPX PING Command 487
The DEBUG IPX NLSP EVENTS Command 487
Exercise 8-3: IPX PING 489
The EXTENDED IPX PING Command 489
The IPX TRACEROUTE Command 490
Problem Isolation in IPX Networks 491
The Client 491
Bridges and Switches 493
Case Study: Switch Port Failure 495
The Local Gateway 496
Case Study: Token Ring Port Failure 497
Case Study: Ethernet Port Failure 498
Remote Routers 500
Exercise 8-4: Timer Mismatch 505
The Server 506
IPX Symptoms, Problems, and Action Plans 509
Client Cannot Access Local Server 510
Client Cannot Access Remote Server 511
Exercise 8-5: Duplicate Network Problem 515
SAP Updates Not Propagated by the Router 520
Data Link and Network Layer Traces 531
Client Broadcast GNS Request 532
Server GNS Response 532
Client RIP Route Request 533
Server RIP Route Response 533
Client NCP Request 533
Client NCP Connection Request 534
Server NCP Response 534
Exercise 8-6: Protocol Analyzer 535
Two-Minute Drill 537
Q A Self Test 541
Lab Question 546
Self Test Answers 548
Lab Answer 553
9 Diagnosing and Correcting Apple Talk Problems 555
9 Diagnosing and Correcting AppleTalk Problems 555
LocalTalk 557
The Evolution of Apple Talk 557
Bootup and Address Assignment 558
Apple Talk Connection Sequence 559
Apple Talk Phase II 561
Apple Talk Address Resolution Protocol 561
Routing Protocol Traffic 562
ZIP Queries and Replies 565
GETZONELIST Request and Replies 566
NBP Broadcast Requests and Propagation 567
NBP Lookup and Response 568
ATP Request and Reply 569
Cisco IOS Software Tools and Commands 569
The SHOW APPLETALK INTERFACE Command 570
The SHOW APPLETALK ROUTE Command 570
The SHOW APPLETALK ZONE Command 571
The SHOW APPLETALK TRAFFIC Command 572
The SHOW APPLETALK ACCESS-LIST Command 573
The SHOW APPLETALK ARP Command 573
The SHOW APPLETALK GLOBALS Command 574
The SHOW APPLETALK NAME-CACHE Command 574
The SHOW APPLETALK NBP Command 574
The SHOW APPLETALK EIGRP INTERFACES Command 575
The SHOW APPLETALK EIGRP NEIGHBORS Command 575
The SHOW APPLETALK EIGRP TOPOLOGY Command 575
The DEBUG APPLETALK ROUTING Command 576
The DEBUG APPLETALK NBP Command 576
The DEBUG APPLETALK ZIP Command 576
The DEBUG APPLETALK PACKET Command 577
The DEBUG APPLETALK ARP Command 577
The DEBUG APPLETALK ERRORS Command 577
The DEBUG APPLETALK EVENTS Command 578
PING and TRACE 578
The Standard AppleTalk PING Command 579
The Extended AppleTalk PING Command 579
The Standard AppleTalk TRACE Command 579
The Extended AppleTalk TRACE Command 580
AppleTalk Symptoms, Problems, and Action Plans 580
Host Cannot See Zones or Remote Services 581
Services Are Not Available Outside the Local Network 582
Zones Are Missing from Chooser 583
Services Are Not Always Available 584
Services Visible But Not Usable 586
Zone List Changes Each Time Chooser Is Opened 587
Connection to Services Drops 587
Router Ports Stuck Restarting or Acquiring 588
Old Zone Names (Ghost Zones) Appear in Chooser 589
Two-Minute Drill 591
Q A Self Test 593
Lab Question 599
Self Test Answers 600
Lab Answer 603
10 Diagnosing and Correcting Catalyst Problems 605
10 Diagnosing and Correcting Catalyst Problems 605
The Catalyst 5000 Series 607
Catalyst Light-Emitting Diodes 607
The Catalyst 6000 Series 610
Exercisse 10-1: Matching LED Color with the Correct Meaning 614
Cisco Works 2000 615
Cable and Media Distance Limitations 616
Copper Cabling 616
Optical Networking 617
Exercise 10-2: Cable Media Types and Segment Distances 619
Catalyst Power-On Self Test 620
System Tests 620
The SHOW TEST Command 627
Summary 627
Exercise 10-3: Command-Line Physical Diagnostic Commands 627
Level 2 Troubleshooting 628
Spanning Tree 628
Loop Avoidance 629
Election of the Root Bridge 629
Root Ports and Designated Ports 630
Port States 630
Exercise 10-4: Spanning-Tree Diagnosis 636
Inter-Switch Link and Frame Tagging 637
Internal Frame Tag and ISL 638
ISL Frame Format 638
VLAN Trunking Protocol and Propagation 641
VTP Propagates the VLAN Table 641
Cisco Discovery Protocol 646
CDP Advertises Network Layer Information 647
Diagnostic Commands 648
The SHOW SYSTEM Command 649
The SHOW INTERFACE Command 649
The SHOW LOG Command 649
The SHOW CAM DYNAMIC Command 649
The SHOW MAC Command 649
The SHOW MODULE Command 650
The SHOW PORT Command 650
The SHOW CONFIG Command 650
The SHOW SPAN Command 650
The SHOW SPANTREE Commadn 650
The SHOW TRUNK Command 651
The SHOW VLAN Command 651
The SHOW FLASH Command 651
The SHOW VTP DOMAIN Command 651
The SHOW CDP NEIGHBOR Command 652
Exercise 10-5: Diagnosing Trunking Issues 652
Configuration 653
The Switched Port Analyzer 653
Catalyst Problem Isolation 654
LDEs 654
Switch Configuration 655
Physical Connections 655
Layer 2 Path Between Switches 655
Catalyst Symptoms, Problems, and Action Plans 656
VLAN Configuration 656
Switch Cannot Communicate with Local Devices 657
Terminal Cannot Communicate with Switch 658
Server Cannot Communicate with Remote Devices on Another LAN 659
VTP Isn t Propagating VLAN Information 660
Two-Minute Drill 662
Q A SelfTest 665
11 Troubleshooting VLANs on Routers and Switches 667
Lab Question 670
SelfTest Answers 671
Lab Answer 675
11 Troubleshooting VLANs on Routers and Switches 677
Inter-VLAN Routing 678
Exercise 11-1: Modern Network Design 681
Router and Switch Configuration 682
The Router on a Stick Configuration with Subinterfaces 683
Switch Configuration in Talking to a Router 684
RSM Configuration 684
Configuring Bridging Between VLANs 686
Exercise 11-2: Inter-VLAN Routing Configuration 687
VLAN Design Issues 688
STP Timers 688
Network Diameter 689
STP in Routed VLANs 690
Default VLANs 691
Exercise 11-3: Placement of the Root Bridge 692
VLAN Troubleshooting 693
Troubleshooting Strategies 694
Exercise 11-4: Troubleshooting VLANs 695
Router VLAN Diagnostic Tools 696
The SHOW VLANS Command 696
The SHOW SPAN Command 697
The SHOW BRIDGE Command 698
The SHOW INTERFACE Command 699
The SHOW ARP Command 700
The SHOW IP CACHE Command 700
The DEBUG VLAN PACKET Command 701
Exercise 11-5: Using Troubleshooting Tools 702
Router and Switch Problem Isolation 704
Physical Layer 704
Switch Configurations 704
Router Configuration 705
Exercise 11-6: Troubleshooting an RSM 706
Router VLAN Symptoms, Problems, and Action Plans 707
Slow or Unreliable Performance Within a VLAN 707
Flooded Network: Spanning-Tree Loop 708
Local Devices Cannot Communicate to Devices Beyond the Router 710
Exercise 11-7: Examining a Catalyst 6000 Series Switch 711
Two-Minute Drill 713
Q A SelfTest 715
Lab Exercise 720
Self Test Answers 722
Lab Answer 724
12 Diagnosing and Correcting Frame Relay Problems 725
12 Diagnosing and Correcting Frame Relay Problems 725
Frame Relay Diagnostic Tools 726
The SHOW CONTROLLERS SERIAL Command 728
The SHOW INTERFACES SERIAL Command 729
The CLEAR COUNTERS SERIAL Command 731
Exercise 12-1: Clear Counters 731
The SHOW INTERFACES SERIAL Command 732
The SHOW FRAME-RELAY LMI Command 737
Exercise 12-2: Manually Set LMI Type 740
The SHOW FRAME-RELAY MAP Command 740
The SHOW FRAME-RELAY PVC Command 742
Loopback Tests 743
Exercise 12-3: Loopback Testing 744
The DEBUG SERIAL INTERFACE Command 746
The DEBUG FRAME-RELAY LMI Command 748
The DEBUG FRAME-RELAY EVENTS Command 749
The DEBUG FRAME-RELAY PACKET Command 751
Problem Isolation in Frame Relay WANs 752
Exercise: 12-4: Checking That the Correct DLCI Is Assigned 755
Frame Relay Symptoms, Problems, and Action Plans 756
Exercise 12-5: Checking the Serial Interface for Input Errors 759
Two-Minute Drill 764
Q A SelfTest 765
Lab Question 771
SekfTest Answers 773
Lab Answer 776
13 Diagnosing and Correcting ISDN Problems 779
13 Diagnosing and Correcting ISDN Problems 779
BRI Overview 780
2B+D=3DSOs 781
ISDN and the OSI Model 782
Differences Between B Channels and the D Channel 782
Exercise 13-1: Troubleshooting Basic ISDN Connectivity 783
ISDN Reference Points 784
The ISDN S/T Interface 787
Exercise 13-2: ISDN Reference Points 787
Exercise 13-3: ISDN Cabling 789
ISDN S/T Line Framing 789
S/T Layer 1 Activation 791
Exercise 13-4: Troubleshooting Layer 1 Problems 791
ISDN Q.931 Call Setup 792
Q.931 Message Format 793
Exercise 13-5: Troubleshooting ISDN Calls 794
PPP and Multilink PPP 795
PPP Multilink 795
Exercise 13-6: Identifying PPP Protocols 796
ISDN BRI Diagnostic Tools 796
The SHOW INTERFACES BRI O Command 797
The SHOW INTERFACES BRI O 1 Command 798
The SHOW INTERFACES BRI O 2 Command 799
The SHOW CONTROLLER BRI O Command 800
The SHOW ISDN STATUS Command 801
The SHOW DIALER Command 802
The SHOW PPP MULTILINK Command 803
The DEBUG ISDN Q921 Command 803
The DEBUG DIALER Command 805
The DEBUG ISDN Q931 Command 806
The DEBUG BRI Command 809
The DEBUG PPP NEGOTIATION Command 809
The DEBUG PPP AUTHENTICATION Command 810
The DEBUG IP PACKET Command 812
Exercise 13-7: Troubleshooting ISDN Activation 813
Problem Isolation in ISDN BRI WANs 814
Physical Layer 814
ISDN Layer3 815
ISDN Layer2 815
DDR Configurations 816
PPP and Authentication 816
Exercise 13-18: Troubleshooting with DEBUG BRI 817
ISDN Symptoms, Problems, and Action Plans 817
ISDN Router Does Not Dial 818
Users Cannot Connect 819
ISDN Call from Router Does Not Go Through 820
ISDN Router Cannot Ping the Remote Side 820
Second ISDN B Channel Does Not Come Up 820
Exercise 13-9: Troubleshooting Disconnects 822
Two-Minute Drill 824
Q A SelfTest 827
Lab Question 832
SelfTest Answers 834
Lab Answer 836
A About the CD 839
A About the CD 839
Cert Trainer 840
System Requirements 840
ExamSim 840
Installing CertTrainer 840
Saving Scores as Cookies 841
E-Book 842
Help 842
Upgrading 843
Glossary 845
Glossary 845
Index 887
Index 887