Thursday, January 8, 2009

Grrr...Solaris 10 install

I have generally always HATED installing operating systems. Unfortunately today caught me doing just that. Installing Solaris 10 on a v490 that just refused to cooperate with me. So there I was huddled in the data center at temperatures my body doesn't agree with; an installation that took a minimum 2 hours (had to do it twice - resized the disks wrongly the first time) so 4 hours....grrr.....a fun fun......If I curse or throw a shoe at you in traffic blame SUN:-) or solaris...and forgive me!!

and it still bloody refused to boot in the end. damn thing couldnt initialize the boot disk without my help , kumbe the boot sequence was wrong:-(

Retrying network initialization
Evaluating:

Can't open boot device

{3} ok boot disk <<---this resolved my first problem, finally booting - 4 hrs down the line
Boot device: /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@0,0 File and args: Loading ufs-file-system package 1.4 04 Aug 1995 13:02:54. FCode UFS Reader 1.12 00/07/17 15:48:16. Loading: /platform/SUNW,Sun-Fire-V490/ufsboot Loading: /platform/sun4u/ufsboot

then came installing squid, which is what i wanted to do to begin with. If you've installed solaris you know how bare minimal instal gets done. So we always keep a set of standard packages to install to get things going:

## pwd
/export/home/jgitau/SUNFREEWARE
# ls
apache-2.2.6-sol10-sparc-local libintl-3.4.0-sol10-sparc-local php-5.2.5-sol10-sparc-local
curl-7.18.0-sol10-sparc-local libxml2-2.6.31-sol10-sparc-local php-5.2.6-sol10-sparc-local
expat-2.0.1-sol10-sparc-local make-3.81-sol10-sparc-local python-2.5.1-sol10-sparc-local
flex-2.5.33-sol10-sparc-local mysql-5.0.51-sol10-sparc-local sed-4.1.5-sol10-sparc-local
fping-2.4b2-sol10-sparc-local ncurses-5.6-sol10-sparc-local thttpd-2.25b-sol10-sparc-local
gcc-3.4.6-sol10-sparc-local netsnmp-5.4.1-sol10-sparc-local wget-1.11.1-sol10-sparc-local
libiconv-1.11-sol10-sparc-local openldap-2.4.11-sol10-sparc-local zip-2.32-sol10-sparc-local
libidn-1.6-sol10-sparc-local openssl-0.9.8f-sol10-sparc-local zlib-1.2.3-sol10-sparc-local
so we keep those around and use the standard pkgadd'er:

pkgadd -d

so how long do you think that sort of took?....

oh yes I also had missing paths:
so:

Bash:
export PATH=/usr/bin:/usr/local/bin:usr/sbin

sh:
setenv PATH /usr/bin:/usr/local/bin:usr/sbin


adding me as a user so we can ssh and leave this cold place...:

bash-3.00# mkdir /export/home/jgitau
bash-3.00# useradd -d /home/jgitau -m -s /usr/bin/bash -c "John Gitau" jgitau
bash-3.00# passwd jgitau
New Password:
Re-enter new Password:
passwd: password successfully changed for jgitau

scp/ftp the files or use whichever method makes you happy....eg for apache

scp apache-2.2.6-sol10-sparc-local jgitau@196.201.xx.xx:/export/home/jgitau/SUNFREEWARE

aaanywayz....i finally got to compile squid and it was a happy day, just not a single ccie sort of reading will get done today...

Ha...yes this post is mainly just to remind me what packages to get downloaded next time, and simple commands I sometimes forget...and general noise for the blog...:-)...

seriously though, you can let me know if you do get a problem with this sort of thing, live around nairobi, and you promise not to take me to a data center with working air conditioners - because then it would be a nice warm environment-) for me:-) hmm I didn't see any DustPuppies, I relax over at userfriendly alot..probably too much

Tuesday, January 6, 2009

Understanding OSPF....

After going through ospf, and to properly understand some concepts, I hit the intanet looking for pre-done dynamips labs:-) to save on time. during my cyber walk, I came across the article
Configuring Basic OSPF (Dynamips). It was excellent and had a very well detailed instruction on things.

However one of the requirements for the lab was:

That the ISP router always be the DR, one of the Remote sites always be a BDR and one remote site never participates in the election process. (Please go to evilrouters.net) for the diagram and full article - but it looks something like below all routers connected through switches.

Remote1===\
======ISP
Remote2===/

I redid the whole lab with emphasis on meeting the designated router (DR)/backup designated router (BDR) requirements
Configurations:

ISP#sh ip int br | exclude un
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 10.10.10.1 YES manual up up
Loopback0 188.46.37.254 YES manual up up

------
Remote1#sh ip int br | ex una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 10.10.10.3 YES manual up up
Loopback0 192.168.1.1 YES manual up up
Loopback1 192.168.1.161 YES manual up up
-------
Remote2#sh ip int br | ex una
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 10.10.10.2 YES manual up up
Loopback0 192.168.1.65 YES manual up up
Loopback1 192.168.1.129 YES manual up up

After bringing up the interfaces, I tested connectivity as evilrouter dude had done it - well almost:-):

ISP#tclsh
ISP(tcl)#for
ISP(tcl)#forea
ISP(tcl)#foreach address {
+>10.10.10.2
+>10.10.10.3
+>} {ping $address}

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 32/58/96 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.3, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 16/45/88 ms
ISP(tcl)#

Now we get OSPF going, just as he had done it, I'll just show the neighbors here:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
192.168.1.129 1 FULL/BDR 00:00:34 10.10.10.2 FastEthernet0/0
192.168.1.161 1 FULL/DROTHER 00:00:38 10.10.10.3 FastEthernet0/0
------
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
188.46.37.254 1 FULL/DR 00:00:31 10.10.10.1 FastEthernet0/0
192.168.1.129 1 FULL/BDR 00:00:39 10.10.10.2 FastEthernet0/0
-----
Remote2#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
188.46.37.254 1 FULL/DR 00:00:39 10.10.10.1 FastEthernet0/0
192.168.1.161 1 FULL/DROTHER 00:00:31 10.10.10.3 FastEthernet0/0

-we have proper basic ospf

Since Im interested in the DR/BDR, I'll skip the details here since the original
post
at evilrouters.net still has them.

lets go over the requirements again in proper detail for clarity:
1: ISP - 10.10.10.1 will always be the DR
2: Remote1 - 10.10.10.3 will always be the BDR
3: remote2 - never participates in an election.
Remote2 is easy:

Remote2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Remote2(config)#interface fastethernet 0/0
Remote2(config-if)#ip ospf priority 0
Remote2(config-if)#


ISP is also easy:

ISP(config)#interface fastethernet 0/0
ISP(config-if)#ip ospf priority ?
<0-255> Priority
ISP(config-if)#ip ospf priority 255

The neighbors currently look like this:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
ISP#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:32 10.10.10.2 FastEthernet0/0
10.10.10.3 1 FULL/DR 00:00:35 10.10.10.3 FastEthernet0/0
-------
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/BDR 00:00:36 10.10.10.1 FastEthernet0/0
10.10.10.2 0 FULL/DROTHER 00:00:38 10.10.10.2 FastEthernet0/0
-------
Remote2#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/BDR 00:00:33 10.10.10.1 FastEthernet0/0
10.10.10.3 1 FULL/DR 00:00:38 10.10.10.3 FastEthernet0/0


so according to the above output:

10.10.10.2 - is fine, it won't participate in the election process.

Note:
Immediately something is wrong with the BDR/DR election . Since I reset the Remote2 OSPF process first, it got to be DR first and the ISP router had to be the BDR. That I'm sure starts to make my point but, just to ensure we go through all the evil router guy's/guyette's??:-) steps: I'll go ahead and change the priority of the RemoteRouter to 254 and clear the ISP router's ospf process first - !so it gets to be DR, followed by the others and the output is as follows:

ISP#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:30 10.10.10.2 FastEthernet0/0
10.10.10.3 254 FULL/BDR 00:00:34 10.10.10.3 FastEthernet0/0
-------
Remote1#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/DR 00:00:32 10.10.10.1 FastEthernet0/0
10.10.10.2 0 FULL/DROTHER 00:00:34 10.10.10.2 FastEthernet0/0
-------
Remote2#sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 255 FULL/DR 00:00:30 10.10.10.1 FastEthernet0/0
10.10.10.3 254 FULL/BDR 00:00:36 10.10.10.3 FastEthernet0/0

However, If I now reset the ISP router's ospf interface, or if the link to the ISP were to go off:

Remote1#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:38 10.10.10.2 FastEthernet0/0

Remote2#
*Mar 1 01:06:44.087: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.10.1 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expiredsh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.3 254 FULL/DR 00:00:39 10.10.10.3 FastEthernet0/0

So now we have a situation where the former BDR was upgraded to DR. The reason I think your requirement for the ISP to always be a BDR cant be met with
this configuration is so long as there's a BDR that gets upgraded to a DR and no pre-emption happens, then your ISP will always be either a DR or BDR even
under normal circumstances. The priority only kicks in if the processes are reset at the same time.

Here's the output after bringing up the ISP router.


ISP(config-if)#
*Mar 1 01:12:17.647: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar 1 01:12:18.647: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
*Mar 1 01:12:25.631: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.10.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
*Mar 1 01:12:25.719: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.10.3 on FastEthernet0/0 from LOADING to FULL, Loading Done
ISP(config-if)#do sh ip ospf ne

Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/DROTHER 00:00:37 10.10.10.2 FastEthernet0/0
10.10.10.3 254 FULL/DR 00:00:34 10.10.10.3 FastEthernet0/0

ISP#sh ip ospf interface f0/0
FastEthernet0/0 is up, line protocol is up
Internet Address 10.10.10.1/29, Area 0
Process ID 1, Router ID 10.10.10.1, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State BDR, Priority 255
Designated Router (ID) 10.10.10.3, Interface address 10.10.10.3
Backup Designated router (ID) 10.10.10.1, Interface address 10.10.10.1
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:05
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 2/2, flood queue length 0
Next 0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 1
Last flood scan time is 0 msec, maximum is 4 msec
Neighbor Count is 2, Adjacent neighbor count is 2
Adjacent with neighbor 10.10.10.2
Adjacent with neighbor 10.10.10.3 (Designated Router) !<<<------ note the DR!
Suppress hello for 0 neighbor(s)

So the question is, how do you ensure that a router is always a BDR? Is that even possible? How to go about it?

Lastly, many thanks to whoever runs http://evilrouters.net for this article

Sunday, January 4, 2009

confused laziness.....and expect:-)

...today (sunday) caught me trying to catch up with OSPF but I just couldn't stand the books, sites - and a friend who'd come for an apparently 'intelligent discussion on nssa's and routers, or sgsn's etc etc', in the end we opened a couple of beers, fed the fish (the only other living thing in my house is 4 fish) and yapped about current affairs, state of african economies and how soon we're going to be travelling all over the place setting up networks sampling local brews:-) - Please note kenya and the general (east) african region has relied on satellite communication for so long, however we have submarine fiber coming in by 2010-2012 so yes I see a very bright future for networkers:-).....so no bootcamps, no authorized training centers for high end certs, no internetwork expert, no narbik...to get certified here you really have to lab up and put in the time...ahh the joys.

Anyway after several hours of that 'drinking', the neurons were perfectly fired up to engage in some creative boredom activity. I thought of one of the most annoying repetitive tasks my team has to undertake daily and decided to get rid of it.

Every once in a while we get customers calling our 1st line support guys just to check if their remote sites have established a pdp session successfully. On a cisco GGSN the command would be:

show gprs gtp pdp-context msisdn 2547221x1x2xx

To get the following:

TID MS Addr Source SGSN Addr MSISDN APN
36xx72xxxxxxxxx 196.xx.xx.01 LOCAL 196.20x.xxx.xx 2547221x1x2xx safaricom
In most cases the next request could be to clear the pdp context so they can re-establish it again

clear gprs gtp pdp-context tid
So in comes EXPECT:
this is an extraordinarilly great tool for writing scripts for the lazy sysadnmin to drive other programs. It recognizes prompts and sends keystrokes in response. It was written by Don Libes of NIST, and you can find papers on it in Usenix LISA (Large Systems Administration) conference proceedings, as well as on the Internet.

My drunken goal was to create a web interface where the users can check this for themselves.

The expect script itself was pretty straight forward:


#!/usr/local/bin/expect -f

set force_conservative 0 ;# set to 1 to force conservative mode even if
;# script wasn't run conservatively originally
if {$force_conservative} {
set send_slow {1 .1}
proc send {ignore arg} {
sleep .1
exp_send -s -- $arg
}
}


set msisdn [lindex $argv 0]
set timeout 3
spawn /usr/bin/bash
match_max 100000
send -- "telnet 196.x.x.x\r" # your GGSN/cisco routers IP address
expect -exact "telnet 196.x.x.x\r # your GGSN/cisco routers IP address
Trying 196.x.x.x...\r
Connected to 196.x.x.x.\r
Escape character is '^\]'.\r
\r
\r
User Access Verification\r
\r
Username: "
send -- "drunkenmaster\r"
expect -exact "drunkenmaster\r
Password: "
send -- "jedimaster\r"
expect -exact "\r
\r
GGSN-xx01>"
send -- "en\r"
expect -exact "en\r
Password: "
send -- "jedimaster\r"
expect -exact "\r
GGSN-xx01#"
send -- "show gprs gtp pdp-context msisdn $msisdn"
expect -exact "show gprs gtp pdp-context msisdn $msisdn"
send -- "\r"
send -- "exit\r"
send -- ""
expect eof


Next do a simple perl,php or whatever makes you happy script, put it on a web server somewhere and guys can access it with a url. The handiwork can be viewed here:
* i just modified an old script we use for users to change unix passwords still using expect and adapted it for my needs.


Please note the above url is only active as a demo - in any case it would only be applicable to our users but im sure it explains the concept

On putting in a valid msisdn the output should show up in full....as follows - or you can use the url above:
Please check from LINE 17 (seventeen) thats where your output will be

GGSN returned following information:
Array
(
[0] => spawn /bin/bash
[1] => telnet 196.x.x.x
[2] => www-data@monitor01:/var/www$ telnet 196.x.x.x
[3] => Trying 196.x.x.x...
[4] => Connected to 196.x.x.x.
[5] => Escape character is '\^]'.
[6] =>
[7] =>
[8] => User Access Verification
[9] => [10] => Username: drunkenmaster
[11] => Password:
[12] =>
[13] => GGSN-xx01>en
[14] => Password:
[15] => GGSN-xx01#show gprs gtp pdp-context msisdn
[16] => TID MS Addr Source SGSN Addr MSISDN APN
[17] => 36xx72xxxxxxxxx 196.xx.xx.01 LOCAL 196.20x.xxx.xx 2547221x1x2xx safaricom
[18] =>
[19] => GGSN-xx01#exit
[20] => Connection closed by foreign host. [
21] => www-data@gitaus-TestServer:/var/www$

please note this is a sample output and no script is actually run to get this output

Please contact the data team! To run another query, : Please click me to run another query! "


Ahh finally success, the php script just inserts the msisdn....if you're interested in it, leave a comment and I'll probably just upload it...

You can run this for virtually anything, show bgp neighbors, for users to change unix passwordsetc etc..just modify the commands as you see fit. You can for instance give your users access a router to check the status of one of their interfaces, or clear statistics or whatever makes you happy, its a bonus if you have the security guys screaming at you:-) no seriously ensure your corporate policy is followed when doing some of this things...

Also if you forget to modify the tty's on a router, users might lock you out of telnet access:-)

please note this could obviously work better, but i wasn't looking for better, i needed to kill some time and be productive at the same time. For instance we could post process the output to only show line 17 ( I also generally prefer if guys especially co-workers to at least have an idea of whats run and what the output would look like from a console)...but time's up, and I think I'll have an easier time tomorrow...maybe some day....

http://oreilly.com/catalog/expect/chapter/ch03.html
http://www.marcelgagne.com/node/582
ftp://ftp.cisco.com/pub/cisco-expect.shar

Saturday, January 3, 2009

Understanding EIGRP from the bottom up Part I I

EIGRP Metrics:
Routers 'discuss' their topology tables. So unlike other protocols, EIGRP has a table with all the routes to a destination.

show ip eigrp topology all-links
!--is the command that shows this table - all links is optional
This topology table is the one holding all the info to make decisions, the outcome is usually a distance and vector to each destination.

Minimum bandwidth and the total delay is the information used to compute the metric. The values are automatically picked from configured values on your interfaces. Generally bandwidth will be more critical for lower bandwidth interfaces while delay is more key where high speed interfaces are used.

Formula to get :
the bandwidth:
bandwidth = 10 000 000 / bandwidth * 256
The delay
delay = delay * 256

The values got for bandwidth and delay are then used in the metric computation. I won't go into detail, please check this cisco link or read a book.

the default behaviour is to just calculate as follows:
metric = bandwidth + delay

(remember to round off your figures after each calculation, I can't remember why but i think floating point math on cisco was the reason)


so if you have the following scenario:


bw 56k
/Delay 2000---Router4---| |
/ | |
Router1== |---------Router2------|
\ |bw10000 |
\__bw 128 |delay100 |Destination Network
Delay 1000-Router3---| |bw 10000
|delay 100


metric = bandwidth + delay
minimum bandwidth = 56k
Total Delay = 100 + 100 + 100 + 2000
= 2200

[(10 000 000/56) + 2200] x 256 = (178571 +2200) x 256
= 180771 x 256
= 46277376

10 000 000/56 is actually = 178571.42857142857142857142857143 but we round it off...
If you calculate the metric through router 3, you find the metric is 20307200

so router one will use the path via Router3 to get to the destination network.
  • The bandwidth is calculated from the configured interface through which the desired network is visible ie where the update is coming in through.
  • The delay is cumulative; ie each router adds a delay and sends it backwards...Im not sure how to explain this if you can't see it....just count the delay from the destination backwards

Feasible Distance (FD)
  • This is the best metric/best path to the destination network - includes the metric to the neighbor advertising the network - from our diagram thats Router2.
reported Distance (RD)
  • This is the total metric as advertised by an upstream router/neighbor. from the ascii diagram above, that will be the distance advertised by Router4 or Router3 (obviously one is going to be used as a FD
feasible Successor (FS)
  • This is the path whose reported distance is less than the feasible distance. This is usually installed in the topology table as a backup.
The reported distance is always calculated from the router advertising the route to the network. so the reported distance from router4 is the metric to get to the destination network from Router4 and the same goes from the reported network from Router3.

If the link between Router1 and Router3 goes down, the convergence is almost instant, users will probably not even notice it since the feasible successor/backup route is immediately picked.

Checking loops:
The FD,FS and RD concepts are used to breack loops using the logic that the reported distance CANNOT be higher than the feasible distance. A route/path with a higher Rd than the FD won't show up on the topology table.

Other loop 'taking care of mechanisms'

Split Horizon - A route WILL NEVER be advertised through the interface it was learnt from.

Poison reverse - after learning of a route through a certain interface, any advertisements back the same interface are sent us unreachable for the said route/network.

Notes:
  • If a FD sends an update, queries regarding the same network are not sent to it.
  • Stuck in active (SIA) occurs if a query takes too lon gto be answered by a neighbor....
Redistribution: Ahh this is when I started having fun....I think I'll deal/write about redistribution after going throroughly through OSPF

now lets see if my trusty old motor bike survived the holidays:-).......happy new year....

Want to read some more.....go here

Friday, January 2, 2009

Understanding EIGRP from the bottom up Part I

When tackling the BSCI, I went through the EIGRP material, just enough to pass the exam. So I concentrated more on the theory end of things. I rarely if ever needed the certifications for my job ie it was'nt a requirement for promotions or anything. (telcos in kenya have very poor skill spotters in my opinion), either way we do run EIGRP so this time I took the time to understand it properly.

Key fields in the EIGRP header are as follows:

* The opcode field specifies the EIGRP packet type (update, query, reply, hello).
* The checksum applies to the entire EIGRP packet, excluding the IP header.
* The rightmost bit in the flags field is the initialization bit and is used in establishing a new neighbor relationship
* The sequence and ack fields are used to send messages reliably
* The AS number identifies the EIGRP process issuing the packet. The EIGRP process receiving the packet will process the packet only if the receiving EIGRP process has the same AS number; otherwise, the packet will be discarded.

EIGRP is very widely documented so googling for EIGRP header will robably give more details.

Other highlights:

  1. Its Hybrid and uses the DUAL (Diffused Update ALgorithm).
  2. Neighbor discovery and maintenance ensures only updates are sent when needed (hellos). The hello interval is 5 and 60 seconds, the default hold time is three times (15 and 180 seconds) the hello timer duration. Hold time is the amount of time a router will consider a neighbor alive without receiving a hello packet. The timers can be adjusted per interface with the ip hello-interval eigrp and ip hold-time eigrp.
Lets take a look:
We'll use the same topology we used for RIP to explore the EIGRP timers.

Loopbacks
R0: : 172.20.1.1/32
R1: : 172.20.2.1/32

Network Addresses:
R0: Serial1/0 : 192.168.10.1/30
R1: Serial1/0 : 192.168.10.2/30

ffr Addresses:
R0: Serial1/1.105 : 192.168.20.1/30
R1: Serial1/1.501 : 192.168.20.2/30

Ethernet interfaces:
R0: Fastethernet0/0: 20.20.20.1/24
r1: FastEthernet0/0: 10.10.10.1/24

Test reachability - before continuing - If all is well, add in EIGRP:

On R0

R0#sh run | section eigrp
router eigrp 1
network 192.168.0.0 0.0.255.255
no auto-summary

on R1

router eigrp 1
network 192.168.0.0 0.0.255.255
no auto-summary

What pops up immediately:


*Mar 1 00:02:32.223: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is up: new adjacency
*Mar 1 00:02:32.383: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.20.1 (Serial1/1.501) is up: new adjacency
*Mar 1 00:02:35.447: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.20.1 (Serial1/1.501) is resync: peer graceful-restart
*Mar 1 00:02:35.455: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is resync: peer graceful-restartR0#sh ip eigrp neighbors

IP-EIGRP neighbors for process 1

H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 192.168.20.2 Se1/1.105 12 00:10:52 894 5000 0 15
0 192.168.10.2 Se1/0 12 00:10:52 127 762 0 16

The value of the hold column shouldn't be more than the timer unless you're losing packets. Also the Q(queue) count column should always be '0' unless there's a problem.

For instance, lets block EIGRP on R0 and see what shows up on R1:

RO
R0(config)#access-list 100 deny  eigrp any any
R0(config)#access-list 100 permit ip any any
int s1/0
ip access-group 100 in

This pops up on R0
*Mar  1 00:38:34.547: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.2 (Serial1/0) is down: holding time expired
and on R1 we now have the following:
R1#
*Mar 1 00:38:34.899: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is down: Interface Goodbye received
*Mar 1 00:38:39.623: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is up: new adjacency
*Mar 1 00:39:59.147: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is down: retry limit exceeded
*Mar 1 00:40:03.619: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 192.168.10.1 (Serial1/0) is up: new adjacency
R1#sh ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 192.168.10.1 Se1/0 11 00:00:36 1 5000 1 0
1 192.168.20.1 Se1/1.501 11 00:38:08 137 822 0 10
Note the queue count is 1 (one) , this implies an un-acknowledged hello.

On R0 the neighbor is removed:
R0#sh ip eigrp neighbors
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 192.168.20.2 Se1/1.105 13 00:38:54 894 5000 0 15
Other commands you can run to troubleshoot: I'll just BOLD key info from each command
R1#show ip eigrp interfaces
IP-EIGRP interfaces for process 1

Xmit Queue Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes
Se1/0 1 0/0 124 0/15 563 0
Se1/1.501 1 0/0 137 0/15 639 0

R1# show ip eigrp topology
IP-EIGRP Topology Table for AS(1)/ID(172.20.2.1)

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status

P 192.168.10.0/30, 1 successors, FD is 2169856
via Connected, Serial1/0
P 192.168.20.0/30, 1 successors, FD is 2169856
via Connected, Serial1/1.501


R1# show ip eigrp traffic
IP-EIGRP Traffic Statistics for AS 1
Hellos sent/received: 1202/1179 <---remember we had blocked some hellos with access list 100
Updates sent/received: 109/9
Queries sent/received: 2/2
Replies sent/received: 4/2
Acks sent/received: 1/12
Input queue high water mark 4, 0 drops
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello Process ID: 174
PDM Process ID: 169
R1# show ip eigrp accounting
IP-EIGRP accounting for AS(1)/ID(172.20.2.1) <<--note teh ID is our loopback
Total Prefix Count: 2 States: A-Adjacency, P-Pending, D-Down
State Address/Source Interface Prefix Restart Restart/
Count Count Reset(s)
A 192.168.10.1 Se1/0 1 0 0
A 192.168.20.1 Se1/1.501 1 0 0
R1#show ip eigrp 100 ?
accounting  IP-EIGRP Accounting
interfaces IP-EIGRP interfaces
neighbors IP-EIGRP neighbors
topology IP-EIGRP Topology Table
traffic IP-EIGRP Traffic Statistics
!use this if running more than one AS on the same router


show ip eigrp neighbors
Already covered
!Note since these routes are learnt via connected, our routing table currently wont
show any eigrp routes. To get this we need to advertise our loopbacks via eigrp

R1#sh ip route
!--------edited----------!

192.168.10.0/30 is subnetted, 1 subnets
C 192.168.10.0 is directly connected, Serial1/0
172.20.0.0/32 is subnetted, 1 subnets
C 172.20.2.1 is directly connected, Loopback0
192.168.20.0/30 is subnetted, 1 subnets
C 192.168.20.0 is directly connected, Serial1/1.501
10.0.0.0/24 is subnetted, 1 subnets
C 10.10.10.0 is directly connected, FastEthernet0/0

See!! no EIGRP routes, now lets add the loopback addresses:
R0#sh run | section eigrp
router eigrp 1
network 172.20.1.0 0.0.0.255
network 192.168.0.0 0.0.255.255
no auto-summary

and on R1:
R1#sh run | section eigrp
router eigrp 1
network 172.20.2.0 0.0.0.255
network 192.168.0.0 0.0.255.255
no auto-summary
our output:
R0#sh ip route eigrp
172.20.0.0/32 is subnetted, 2 subnets
D 172.20.2.1 [90/2297856] via 192.168.20.2, 00:01:05, Serial1/1.105
[90/2297856] via 192.168.10.2, 00:01:05, Serial1/0
R1#sh ip route eigrp
172.20.0.0/32 is subnetted, 2 subnets
D 172.20.1.1 [90/2297856] via 192.168.20.1, 00:01:23, Serial1/1.501
[90/2297856] via 192.168.10.1, 00:01:23, Serial1/0
Notes:
*EIGRP doesnt build neighbor relationships over secondary addresses.
*Please note changing the hello interval using ip hello-interval eigrp doesn't adjust the hold. time timer for you.
*Load balancing: routing protocols install routes to your routing table, the switching fabric within the router/switch does the load balancing based on various things. eg per packet load balancing, per destination etc etc...all this depends on your switching (cef, fast switching)
let me make a quick demonstration:

If we try to reach R1's loopback with CEF enabled and the routing table as it is now (by default we use per destination loadbalancing) -
R1#debug ip icmp
ICMP packet debugging is on

R1#
*Mar 1 01:04:19.239: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.371: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.459: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.495: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:04:19.507: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
Note the path taken is the same, all packets to the same destination will use
same path

now disable CEF (globally - no ip cef)on R0 and make the same 'ping'
R1#
*Mar 1 01:07:27.411: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:07:27.495: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.10.1
*Mar 1 01:07:27.539: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
*Mar 1 01:07:27.583: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.10.1
*Mar 1 01:07:27.591: ICMP: echo reply sent, src 172.20.2.1, dst 192.168.20.1
Note the alternating paths, this is per Packet Load balancing
You can use sh ip interface to figure out what sort of switching is in use. I only did this to show how the routing protocol really only populates the routing database, how traffic is moved across networks is not its business.....I'll follow this up with EIGRP metrics,feasible distance, reported distance and feasible successors.....

References:
Cisco
Oreilly's IP routing by By Ravi Malhotra - I found chapter4 available on oreilly:-) good for you