30 days to go!

Closing some key gaps. From the blueprint i can pretty much tell I'll have issues with multicast primarily because I rarely work with those technologies.....for some very interesting issues from the INE workbook I curiously got stuck on a bgp (confederation) configuration that increased the time I took to do that lab by a whole 40 minutes....grr...oh well...on and on we go...

Key migration - done

Well it happened, our new core is fully functional. Lots of lessons learnt, most of them non technical.

This migration was mainly breaking up the old Huawei mpls core and making all their P/PE's into CE's to the new core comprising of CRS-1's and 7609's. It went very smooothly.

it seems the most annoying thing in this whole process is going to be mainly dealing with people.

- due to the size and components involved, even my manager was not sure we could pull it off. Which creates interesting scenarios, all of them annoying time wasters. Im happy for them to waste the time they pay for, unfortunately this has started eating into my personal time and I dont think I can/will allow that

- Other departments also need to be given the confidence that everything will work out and for us (my team) this (running/managing a network) is easier than a walk in the park - we have to watch out for thugs and other miscreants though. IP transformation needs to be understood as a process that takes time.

- I hate project managers especially if they dont have a technical background, even worse if they are not willing to listen and happen to sign your paycheck.

Next up of course there will be the usual setting up all the network monitoring systems, ensuring qos is well done, all this have been tested but we now have to confirm how it fares with live traffic.

If you work for a telco and have had to deal with the ip transofmation process, Id be very interested to know how you handled peoples confidence levels in the process without going nuts!

- the next four weeks are focused on my lab. and I feel like I've pretty much achieved what I wanted here. oh....from now on I am also scouting for contracts especially if they have anything to do with design/implementation and general IP transformation ....

lab dates set

Well...big project done, some free time coming up, mobile lab right here at home:-)

Lab Details
Lab Type Routing and Switching
Lab Date 07-MAR-11
Lab Location Nairobi

I wish me well:-)
sleepless nights coming up....

CRS1 snmp and other counters

Sooo Im almost done with a very interesting project and wanted to share a little something on what happened when we wanted to monitor the optical interfaces.

First off the core network has a bunch of CRS1's interconnected using 3 x 10G DWDM links to the rest of the network giving us what someone fondly called a 10Gig core, or 30 depending on how/what you calculate.

Now the dwdm link characteristics are very important to ensure the deck of cards above it doesnt come crashing down. Some fiber was not very clean and you'd get parameters like:

RP/0/RP0/CPU0:xxxxxxx1#sh controllers dwdm 0/0/0/2
Thu Nov 25 14:10:16.303 EAT

Port dwdm0/0/0/2

Controller State: up

Loopback: None

G709 Status


OTU
         LOS  = 0           LOF  = 0           LOM  = 0
         BDI  = 0           IAE  = 0           BIP  = 0
         BEI  = 0           TIM  = 0

ODU
         AIS  = 0           BDI  = 0           OCI  = 0
         LCK  = 0           BIP  = 0           BEI  = 0
         PTIM = 0           TIM  = 0

FEC Mode: Enhanced FEC(default)
         EC(current second) = 3217480     EC   = 602132861570  UC   = 43432861570
         pre-FEC BER = 2.89E-4            Q  = 3.42          Q Margin = 1.74

Remote FEC Mode: Enhanced FEC
         FECMISMATCH = 0

Detected Alarms: None
Asserted Alarms: None
Alarm Reporting Enabled for: LOS LOF LOM IAE OTU-BDI OTU-TIM OTU_SF_BER OTU_SD_BER ODU-AIS ODU-BDI OCI LCK PTIM ODU-TIM FECMISMATCH
BER Thresholds: OTU-SF = E-2  OTU-SD = E-5

OTU TTI Sent     String ASCII: Tx TTI Not Configured
OTU TTI Received String ASCII: Rx TTI Not Recieved
OTU TTI Expected String ASCII: Exp TTI Not Configured

ODU TTI Sent     String ASCII: Tx TTI Not Configured
ODU TTI Received String ASCII: Rx TTI Not Recieved
ODU TTI Expected String ASCII: Exp TTI Not Configured

Optics Status

         Optics Type:  Cisco 10Gb DWDM
         Wavelength Info: C-Band, XXX ITU Channel=27, Frequency=1xx.80THz, Wavelength=1534.976nm
         TX Power = 4.45 dBm
         RX Power = -10.90 dBm
TDC Info

        TDC Not Supported on the Plim

So any way we needed a way to graph and send alerts based on output like that.

IOS/XR's MIB's do not seem to have anything I could work with.

*if you know the MIB/OID to get me TX/RX power,UC,EC and prefec values I'll be very grateful.

*If you also happen to have an explanation on how to interpret the pre fec values eg:

pre-FEC BER = 2.89E-4 

I'd also be very happy.

Enter some scripts:
to get this info we used a combination of expect and perl.

Basically we wrote a script that connects to each node and runs runs the command and dumps it on a text file. our perl-fu then comes on along and picks out the bits and pieces we need and dumps it on a database.

Zabbix (our mrtg'like monitoring system) graphs those. A few other scripts send us sms and email alerts.

Sample expect script:

set timeout 3
spawn /bin/bash
match_max 100000
send -- "telnet 192.168.1.1\r"
expect -exact "telnet 192.168.1.1\r
Trying 192.168.1.1...\r
Connected to 192.168.1.1.\r
Escape character is '^\]'.\r
\r
\r
User Access Verification\r
\r
Username: "
send -- "yadada\r"
expect -exact "durangor\r
Password: "
send -- "django\r"
expect -exact "\r
\r
RP/0/RP0/CPU0:xxxxxxxx1#"
send -- "show controllers dwdm 0/0/0/2"
expect -exact "show controllers dwdm 0/0/0/2"
send -- "\r"
send -- " exit\r"
send -- " "
expect eof

its a simple one that one.

other helpful commands for graphing other things:
sh snmp mib object-name
sh snmp interface
sh snmp interface tenGigE 0/0/0/0 ifindex

I am having alot of fun with IOS xr, the crs1 has some interesting features/utilities too....more on this later...

the matrix & stupidity

sunday afternoon, nice sunny day re-running 'The Matrix'....while listening to the line with morpheus saying the matrix is all around us , it immediately jumped to mind and I completed the sentence, just like stupid people....

the matrix is all around us, just like stupid people -

* one of my biggest fears is that i become stupid or thick or......

Gitau