David Ramsden – Network engineer, general geek, petrol + drum and bass head
2Aug/14

Automating mass Cisco IOS upgrades

This morning I needed to upgrade the IOS on 29 Cisco 3560G switches. Rather than login to each one, clean up the flash storage, FTP on the IOS image and set the boot image, I wrote a simple shell script and used clogin from RANCID to automate this task. Of course, nearly every Network Configuration Management platform that's any good should be able to do this but I prefer the personal touch.

The commands required on the switch were as follows:

First I tell IOS to not prompt on file operations. This makes automation easier as there's no need to deal with questions. Then I clean up the flash storage on the switch by removing any old IOS images. The IOS image is copied from an FTP server to the flash storage. The file prompt is put back to defaults and the boot system variable is set to the new IOS image. Finally the configuration is committed to NVRAM because at some point the switch will need to be reloaded.

The shell script will read in a list of IP addresses to connect to and then using clogin it'll login to each switch and execute the commands above.

The script I wrote is as follows:

A file called ips.txt has the list of IP addresses for the switches (one IP address per line). The commands listed above go in to a file called commands.txt. And lastly there's a file called clogin.txt that contains the login details that clogin needs. This would look like:

This tells clogin that there's no need to enter enable and to first try SSH and followed by telnet.

When the script is run it will grab the first IP address in ips.txt, execute clogin to login to the switch and then execute each command in commands.txt. When clogin exits, the IP address in ips.txt will be removed and placed in to a file called processed.txt. The script then prompts if it should continue to the next IP address, allowing you to review what happened to make sure the IOS image copied on OK.

This allowed me to upgrade 29 switches, whilst watching some morning TV and sipping a coffee with my feet up. All that's required now is a reload of each switch.

2Aug/14

Automating Cisco switch swap outs

So you can't automate the entire process unfortunately. You're still going to need to pull a late night and get your hands dirty...

Recently I was tasked with swapping out 4 old Cisco 10/100Mb switches with new 10/100/1000Mb switches. The old switches were a combination of Cisco 3560, 2950 and 3548 series. The old switches also had some old configurations that needed to be updated and the interface configurations weren't consistent. The interfaces also had different VLAN configurations and this was my main concern. What if I made a mistake? It's not very realistic to chase 192 ports and make sure every single one was working as expected.

Going back to a previous blog, should network engineers be programmers, writing a script to speed up the configuration process and eliminate any mistakes was the answer.

The process would be:

  1. Get the existing IOS configuration.
  2. Find the interfaces.
  3. Convert the old 10/100 ("FastEthernet") interfaces to 10/100/100 ("GigabitEthernet") interfaces.
  4. Extract the important parts of the interface configuration (description, speed/duplex if set, VLAN configuration, trunk configuration etc).
  5. Pump out the converted interfaces.
  6. Copy/paste in to a template after manually making sure it looks good.
  7. Upload the configuration to the startup-config of the new switch.
  8. Swap the switch out.

First of all, the existing IOS configurations are stored in SVN. They get here via RANCID. So I had the old configurations easily to hand. Also if anything did happen to go wrong on the night, this served as a good reference point.

My weapon of choice for this scripting task was Perl. The script I wrote took in an existing IOS configuration and extracted the physical interfaces and SVIs, converted them from FastEthernet to GigabitEthernet and grabbed all of the important stuff such as description, speed/duplex, VLAN configuration, trunk configuration, IP configuration it's an SVI etc.

Here it is:

I ran this against each of the 4 existing switch IOS configurations, checked the output quickly and then copied/pasted the interfaces in to my switch template that has everything else set such as Spanning Tree configuration, errdisable recovery, NTP, AAA, SNMP, syslog etc etc. VTP would take care of the VLAN database once the switch was connected to the core.

I connected each of the new switches to my laptop and configured the Vlan1 SVI with a temporary IP, then TFTP'd the switch template to the startup-config along with the IOS version required. Turn the switch off and on again to make sure it looks good and job done on the configuration front. Each switch took a very small amount of time to configure and I could be safe in the knowledge that all the interfaces were correct.

The 4 switches were swapped out in the dead of night. It took 4 hours from start to finish, including testing and monitoring. Out of roughly 192 ports there was only one device which didn't work the next morning and that was due to an auto-negotiation issue.In my book it was a very successful, painless and efficient change. One which I wasn't particularly looking forward to but thanks to a bit of scripting ended up being easy.

2Aug/14

Should network engineers be programmers?

Short answer: Yes.

Maybe not a programmer in the sense that you need to be proficient in C++, .NET, assembler, know UML etc but having some general programming knowledge is very useful. In my opinion and experience the most important programming skill to have is a fairly in-depth knowledge of a scripting language. Be that shell, Perl, Powershell or even batch scripts. A week doesn't go by where I don't write a script to help me with my day to day tasks. Either to automate a process or format some logs or debug output I've collected.

Personally my scripting language of choice is either shell or Perl. Shell for easy repetitive tasks and Perl for formatting data or even creating configurations. Here's a very simple example of a Perl script I wrote recently:

What does this do, apart from make my life simpler? It generates a Cisco IOS config with 29 LACP port channels and configures the physical interfaces. Then it's a case of running the script and copying/pasting the result in to the device. It also eliminates any human error. If you were having to create 29 port channels and configure 58 physical interfaces, the chances are you'll make a mistake. Such as forgetting to configure the interface as a trunk, setting the wrong channel group ID on the interface or generally getting in to a bit of a mess.

I'm going to post a few other blogs containing some scripts I've recently used to help automate tasks. Time to sharpen your scripting skills!

10Jun/12

Editing Cisco IOS ACLs

If you've administered Cisco PIX or ASA security appliances, you'll know how easy editing ACLs is. If you want to insert a new rule in to an existing ACL you can easily insert it where you want. For example:

 

This will insert the rule at position 12 of the outside_access_in ACL, pushing the existing rule at position 12 down to position 13 and re-ordering everything.

In Cisco IOS there's no obvious way to do this when working with ACLs. A lot of the time people will copy the ACL out, edit it in a text editor, "no access-list" the ACL and then paste in the modified ACL. Which does work but can be risky when working remotely as it's easy to lock yourself out of the IOS device. This can happen if you don't remove the ACL from interfaces before deleting the ACL.

But you can edit ACLs on the IOS device itself when using an extended ACL. Lets create an ACL:

 

If you view this ACL you'll notice line numbers:

 

Lets say you need to add another rule before the deny (sequence number 20). Enter the extended ACL:

 

Now you can insert a new rule by specifying a sequence number less than the deny rule (which is sequence 20):

 

If you view the ACL you'll see the new rule:

 

What you can now do is resequence the ACL so that all the sequence numbers are sequential. For example if you wanted the sequence numbers to start at 10 and go up in increments of 10:

 

If you want to delete a specific rule:

 

Tagged as: , No Comments
10Jun/12

Unable to Check Out Files, Create or Edit Pages in SharePoint

Recently had an issue with a SharePoint site where sometime over the weekend, the entire site broke and wouldn't let anyone check files out of document libraries, upload new files, edit or create pages etc.

The Check Out option on files was completely missing. Options under Site Settings were completely missing. When users opened Word and Excel documents from document libraries and tried the Check Out option from there, they'd be repeatedly prompted for authentication. I spent a long time checking permissions within SharePoint, checking service accounts, using Wireshark etc but couldn't find the cause.

Eventually I happened to noticed that the entire site was locked, in read-only mode. But how did it get like this? There's only four administrators that can do this and no one logged in over the weekend and made any changes. I then happened to read that as of WSS 3.0/MOSS 2007 SP2, when using the backup option of stsadm, the site is automatically locked. Previous to SP2 it was only recommended that you did this. It's now enforced.

The problem appears to be that if the backup fails for whatever reason, the site stays locked. To check if the site is locked:

 

To unlock the site:

 

If you read the Microsoft TechNet for the Setsitelock option for stsadm, you'll notice that it does mention that the backup now locks the site. You can force the backup to not lock the site. I've modified our backup routine so that a setsitelock operation is always run after a backup, regardless if the backup was successful or not.

Tagged as: No Comments
19Feb/12

Cage The 106

I was having a little break from doing anything more to the 106 but then a roll cage come up for sale and it was too good to pass up on. It's a 6 point cage, in red, fits around the sunroof and doesn't penetrate the dash. Every box ticked! Made by Geronimo Cages (Hartlepool) but I don't think they're in business anymore. The seller was just up the road and said he'd help fit it. Got it in within a couple of hours and I finished it off this weekend.

Not great pictures, thanks mostly to the daylight fading and using my iPhone. Excuse how unclean the bodywork is too. That's a job for next weekend.

Also replaced the fuel line that runs from the fuel filter, inside the car, through the bulkhead and to the fuel rail. Had a slight accident last weekend with a knife when initially fitting the cage.

What's next? I'd like the overhaul the brakes. There's too much going to the rear callipers which makes the car "squirm" under heavy breaking. So the plan is to run fully braided lines, have them run inside the car and fit a bias lever so that I can adjust the rear brake bias. On a dry track day the lever would be all the way forward so that the rears aren't doing anything. The problem comes about because the car is stripped out at the back and there's no load compensator on the rear callipers.

This year I'm hoping to get to Castle Combe and Silverstone. There's also talk of a trip to the Nürburgring...

Tagged as: No Comments
2Feb/12

Virtual Hosting With mod_proxy

The other day I had someone ask if there's a nice solution to the following problem:

Multiple web development virtual machines but only one external IP address.

The quick solution is to port forward on different ports to each virtual machine. For example 81 goes to VM1, 82 goes to VM2, 83 goes to VM3 etc. Which granted, would work, but isn't a "neat" solution.

Using mod_proxy under Apache is a much better solution to this problem.

Deploy a "front-end" server running Apache and mod_proxy. Create a virtual host for each virtual server and then using mod_proxy, reverse proxy to the virtual server. Port forward from the WAN to your front-end Apache server running mod_proxy.

Here's what an example config would look like on the front-end Apache server:

Requests for cust1.dev.domain.com would be reverse proxied to 192.168.0.100 and requests for cust2.dev.domain.com would be reverse proxied to 192.168.0.101. All with one external IP address and one port forward rule.

Just one of the many uses of mod_proxy. You can also use it for SSL bridging and SSL offloading. Neat!

Tagged as: , 1 Comment
2Jan/12

Creating a HA iSCSI Target Using Linux

Some time ago I created a High Availability iSCSI target using Ubuntu Linux, iscsi-target, DRBD and heartbeat. The HA cluster consisted of two nodes and the iSCSI initiators were Windows Server 2008. I was able to mount the LUN and copy a video to it, play it back and then pull the power from the primary iSCSI target. A few seconds later the second iSCSI target took over and video continued to play.

Pretty cool, huh?

Here is my guide if you want to try this. Although I've not gone back through the guide to make sure it's correct. But if you spot anything that's wrong or not very clear, please leave a comment.

Tagged as: , , 8 Comments
1Jan/12

Mallory Park Track Day

Here's a short video from my last track day of 2011 at Mallory Park. First time out with the 196bhp race engine.

All in all it was a good day without any mishaps and the engine performed brilliantly! Can't wait to get back in track this year.

1Jan/12

My 196bhp Peugeot 106 Track Toy

I've owned this 106 for quite a few years now. It stated life off as a humble 1.4 8v, standard gearbox, standard suspension. Standard everything. It's now a 196bhp naturally aspirated track toy but it's still usable as my daily driver.

At this stage I've got to say a massive thank you to Pug1Off who have done all of the major work, such as original engine conversion to a 1.6 16v and the current purpose built race engine. And credit to Northampton Motorsport who mapped the race engine.

Engine (built by Pug1Off) - 196.3bhp @ 8000rpm, 135.7lb/ft

  • 1.6 16v C2 (TU5JP4)
  • Re-bore and flowed head
  • Oversized pistons
  • One piece valves
  • Catcams 737's
  • Race bearings
  • AT Power throttle bodies
  • Emerald K6 ECU
  • 4-2-1 manifold
  • Mocal 13 row oil cooler
  • Lightened and balanced flywheel
  • Aluminium crank pulley

Drivetrain

  • VTR gearbox
  • Quaife ATB differential
  • SatchShift
  • Helix Organic clutch

Suspension

  • -45mm Eibach springs
  • GrpN Bilstein shocks
  • B8 Bilstein dampers
  • GrpN upper and lower engine mounts
  • Powerflex gearbox mount
  • OMP upper brace
  • OMP lower brace
  • Disc rear beam with 24mm rear ARB

Brakes

  • 266mm callipers
  • Ferodo DS3000 pads
  • Tarox JapanSport discs
  • Braided lines
  • Motol RBF660 fluid

Filed under: Car No Comments