Ramblings From The Litter Tray of Life

Posts Tagged ‘servers’

Upgrading old servers …

Posted by graycat on 5 May 2009

sometimes it’s just not worth it.

I’ve been lucky this weekend and whilst it was a three day one (bank holiday Monday!) I’ve spent about 30hrs working in various offices. Oh yeah, check out my rock n roll lifestyle! Bet you’re jealous, huh?


Anyway, twelve of those lovely hours were spent on Monday trying to upgrade a server (I’ve just found out it’s over 7 years old now.). Funnily enough, it was a bit of a struggle.

6hrs were spent just getting to the hardware to talk properly. I did indeed rock a little happy dance when I finally managed to get it booting from the CD. Yes, it was that bad.

2hrs were spent trying to get into the BIOS as some previous owner / admin / clown had put a password on it and not written it down.

4hrs were spent trying to get Windows to install. Only for it to fail on restart with hard drive issues.

So the primary file server for an office is down and i’m 3hrs into the next day and guess what – yep, still no further forward.

Why can’t people accurately document things?! I mean seriously, if you’ve built something a particular way for a reason – put down why. For example, there’s a NAS box server here where the OS drive is D and not C. Why? Because it had a raid card failure and the new one swapped the drive letters about. Well actually it destroyed the OS partition and left the data one and at that contained a nice high fraction of a terabyte of data, it was decided to leave it that way round. Yes, it’s annoying but not as much as spending the whole weekend breaking it down and rebuilding it then watchingabout 0.9Tb replicate from a really slow site. All of this is noted in the documentation though so “the next guy” can look at it and see why.

The box I’ve been dealing with this weekend is another thing entirely. Not only was it built “wrong” using the OS to do the RAID for all of the drives but there’s no documentation as to what it is or where the drivers are. Fortuanately we managed to liberate a few of them but we’re still playing “guess the part” with some bits.

Anyway, the upshot of this is that upgrading a seven year old file server has probably cost us more in down time and my overtime (as if I get paid that any more!) then buying a brand new one! So i’m just going to leave you with this thought: “sometimes the most expensive route is cheaper in the end”

Oh, and don’t even ask what the server is coz we don’t know!!

Posted in IT | Tagged: , , , , , | Leave a Comment »

Going Virtual!

Posted by graycat on 16 June 2008

Well it seems like it’s the hot topic at the moment in the server arena and I can finally say that we are dipping our toes into the virtual server paddling pool and I’ve brought my costume too!

My reasoning for a long time is that we are very very server heavy to the tune of one server for every 3-5 people. Now that is heavy even if he is my brother. This is partly due to the geographical spread of our sites with each site requiring certain local resources and offsite DR capabilities. However, I think the real reason we’ve grown to so many servers is the fear of putting eggs into a single basket – no matter how nicely it’s weaved, something is going to get broken eventually. Over time this has meant that we’ve got servers that just do this one job, or this few jobs or this and this but nothing else mainly for the fact that the servers were unstable and when, not if, they went down then you lost all the rolls at once. Not good, as they say. This sad state of affairs was, in my opinion, exacerbated by buying “custom” or white box servers and not maintaining them sufficiently for them to function for a long and happy life. More often then not the failures would be hardware related so the theory went “it you can’t trust that bit of kit, best we get another one and put this role on it over there”.

Not a bad theory and it did work.

However we’re now in the situation where the sheer importance of the IT infrastructure has been acknowledge throughout the upper echelons of the company and we are allowed to implement “care and support” of the servers as it should be. Things that seem no brainers now like separate comms rooms from the general office space, run and standby cooling, actually cooling of any type was a major fight in some places not to mention redundancy in power and backups. Because of this shift in attitude it is now very rare that we have any hardware failures.
When this is combined with an agreed rolling replacement of servers and an eye to consolidating to fewer physical machines vitrualising servers becomes very attractive.

From a support point of view, I’ve found that the one thing that causes the most outages is the OS itself. The hardware is now very high quality and redundant within each server so if you do fail a drive, the server just keeps on rocking. However, if you kill the OS …. you’re basically buggered. On this basis alone, why buy over-spec’d servers that will at more be 20% used?

So after floating the idea a few times over the last few months (usually to be confronted with a big “Not going to happen. We need all these servers”) I’ve got approval to run a test case and consolidate three existing servers onto the one VMWare rig as a proof of concept. If all goes well, we’ll look to expand the rig with more RAM and hard drives and move more servers over as thing are needed.

Our initial build will be a HP ML380 with two Quad Core 3GHz CPU’s running 12Gb of RAM with ~1.5Tb of external storage on a SCSI array. This can easily be expanded up to over 20Gb of RAM and another 1.5Tb external array or even directly on to a SAN if required ….. but that’s a long way off yet.

The servers we’re looking to consolidate have a variety of roles but the main ones are an existing “IT” server, a print and application server and SharePoint test machine. All of which are well over 3 years and most likely beyond 5 years too if memory servers me correct. Our initial plan will be to do a “physical to virtual” cloning of these servers and see how things go. However, once the dust has settled I’m planning on separating the application server into separate virtual servers for each application.

Why? You might ask. Won’t that just make more servers for you? Well, yes and no. The issue we have is not so much the number of servers but the number of physical boxes that all need looking after. With making them virtual we will be able to separate out those tricky applications that even after almost a decade of working with them, I still consider them to be just so much black magic and voodoo. If we have a completely clean OS install with just the application on-top then it can be customised to suit and completely sandboxed from other interferences. Trust me – for some of these applications when even if you update it then remove the update as per the guidelines they still won’t work, this is a great bonus!

So it looks like I’m going to be getting a crash course on VMWare Enterprise in the very near future. Not that I’ve got nothing else to do (anyone need a job? Got a helper spot or two going!) but I’m actually really looking forward to this …. and it looks like I’ve been given it as my baby too so fingers crossed! I’ll write about anything interesting I come across. Probably.

Posted in IT | Tagged: , , , | Leave a Comment »