Stephenmccarron.com

We’re a hosting provider, a ‘citizen of the internet’ and, registered with RIPE, we maintain a moderately large pool of IPv4 IP addresses (many tens of thousands). You may or may not have heard stories about IPv6 (the new, sooper dooper IP number system) that is planned to replaced IPv4, particularly as IP 4 is due to ‘run out’ of address space relatively soon.

The reality though, is that there is sufficient IP space in v4 for current requirements, and, even if the ‘free pool’ runs out (some analysts estimate as early as 2010), there will still be lots of IP’s available and being traded between providers. What’s more, due to incompatibilities between the two systems, there will be a fairly substantial IPv4 only internet for at least the next 10-15 years.

Fundamentally, IPv6 doesn’t really do anything differently to IPv4. It doesn’t eliminate NATs, it doesn’t reduce routing load, traffic engineering is the same and while large, the IPv6 space is not infinite. To make things more difficult, there is not way to incrementally deploy IPv6; -everything- must be changed all together, all the way from the back-end servers to the front-end routers for it to work. That also means the ISP’s and hosting companies need to change their provisioning, billing, monitoring, measurement systems and all of this is only possible with support from all the vendors involved.

Early adopters and Pioneers are moving cautiously in this space,  actual traffic is extremely small and several things need to be ‘fixed’ before we can start the ball rolling in earnest (as a small example, Windows XP doesn’t support DNS queries over v6, therefore won’t work in an IPv6 only environment). Another simple example is SMTP – the internet’s killer app – Email. Until the whole internet is IPv6, all SMTP servers will need dual-stack relaying (otherwise IPv4 mail servers couldn’t sent to IPv6 server and vice versa).

Don’t get me wrong, we need to move to IPv6 and we need to start that process as soon as we can, but we need to keep an holistic, end to end view and filter the marketing hype or we will end up with a fragmented, kludged internet (and then what will we do with IPv10 )

In the last few months, we’ve all seen various providers and services have problems.

http://url.ie/8xh documents a long list of infrastructure issues with some ‘big name’ hosts, some of whom market their ‘zero downtime’ networks – you can see our own status site for an issue we experienced this week – http://www.hosting365status.com – and more recently Amazon S3/EC2 suffered an outage – http://url.ie/97n

Personally, I hate seeing providers in trouble. I’ve been through the same, I know how it feels and it’s not pleasant. You stress, worry, fret and get not a little angry after having planned, invested, planned more, invested (a lot) more and then planned some more and yet still things go wrong.

Professionally though, I am delighted when issues like these happen. Not because I am some kind of masochist, nor do I get pleasure from another providers problems, but because it helps educate the customer.

Customers want 100% uptime of hardware, networks, datacentre, chillers, staff, they want 5 minute responses to their questions 24/7 and they want it all for less than most of them pay for advertising the service they are running online.

This isn’t a rant, I’m not saying ‘customers’ are dumb’, I guess I’m just reiterating the one truism in this business;

- stuff breaks

No matter what you spend, how much you plan or how good your staff (inhouse or outsourced) are, there is no getting away from that fact.

You can invest in multi-site, cloud based, real-time-replicated systems, and reduce your risk of exposure to tiny percentages, but, as Amazon just found out, a simple software glitch can happily replicate itself and upset those plans.

Service providers can certainly do their part. Hosting365.com is about to launch our new ‘Cloud’ Platform, which offers a huge grid of computing and storage, built on world-class HP Blades and SAN, running a super resilient Cisco network and leveraging the best in class virtualisation technologies HP and VMware have to offer. With offsite replicated DNS and load balancing, and multiple ‘clouds’ in multiple datacentres on separate networks, we are getting close to that 100% target – and are doing it at a price point that doesn’t break the bank. Customers need to also review their own continuity plans – do you have backups for the backups of the backups?

Customer support.

As a service provider, it’s the one thing that causes the most stress, hassle, grief and pain (both for me, my team and my customers). The old adage ‘Wouldn’t life be great without customers’ rests probably on the lips of many service providers

Customers who need to be trained or educated; who are not familiar with the systems or processes; who are not happy with perceived or real failings or mistakes; who bemoan the ‘crap’ service on forums, blogs, twitter, etc, etc, etc.

Every service provider has seen it, every provider gets the complaints and suffers the indignant customer on the phone or has to deal with the irate customer on public fora, be it right or wrong.

Personally, I’m ‘happiest’ (and I do use the term loosely ) when I’m getting to hear customer issues.

To use another adage – ‘if you think everything is going well, you clearly haven’t a clue what is going on!’

No one is perfect, certainly not in service industries where even huge companies, devoted to service, can get it wrong and upset the customer or deliver a poor experience. It’s surprisingly easy to screw up, for a system to break or act in an unpredictable way, or, simply, for a customer care agent to ‘have a bad day’ and the customer to experience that.

Not for 1 second do I make excuses for these things. The only way to learn, to grow, to improve, to truly become a great service provider, is to take time and listen. To digest and understand the problem, the failing, the screw up, and work to improve it.

Some concrete examples from hosting365… We suffered a (fairly) high profile power outage in our main facility about two years ago .We analysed, studied, brought in the experts, listened to our customers and spent over 600k in the 8 months that followed building a new Substation, installing new transformers and building a brand new MV Switch Room for our facility. We can now stand up tall and proud and guarantee we’ll never suffer the same issues again. In the same vein, we’ve spent hugely on skilled staff, infrastructure, networks, kit, etc, etc, to ensure our customers get an experience that exceeds their expectations. In fact, as a privately held company, we’ve spent over 4 million euro or so on continuously improving our service and growing our team.

The key issue though, is that unless a customer actually comes here, physically to our datacentre, they don’t see much of this work. Yes, things are stable and reliable and everything stays online, but that’s a given these days. From the outside world’s perspective, the same kind of reliability can ‘appear’ to be provided by hosts with a single server rented from a decent data centre in the USA. Unfortunately, and as I found out about two to three years ago, once you exceed a certain scaling point as a service provider (I believe around 12-15 staff) you can no longer ‘infect’ the customer with your personal passion and enthusiasm. You just can’t meet everyone that sign’s up, discuss everyones requirements and walk them all through the experience. For that you need to depend on building an excellent team and giving them excellent tools to enable them to deliver excellent service. (I know, too many ‘excellents’ )

I remember answering support tickets at 3 in the morning, standing in data centres until my feet were sore and dealing with customer issues on my honeymoon in Mauritius 5 years ago.

I take my business very personally (perhaps sometimes too much so) and believe very firmly in listening carefully to every single customer issue. The customer who takes time out of their day to give you feedback on a failure or problem is the most valuable customer you have, and deserves your time, your patience and your action. This post was prompted by a local provider who did quite the opposite recently on a well known public forum, shutting down a customer site when they expressed dissatisfaction publicly, a move which horrified me personally. The day a provider stops listening to customers and acting on what they hear, is the day they are finished in the service business.

An article at ENN got me thinking, and deciding to look at our stats (http://www.enn.ie/article/10123837.html)

Reported by IE Internet, they claim Spam is running at 63.3%, based on 34k mailboxes they claim they are scanning mail for.

A quick look at our own systems show the following numbers:

As you can see, we’re seeing clean mail of just 6.7% – indicating Spam, Virus and general ‘junk’ levels of 93.3%

Why the discrepancy? Well, for one, I think we might scan a bit more email, and larger providers who scan even more will no doubt have even more refined numbers.

Spam Filtering costs a lot of money to deliver well, we run HP DL580 servers with 4 physical CPU’s and oodles of RAM to keep up with the effort required to filter all the email, without slowing it down too much (a fine line).

In the last month, we’ve processed just over 75 million mails (generating the percentages in the Pie above). The crazy thing is that we only scan mail for 1,752 domains (as we charge for the service) which is a very very small percentage of the nearly 100k domains we host. While people don’t like spam, few are happy to pay for it’s prevention as a commercial service.

I spoke to John Kennedy from Silicon Republic / Irish Independant about our Cloud platform investment and the future of the web.

Our new ‘Application Aware Infrastructure’ allows us to offer a grid of computing resources and storage, tied to a world class network and data centre, and soon, tied to a wide array of different datacentres. The future of application delivery online is these ‘cloud grids’ much like the power utility grid replaced stand-alone and in-house power generation at the turn of the last century.

Virtual data centres and application delivery networks are the future, and hosting365 are at the head of the wave…

……

Dublin-based data hosting firm Hosting 365 has told siliconrepublic.com that it has in the past year invested €2m to upgrade its facilities to include a blade server environment and virtualisation technologies. Hosting 365 managing director, Stephen McCarron, explained: “A year ago we took a risky outlook on what is required and we decided to build a cloud platform. We had the idea that hosted services can be delivered entirely across a virtualised environment, but bespoke for each customer.

“It cost us €2m to build out the high-end carrier-grade infrastructure because it required increased processing power and data storage.”

McCarron said the company signed key deals with HP for the blade servers and VMware for the virtualisation software.

“It effectively allows us to put server technology into 60sq ft of space that traditionally would have demanded 6,000sq ft.”

McCarron explained that with the new blade technology Hosting 365, which provides services to the Department of Social Welfare as well as Cityjet and Carphone Warehouse, is able to use just 40pc of the power consumption.

“As a result of the virtualisation software we can get an 85pc utilisation of available servers at any one time, whereas in the past it was a 10pc utilisation.”

McCarron said that the new investment will enable Hosting 365 to proceed with its vision of providing Hardware as a Service (HaaS), which will run in parallel with the growing trend of Software as a Service (SaaS).

“For example, the Department of Social Welfare may see a 10pc utilisation of its web servers during the year but right after Budget Day it could jump to 100pc. The Department can upscale their server requirements in line with demand without having to panic,” McCarron said.

…..

Full version here: http://url.ie/8vc

Welcome to my new, new blog. Only the third attempt to start recording ‘cwazee schemes’ in a more structured way…