Joel Snyder’s Blog

I forgot just how bad things can be in IT, if you’re small. A friend asked me to help him with an email gateway. He ordered the smallest—$5,000—box that one of the famous vendors offers. Which is, frankly, a lot of money for a guy running a small consulting company. Of course, when I say “he ordered,” I’m leaving out the part where it took 2 months to order it because the vendor pointed him to Worst Distributor in Dallas who couldn’t get him a quote, wouldn’t answer his emails, and never took the order. He only got his system because one of the vendor’s sales guys took pity on him and took his money.

It was great for about two weeks. We had a conference call and a smart SE helped get it installed and we were happy and life was good. Then, 2 weeks later, the power supply pooped out.

We discovered that this vendor’s tech support system is designed to keep people from actually being able to get support. You can’t send email: no address. You can’t call in—the voice mail says “you must use the web.” But you can’t enter a trouble ticket on the web site unless you have an account. Which, because this vendor seems to have a CRM built using punch cards, they had neglected to give us.

A few days later, the sales guy gets back to us—he’s the only person at the company who will actually return our calls—and sets up a support account. Which is groovy and we are finally able to say “our box is broken.” This is progressing nicely down the chain of problem solving until someone realizes that the company my friend works for (a BIG company) is not the same as his consulting company (a very SMALL company) and promptly cancels all of our technical support accounts, closes the call, and now we’re back to square one with a box that doesn’t work, no support, and no returned calls. Skipping some agonizing details, let’s say that saintly sales guy intervenes again and, 5 weeks after our first system dies, we finally get a new one—which, of course, won’t work.

It doesn’t work because the punch-card based CRM system is too stupid to understand that this box replaces the old, dead, box and it won’t let us activate it. Which I can’t fix because we still don’t have technical support turned back on.

What do I do? I give up. Get out a screwdriver, swap the power supplies, put the old box back in place, and get the system back up and running. What should the vendor do? Well, if you’re not going to bother to support small businesses, you should stop taking their money.

Since 1983, I’ve been doing product testing. It’s not the only thing I do, but over the years it’s filled about a third of my working hours.

It’s a strange kind of thing to do for a living—looking at products, trying to use them, put them into production on a real network, try and guess how someone who actually wants to buy them might use them.

Testing products is so easy to do wrong, and it’s hard to do right. Combine in the pressure of hitting a deadline, making it all work on budget, and fitting the results into a tiny number of words… and it’s a wonder that we get anything at all that’s useful to anyone.

But people eat this stuff up. I guess whether they know or not, they realize that doing what we do as product testers would be tough work. So back to the lab I go, trying to put it all into perspective, trying to generate something useful.

It’s hard work, and it doesn’t pay near what it costs, but it’s important. Of course, the junk out there will all eventually disappear on its own, but I’ve discovered that the good guys have to be called to account as well. Not just on bugs and security problems (there’s a bunch of people who love to deal with *that* little nightmare), but also on product design.

Along the way I’ve made some awful enemies. Tell someone that their baby is ugly, and they can’t ever forgive you. Especially if you say it in public. And, I’ve made some great friends (even people who had ugly babies). A good engineer or designer or manager thrives on feedback, and can build better products because of it.

I just finished a white paper on how to evaluate UTM (Unified Threat Management) products in the enterprise space. I think that this is substantially different from the way we look at them in the SMB space.

Evaluating Unified Threat Management Products for Enterprise Networks

Here’s the Overview:

The term Unified Threat Management (UTM) has as many meanings as there are products that carry that label. While UTM has primarily focused on the small- and medium-sized network, products are coming to market that aim at the enterprise. This white paper will help you understand the specific issues that enterprises need to consider when looking at UTM products, and offers guidance on evaluation criteria for enterprise-class UTM.

At its core, UTM brings together three main ideas: multiple security features, integrated on the basis of a mature firewall, deployed in an appliance form-factor. The intuitive appeal of UTM is obvious: why have two (or three or four) boxes performing separate functions, when a single box will do? As security threats to corporate networks have increased at an alarming rate, the number of devices to combat these threats has grown at nearly the same speed. However, at some predictable point, it’s not feasible to have every new threat addressed by its own dedicated device.

The reasoning behind UTM has resonated strongly with managers commanding small and medium-sized business (SMB) networks, where UTM firewalls—called such because the firewall is the undisputed lynchpin of the UTM product— have quickly become a standard offering from every vendor. In this market space, UTM firewalls, with combined features that include anti-virus protection and intrusion prevention built in to the same appliance, both reduce costs and simplify configuration.

UTM products in larger enterprise networks areisn’t an easy sell primarily because most UTM products are indeed aimed directly at the SMB environment and enterprise network and security managers haven’t had reason to view them as appropriate parts of their security strategy. Fortunately for the higher end, this product deficit is quickly changing, as enterprise-class firewall vendors are adding UTM features to their product lines. [[This is a place where we could list enterprise UTM products.]].

Obviously, evaluation and design criteria for UTM in enterprise networks must be very different from those of SMB-sized networks. When UTM concepts are brought to bear on large networks, in ways appropriate to those networks, they offer the network and/or security architects’ tremendous flexibility to control and mitigate the risks associated with security vulnerabilities.

Because UTM, in general, and especially UTM in enterprise networks, is new, network managers need a framework to evaluate products and match them to enterprise requirements. This white paper offers six separate issues for network and security architects to consider that are important to any enterprise-sized deployment of UTM.

I just finished a new white paper on picking Intrusion Prevention Systems.

Six Integral Steps to Selecting the Right IPS for Your Network

Here’s the Executive Summary:

Network Intrusion Prevention Systems (IPS) can be extremely effective pieces of your overall network security strategy. However, the IPS marketplace is filled with products that all do very different things and are suitable for very different environments. Therefore, buyers beware, because simply throwing any IPS into the network without careful consideration can be a costly error, both in terms of capital outlay and operational provisions.

The critical question to answer is: “Why are you buying an IPS?” (Step 1) Answering this question will help define both what you want in an IPS and help you weigh what you can expect to get from these products as you evaluate them for use in your network.

With the answer to this underlying question in hand, you’ll be well positioned to closely examine four aspects of IPS products that distinguish them from each other:

• Security parameters and coverage (Step 2)
• Performance (Step 3)
• Form factor (Step 4)
• Management (Step 5)

Considering how each product delivers on these four characteristics will allow you to quickly and efficiently create a short list of products that you will need to evaluate and test in your own network (Step 6) as the final –and essential - part of assuring that you are achieving the goals that will justify the costs associated with deploying an IPS in your network.

I just finished a project where I had to use a lot of different security products and I was struck at how different they are, and how much distance there is between good and merely average products.

Much of the time, people don’t decide to buy the best product; they buy the product that they like the best. Those are two very different things. What this means is that many companies have products that aren’t all that great—but they’re pretty happy with them, because they solve at least some problems, and they know how to use them, or they got a good deal, or their sales rep is really good looking or buys them lunch a lot.

I know I’m not going to change the mind of someone who knows the answer before they even ask the question of “what product should we use,” but I wanted to list four characteristics that, in my experience, really separate the great products from merely good ones.

Great products are self-documenting. It’s a rarity to find good documentation nowadays, which means that security products have to be designed to operate in a documentation-poor environment. Context-sensitive help, while a pain in the butt to build, is now a basic requirement for almost any product, and is always present in great products.

However, self-documenting doesn’t just mean context-sensitive documentation. It also means that the design of the product’s user interface makes it clear what the user is doing, and why, and what the effect will be, all without using product-specific jargon. A single extra sentence of explanation in the management interface can save an hour of searching through documentation to figure out what is really going on.

Great products have good logging and debugging output. (that’s two things, in case you’re keeping count.) I’ve never seen a product that didn’t require some troubleshooting, and the key to good troubleshooting is getting the information you need out of the debugging logs. When the logs are in a single place, and when the controls for searching and managing the logs are well put together, it’s easy to uncover what’s going on. When the logs are in 10 different places, in different formats, or (as I’ve seen recently) can’t even be viewed and have to be sent off to some external SYSLOG server, then debugging can be a nightmare.

There’s an art to writing good logging, and you’ll know a great product when you have to dig through the logs. Recently, in solving a problem I spent over an hour reading thousands of lines of logs that a not-very-great VPN product generated in a transaction that took less than a half-second to complete. It wasn’t logging; it was random debugging output that a half-literate developer had inserted to help fix his buggy code. On the other end of the VPN, though, was a great product that put out just 8 lines of logging, including a very clear pointer to where the problem was.

Great products are instrumented so that you can find out what you need to know. I’m continually astonished at how simple things you might want to know about a product are hard to find out. Questions like what’s the CPU load? How much bandwidth is being consumed? What are my outstanding critical alerts?

Security products especially have gone for the “dashboard” idea, trying to cram all kinds of status information into attractive pie charts that will look good in a demo—while not really helping the administrator answer the simple question “what is happening NOW?”

This is one of the more difficult attributes to see in casual use, because you’re often not sure what you care about until you’ve used a product for a while. But one thing is for sure: if you don’t see any status information, there’s something wrong. Many products err on the side of too much information.

I probably won’t convince you to change your buying or coding habits, but if you keep on the lookout for self-documentation, easy-to-use centralized logging, and meaningful status information, you’ll know whether you’re on the road with a good-enough product, or a truly great product.