CUSTOMER SUCCESS STORY
Protecting the internal systems of cyber security company FireEye requires skill, experience, grit and a willingness to think outside the box.
A few years back, cyber security company FireEye’s move to a new corporate headquarters ran into a mystery familiar to many NetOps teams. The problem: CPU loads for some inline scanners were shooting through the roof every 30 seconds, like clockwork.
Looked at through monitoring tools, the latency graphs formed repeating sawtooth patterns. And for a few days, the pattern defied explanation for FireEye Senior NetOps Engineer Matthew Baskett.
“Fortunately, we'd previously deployed a pair of GigaVUE-HC2 visibility nodes to provide traffic redirection, inline bypass and packet deduplication to feed our own appliances and our own security stacks,” he says.
The Gigamon nodes provide effective load balancing of the inline traffic, which is really handy. It allows us to use our own products and allows us to scale.
SENIOR NETOPS ENGINEER, MATTHEW BASKETT
Matt used the GigaVUE-HC2s’ inline bypass feature to dynamically remove and add back the suspect appliances from the device pool, as necessary — without affecting the rest of the stack. This gave the team the necessary breathing room to troubleshoot and find out what exactly was happening.
Putting their heads together, the team soon discovered the answer.
“After much troubleshooting, we found that the inline scanners did not support Q-in-Q VLAN tagging,” says Matt. This caused the erratic behavior. “But amazingly, we maintained uptime and throughput while we were figuring this out and correcting the problem.”
By IT standards, Matt and his team seem quietly confident, even laid back (maybe it's no coincidence that he brews his own beer). If they get a surprise phone call, their first reaction isn't necessarily concern that something broke. It's just as likely there's a new challenge popping up elsewhere in FireEye, and someone needs the team's hard-earned expertise.
“For me, cybersecurity is especially rewarding, knowing that I'm part of a company that's a leader at what it does. I see things on the news and I think, that's us. We did that.”
How FireEye Deploys Gigamon Solutions
How does Gigamon fit into your network?
We use Gigamon GigaVUE-HC2 nodes with our datacenter security stack to handle all of our ingress/egress scanning for threat analysis, which minimizes the where we have to deploy the stack.
Where exactly do you deploy them?
The GigaVUE-HC2 nodes sit in our environment wherever we really want to provide that inline packet redirection. The number-one primary focus for us is at our ingress/egress points. So, we'll typically place a security stack anywhere where we have DIA internet services.
We leverage that to take the traffic in and out of our facility, seamlessly redirecting it to a group of FireEye NX appliances. We typically run anywhere from two to eight NX appliances from a GigaVUE-HC2 node and an inline tool group. We do that for resiliency and redundancy. During standard maintenance, we can take down an NX and upgrade its code with zero impact to traffic flow.
Are there other benefits to that placement?
We can spread out the load and the total amount of throughput across the NX appliances. We also get instant and reliable fail-over for any inline device that might go offline, without having to venture into the expensive and finicky realm of optical fail-over hardware solutions.
Any parting thoughts?
We don't ever have to think about GigaVUE-HC2 nodes! For a networking team with as much breadth as we are responsible for managing, any device we don't have to worry about on a day-to-day basis, that's a win in my book.
“It'll always boil down to the basics of networking. Learn the basics, live them, breathe them—even in complex scenarios."
“Always, always learn something new, every day. Read about new technology, or try to deploy a device in a way that it's never been deployed before.”
“Listen to the old-timers, they've done this. Sometimes they will save you days, weeks, months of headaches because they've already solved that problem.”
Senior NetOps Engineer at FireEye
From Parking Lots to Stopping Bots
Now, based in Texas, Matt didn't always expect to be a network infrastructure architect for a world-leading cybersecurity firm. He found his way there by a roundabout path.
“I started my professional career in civil engineering and quickly got very tired of running rain-runoff drainage calculations for parking lots. Absolutely boring. I found more enjoyment managing and maintaining the IT systems at the engineering firm.”
“From there, my career switched into the networking side because I found a lot more enjoyment controlling the traffic: building solutions that were fast, redundant and resilient and then watching them work.”