The New York Times this morning published a story about the Spamhaus
DDoS attack and how CloudFlare helped mitigate it and keep the site
online.
The Times calls the attack the largest known DDoS attack ever on the
Internet. We wrote about the attack last
week.
At the time, it was a large attack, sending 85Gbps of traffic. Since
then, the attack got much worse. Here are some of the technical details
of what we’ve seen.

Growth Spurt

On Monday, March 18, 2013 Spamhaus contacted CloudFlare regarding an
attack they were seeing against their website
spamhaus.org. They signed up for CloudFlare
and we quickly mitigated the attack. The attack, initially, was
approximately 10Gbps generated largely from open DNS recursors. On March
19, the attack increased in size, peaking at approximately 90Gbps. The
attack fluctuated between 90Gbps and 30Gbps until 01:15 UTC on on March
21.

The attackers were quiet for a day. Then, on March 22 at 18:00 UTC, the
attack resumed, peaking at 120Gbps of traffic hitting our network. As we
discussed in the previous blog post, CloudFlare uses Anycast technology
which spreads the load of a distributed attack across all our data
centers. This allowed us to mitigate the attack without it affecting
Spamhaus or any of our other customers. The attackers ceased their
attack against the Spamhaus website four hours after it started.

Other than the scale, which was already among the largest DDoS attacks
we’ve seen, there was nothing particularly unusual about the attack to
this point. Then the attackers changed their tactics. Rather than
attacking our customers directly, they started going after the network
providers CloudFlare uses for bandwidth. More on that in a second, first
a bit about how the Internet works.

Peering on the Internet

The “inter” in Internet refers to the fact that it is a collection of
independent networks connected together. CloudFlare runs a network,
Google runs a network, and bandwidth providers like Level3, AT&T, and
Cogent run networks. These networks then interconnect through what are
known as peering relationships.

When you surf the web, your browser sends and receives packets of
information. These packets are sent from one network to another. You can
see this by running a traceroute. Here’s one from Stanford University’s
network to the
New York Times’ website (nytimes.com):

1  rtr-servcore1-serv01-webserv.slac.stanford.edu (134.79.197.130)  0.572 ms
2  rtr-core1-p2p-servcore1.slac.stanford.edu (134.79.252.166)  0.796 ms
3  rtr-border1-p2p-core1.slac.stanford.edu (134.79.252.133)  0.536 ms
4  slac-mr2-p2p-rtr-border1.slac.stanford.edu (192.68.191.245)  25.636 ms
5  sunncr5-ip-a-slacmr2.es.net (134.55.36.21)  3.306 ms
6  eqxsjrt1-te-sunncr5.es.net (134.55.38.146)  1.384 ms
7  xe-0-3-0.cr1.sjc2.us.above.net (64.125.24.1)  2.722 ms
8  xe-0-1-0.mpr1.sea1.us.above.net (64.125.31.17)  20.812 ms
9  209.249.122.125 (209.249.122.125)  21.385 ms

There are three networks in the above traceroute: stanford.edu, es.net,
and above.net. The request starts at Stanford. Between lines 4 and 5 it
passes from Stanford’s network to their peer es.net. Then, between lines
6 and 7, it passes from es.net to above.net, which appears to provide
hosting for the New York Times. This means Stanford has a peering
relationship with ES.net. ES.net has a peering relationship with
Above.net. And Above.net provides connectivity for the New York Times.

CloudFlare connects to a large number of networks. You can get a sense
of some, although not all, of the networks we peer with through a tool
like Hurricane Electric’s BGP looking
glass. CloudFlare connects to peers
in two ways. First, we connect directly to certain large carriers and
other networks to which we send a large amount of traffic. In this case,
we connect our router directly to the router at the border of the other
network, usually with a piece of fiber optic cable. Second, we connect
to what are known as Internet Exchanges, IXs for short, where a number
of networks meet in a central point.

Most major cities have an IX. The model for IXs are different in
different parts of the world. Europe runs some of the most robust IXs,
and CloudFlare connects to several of them including LINX (the London
Internet Exchange), AMS-IX (the Amsterdam Internet Exchange), and DE-CIX
(the Frankfurt Internet Exchange), among others. The major networks that
make up the Internet –Google, Facebook Yahoo, etc. — connect to these
same exchanges to pass traffic between each other efficiently. When the
Spamhaus attacker realized he couldn’t go after CloudFlare directly, he
began targeting our upstream peers and exchanges.

Headwaters

Once the attackers realized they couldn’t knock CloudFlare itself
offline even with more than 100Gbps of DDoS traffic, they went after our
direct peers. In this case, they attacked the providers from whom
CloudFlare buys bandwidth. We, primarily, contract with what are known
as Tier 2 providers for CloudFlare’s paid bandwidth. These companies
peer with other providers and also buy bandwidth from so-called Tier 1
providers.

There are approximately a dozen Tier 1
providers on the Internet.
The nature of these providers is that they don’t buy bandwidth from
anyone. Instead, they engage in what is known as settlement-free peering
with the other Tier 1 providers. Tier 2 providers interconnect with each
other and then buy bandwidth from the Tier 1 providers in order to
ensure they can connect to every other point on the Internet. At the
core of the Internet, if all else fails, it is these Tier 1 providers
that ensure that every network is connected to every other network. If
one of them fails, it’s a big deal.

Anycast means that if the attacker attacked the last step in the
traceroute then their attack would be spread across CloudFlare’s
worldwide network, so instead they attacked the second to last step
which concentrated the attack on one single point. This wouldn’t cause a
network-wide outage, but it could potentially cause regional problems.

We carefully select our bandwidth providers to ensure they have the
ability to deal with attacks like this. Our direct peers quickly
filtered attack traffic at their edge. This pushed the attack upstream
to their direct peers, largely Tier 1 networks. Tier 1 networks don’t
buy bandwidth from anyone, so the majority of the weight of the attack
ended up being carried by them. While we don’t have direct visibility
into the traffic loads they saw, we have been told by one major Tier 1
provider that they saw more than 300Gbps of attack traffic related to
this attack. That would make this attack one of the largest ever
reported.

The challenge with attacks at this scale is they risk overwhelming the
systems that link together the Internet itself. The largest routers that
you can buy have, at most, 100Gbps ports. It is possible to bond more
than one of these ports together to create capacity that is greater than
100Gbps however, at some point, there are limits to how much these
routers can handle. If that limit is exceeded then the network becomes
congested and slows down.

Over the last few days, as these attacks have increased, we’ve seen
congestion across several major Tier 1s, primarily in Europe where most
of the attacks were concentrated, that would have affected hundreds of
millions of people even as they surfed sites unrelated to Spamhaus or
CloudFlare. If the Internet felt a bit more sluggish for you over the
last few days in Europe, this may be part of the reason why.

Attacks on the IXs

In addition to CloudFlare’s direct peers, we also connect with other
networks over the so-called Internet Exchanges (IXs). These IXs are, at
their most basic level, switches into which multiple networks connect
and can then pass bandwidth. In Europe, these IXs are run as non-profit
entities and are considered critical infrastructure. They interconnect
hundreds of the world’s largest networks including CloudFlare, Google,
Facebook, and just about every other major Internet company.

Beyond attacking CloudFlare’s direct peers, the attackers also attacked
the core IX infrastructure on the London Internet Exchange (LINX), the
Amsterdam Internet Exchange (AMS-IX), the Frankfurt Internet Exchange
(DE-CIX), and the Hong Kong Internet Exchange (HKIX).
From our perspective, the attacks had the largest
effect on LINX which caused impact over the exchange and LINX’s systems
that monitor the exchange, as visible through the drop in traffic
recorded by their monitoring systems. (Corrected: see below for original
phrasing.)

The congestion impacted many of the networks on the IXs, including
CloudFlare’s. As problems were detected on the IX, we would route
traffic around them. However, several London-based CloudFlare users
reported intermittent issues over the last several days. This is the
root cause of those problems.

The attacks also exposed some vulnerabilities in the architecture of
some IXs. We, along with many other network security experts, worked
with the team at LINX to better secure themselves. In doing so, we
developed a list of best practices for any IX in order to make them less
vulnerable to attacks.

Two specific suggestions to limit attacks like this involve making it
more difficult to attack the IP addresses that members of the IX use to
interchange traffic between each other. We are working with IXs to
ensure that: 1) these IP addresses should not be announced as routable
across the public Internet; and 2) packets destined to these IP
addresses should only be permitted from other IX IP addresses. We’ve
been very impressed with the team at LINX and how quickly they’ve worked
to implement these changes and add additional security to their IX and
are hopeful other IXs will quickly follow their lead.

The Full Impact of the Open Recursor Problem

At the bottom of this attack we once again find the problem of open DNS
recursors. The attackers were able to generate more than 300Gbps of
traffic likely with a network of their own that only had access 1/100th
of that amount of traffic themselves. We’ve written about how these
mis-configured DNS recursors as abomb waiting to go
off
that literally threatens the stability of the Internet itself. We’ve now
seen an attack that begins to illustrate the full extent of the problem.

While lists of open recursors have been passed around on network
security lists for the last few years, on Monday the full extent of the
problem was, for the first time, made public. The Open Resolver
Project made available the full list of
the 21.7 million open resolvers online in an effort to shut them down.

We’d debated doing the same thing ourselves for some time but worried
about the collateral damage of what would happen if such a list fell
into the hands of the bad guys. The last five days have made clear that
the bad guys have the list of open resolvers and they are getting
increasingly brazen in the attacks they are willing to launch. We are in
full support of the Open Resolver Project and believe it is incumbent on
all network providers to work with their customers to close any open
resolvers running on their networks.

Unlike traditional botnets which could only generate limited traffic
because of the modest Internet connections and home PCs they typically
run on, these open resolvers are typically running on big servers with
fat pipes. They are like bazookas and the events of the last week have
shown the damage they can cause. What’s troubling is that, compared with
what is possible, this attack may prove to be relatively modest.

As someone in charge of DDoS mitigation at one of the Internet giants
emailed me this weekend: “I’ve often said we don’t have to prepare for
the largest-possible attack, we just have to prepare for the largest
attack the Internet can send without causing massive collateral damage
to others. It looks like you’ve reached that point, so…
congratulations!”

At CloudFlare one of our goals is to make DDoS something you only read
about in the history books. We’re proud of how our network held up under
such a massive attack and are working with our peers and partners to
ensure that the Internet overall can stand up to the threats it faces.

Correction: The original sentence about the impact on LINX was “From
our perspective, the attacks had the largest effect on LINX which for a
little over an hour on March 23 saw the infrastructure serving more than
half of the usual 1.5Tbps of peak traffic fail.” That was not well
phrased, and has been edited, with notation in place.

Via Cloudflare.com