1. Introduction
2. Approach
2.1 Overlays
2.2 X-Bone
2.3 Features of X-Bone Overlays2.3.1 Overlay selection
2.3.2 Ability to recurse / layer
2.3.3 Dynamic routing
2.3.4 Robustness
2.3.5 Privacy and security2.4 DynaBone
Distributed Denial of Service (DDOS) attacks are becoming a more prevalent threat to network service [3]. There are a number of defenses deployed to resist DDOS attacks, including encryption of packets, firewalls to disable traffic from untrusted sources, and specific filtering of packets with known attack signatures [12]. Ultimately, the use of these defenses presents a trade-off between network service and security (Figure 1).
Figure 1. Spectrum of security and service, and its effect on system performance
Typically, individual DDOS solutions trade service for security, resulting in overall decreased network and CPU performance. Further, each single DDOS solution presents a target for subsequent attacks; the more successful or pervasive the solution or system, the more potential targets it presents, and the higher the payoff to the attacker. Clearly: Distributed attacks necessitate distributed defenses
A common example of distributed defense is layered security, where a combination of 'locks' together provides a stronger defense than any one individual lock. However such layering exacerbates the trade-offs in Figure 1; each layer of lock results in additional delays, CPU load, and decreased throughput. DynaBone provides an alternate approach, deploying a variety of different DDOS defenses in inner parallel overlay networks, and scattering packets on these overlays based on their defense status and throughput capability (Figure 2). The result is a dynamic backbone (a DynaBone) that provides DDOS resistance with increased performance and the ability to react to attacks.
Figure 2. DynaBone dynamic parallel overlays
DynaBone's parallel interior overlays, known as innerlays, provide multiple targets for DDOS attacks while providing a single, coherent network service as an outer overlay (an outerlay). The proactive/reactive multiplexer (PRM) uses attack statistics and performance monitoring to decide how to weight packet distribution among the innerlays. Innerlays under attack are disconnected (unused); the remaining innerlays are used in proportion to their performance.
DynaBone allows innerlays under attack to be disabled. Concurrent use of multiple innerlays for remaining traffic makes DDOS attacks more difficult, and allows overall service to degrade gracefully. Weaknesses of individual protocols or attacks on specific addresses undermine only one innerlay at a time; service is completely disabled only when all innerlays are attacked simultaneously. It is more difficult for service to be completely disabled, and service is automatically restored and reconstituted by shifting traffic to the remaining innerlays.
There are other, additional capabilities enabled by the DynaBone architecture. The use of different innerlays and nonuniform distribution of traffic could present information to an attacker, enabling them to interrupt service on the most heavily used innerlay. Although this does not completely compromise the overall architecture, it could result in slower restoration of original service. Because the innerlays are deployed in a coordinated fashion, traffic confidentiality techniques, e.g., hiding real packets in random packet streams, can be more easily deployed to defeat such information gathering. Other techniques, such as honeypot deployment, dynamic addition of new innerlays, and relocation of critical services are also enabled by this architecture.
Distributed denial-of-service (DDOS) attacks are an increasing threat to wide-scale network infrastructure. The simplest solution to these attacks is to disconnect the affected site from the network; this has the benefit of removing the attack, but the detriment of removing network service for that site. Disconnected hosts can be reconnected at other locations in the network; disconnected routers are somewhat more difficult to 'replace' or relocate.
DDOS defenses using disconnection vary in scope. They include 'air-gap' security, firewalls, and the use of multihoming to provide extra connections to sever [6]. DynaBone extends these paradigms, providing automated management of multi-connected hosts and routers, allowing DDOS responses to include disconnection from attacked networks without completely severing network connectivity.
DynaBone extends the X-Bone overlay deployment system to configure and coordinate a set of concurrent, parallel innerlays, and distribute traffic among them dynamically [21][23]. Internally, it applies feedback-controlled trunk grouping to distribute traffic across innerlay networks based on a variety of security, performance, and attack status parameters [2][13][24]. It applies a variety of innerlay configurations that have different performance and security metrics, providing configuration and algorithmic diversity.
The following sections of the approach provide an overview of overlay networking in general, the X-Bone overlay deployment system and its specific capabilities, and the architecture and features of the DynaBone.
An overlay network is an isolated virtual network deployed over an existing network. It is composed of hosts, routers, and tunnels. Tunnels are paths in the base network, and links in the overlay network. Hosts are packet sources or sinks, and routers are packet transits, as in conventional networks. Individual components (routers or hosts) can participate in more than one overlay at a time or in multiple ways (router, host) in a single overlay. Figure 3 shows an IP network (left); on that network, the X-Bone can deploy a ring (center) or star (right), by using various subsets of the nodes of the base network, connected by a set of tunnels. These tunnels determine the overlay topology, and may traverse multiple links in the base network, or a single link multiple times.
Figure 3. A ring (center) and a star (right) overlay deployed on a base network (left)
Overlays have three primary uses: containment, provisioning, and abstraction. Containment is the ability of an overlay to restrict the visibility of its contents. Tunneling encapsulates the packets of new protocol so it can be tested in a controlled environment. Containment was one of the first uses of overlays in the early 1980's, and motivated their re-emergence in the early 1990's for the M-Bone and later 6-Bone [7]. Tunnels allow incremental deployment, where (primarily) routers lacking new protocol capabilities can be skipped over (or through), avoiding the need for contiguous availability.
Provisioning uses reservation of components and capacity along tunnels to provide service guarantees to the overlay. Provisioned overlays can be used during emergencies to create virtual infrastructure when it is not feasible to deploy new physical resources. They can also limit the scope and impact of network experiments, e.g., limiting them to nominal use of surplus capacity.
Abstraction is a new use of overlay networks. Both provisioning and containment imply the interim the use of overlays that are supplanted by advanced hierarchical reservation in the former case, or more sophisticated dynamic service deployment in the latter. In these cases, overlays are a way to provide such capabilities without requiring contiguous deployment; once a new protocol or service is ubiquitous, tunnels (and thus overlays) can be avoided. However, abstraction remains a useful tool for education (networking classes), deploying testbeds, and simplifying applications. For example, a single lab can support a large number of concurrent experiments, each using a different topology. A testbed can be configured using a graphical user interface, in do what I mean style. Applications can request a deployed topology (e.g., ring) without needing to incorporate network management. In each case, manual intervention by a network manager is avoided, and applications and tools can be simplified.
The X-Bone is a system for the dynamic deployment and management of Internet overlay networks [21][23]. Overlay networks are used to deploy infrastructure on top of existing networks, to isolate tests of new protocols, partition capacity, or present an environment with a simplified topology. Current overlay systems include commercial virtual private networks (VPNs), IP tunneled networks (M-Bone, 6-Bone), and emerging research systems providing quality-of-service guarantees [7][18]. The X-Bone system provides a high-level interface where users or applications request DWIM (do what I mean) deployment, e.g.: create an overlay of 6 routers in a ring, each with 2 hosts. The X-Bone automatically discovers available components, configures, and monitors them.
Other overlay systems require OS and/or application modifications, restrict the number of overlays a router or host can participate in, or require manual component configuration [8][9][16][18]. The X-Bone provides automated deployment of overlays, coordinates their sharing of network components, and monitors deployed overlays. The X-Bone requires no specific OS or application modifications and only basic IP in IP encapsulation, and uses existing implementations of dynamic routing, name service, and other infrastructure [15]. Finally, the X-Bone is a uniform extension of the network to support overlays, and supports stacking (recursion) of overlays for fault tolerance and capacity sub-provisioning for experiments.
The X-Bone uses a two-layer tunnel mechanism, rather than the single layer used in conventional overlays. It is this two-layer scheme which supports stacked overlays, as well as permitting use of unmodified applications and network services inside a deployed overlay [21]. It also permits network resources (hosts, routers) to participate multiple times in a single overlay, and is the only known overlay system that integrates both IPsec support and dynamic routing [22].
The X-Bone is a distributed system composed of Resource Daemons (RDs) and Overlay Managers (OMs) , with a graphical user interface (GUI) and a more direct API. These components are shown in Figure 4.
OMs deploy overlays; a user creates an overlay by sending a request to an OM, either via a web-based GUI (Figure 5) or by sending a message directly to the OM API. Each overlay is coordinated by a single OM. Large overlays can be created by divide-and-conquer, where a single OM will fork sub-overlay requests to other OMs. Fault tolerance can be achieved by replicating state in multiple backup OMs. Both of these latter capabilities (recursion, fault tolerance) are supported in the X-Bone architecture, though not implemented in current releases.
Figure 4. X-Bone architectural components
Figure 5. X-Bone graphical user interface
An OM creates an overlay in phases, using multicast to discover available resources and TCP/SSL to configure and monitor resources [10]. The overlay request is translated to an invitation, and the invitation is multicast using UDP. An invitation indicates a set of simple conditions, e.g., a specific set of host operating systems, bandwidth requirements, etc. Invitations currently fit in a single UDP packet; where they do not, IP's automatic fragmentation and reassembly is utilized. Invitations are repeated with increasing TTLs until a sufficient number of invitees respond, or until a preset search limit is exceeded: i.e., an expanding ring search.
RDs are daemons that configure and monitor the resources of routers and hosts. RDs listen for multicast UDP invitations, and respond when their capabilities, available resources and permissions match. The RDs respond with unicast UDP messages, indicating their willingness to participate in an overlay, and their capabilities (protocol version, OS type, etc.). The OM selects a (currently arbitrary) subset from among the responding RDs , and opens TCP/SSL connections to each chosen RD. The OM determines configuration information, such as tunnel endpoint addresses and routing table entries, and sends specific configuration information to each RD. Figure 6 shows this configuration sequence. Once an overlay is deployed, the TCP/SSL connections are released. Subsequent overlay actions initiated by the OM include keep-alive pings, liveness and status requests, and modifying or dismantling configurations.
TCP/SSL secures the reliable configuration channel only; other mechanisms are needed to secure the multicast UDP invitations and unicast UDP responses. The X-Bone is currently applying S/MIME authentication to invitations, and S/MIME encryption to invitation responses to secure these UDP messages.
The X-Bone exhibits unique overlay capabilities, largely due to a combination of its focus on IP, and the use of two IP in IP encapsulation tunnels for each overlay link [15]. It allows overlays to be selected on a per-application basis and it supports recursive overlays (overlays on overlays). It also enables dynamic routing inside an overlay and provides fault recovery. Finally, it's architecture supports private and secure overlays, both in the invitations and in the configured overlays.
The X-Bone allows applications to be used unmodified inside overlays. On a host, an overlay is selected either directly by IP address, or indirectly by overriding the DNS resolver parameters of a process environment. A deployed overlay includes dynamically configured DNS entries for variants of the names of the participating components. For example, if blue.abc.com belongs to an overlay called apple, then a DNS near the OM (part of the X-Bone deployment) is updated with the name blue.apple.xbone.net as part of the overlay configuration.
Base component names remain the same; the DNS suffix of the Unix LOCALDOMAIN process environment parameter in each window differs. A standard network mapping utility can thus show different network views in different windows. The use of LOCALDOMAIN to set overlay-specific suffixes, and the DNS to subsequently determine IP addresses, allows unmodified applications to select either the base address or an overlay address for a given name prefix. An example of how the overlays and base network from Figure 3 would appear is shown in Figure 7.
Figure 7. User views of a network mapping utility; different views in different windows
X-Bone's tunneling supports layering an overlay on an overlay, known as recursion
or stacking. These recursive overlays are useful for managing groups of experiments
on a shared infrastructure. The shared components are gathered as a single overlay,
and individual experiments performed on overlays on top of that overlay.
Recursive overlays are also useful for managing fault tolerance. A single overlay
provides a layer of indirection, in which individual components can be replaced
or renamed without affecting the superior or inferior overlay layers. A set
of overlays at this middle layer can be exchanged to swap sets of resources.
The ability to support recursive overlays relies on a recursive tunneling mechanism;
the X-Bone's two-layer tunnels have been tested in several recursive layers.
It also relies on the X-Bone's preservation of true IP packet formats; because
it relies on an IP substructure and presents IP as the overlay network, X-Bone
overlays can be stacked without modification. Recursion depends on isolation
of the overlay management components; in the X-Bone, only the IP addresses inside
an overlay are visible to processes on the overlay, so OMs and RDs at one layer
are not visible to other layers.
The X-Bone's dual-layer tunnels allow existing dynamic routing (RIP, via gated or mrtd), multicast (via mrouted) and network diagnostic tools to be used inside an overlay, transparent to the base network. The X-Bone configures gated or mrtd to exchange RIP messages only among sets of interfaces belonging to an overlay. This has been used to deploy dynamic routing across non-cooperating administrative domains, where only the hosts involved need participate in the routing algorithms.
The X-Bone has a variety of fault detection and recovery mechanisms. Each X-Bone action (interface configuration, adding routes, configuring tunnels) has rollback recovery, and all state changes are logged to a state recovery file. The OM emits periodic heartbeat pings to refresh the state of the RD components. When a RD no longer hears from an OM (after several beats), all overlays of that OM are released from the RD state. Both RD and OM state are kept on disk and reloaded after reboots or restarts.
The X-Bone achieves security by the use of secure configuration channels, and the ability to deploy IPsec'd overlays. Secure configuration channels are provided by TCP/SSL using X.509 keys, as used for secure Internet web transactions [10]. The OM sends interface, route, IPsec, and tunnel configuration via these channels, which are opened only to RDs responding to the multicast invitations. The invitations and invitation responses use UDP, and cannot be secured with SSL; instead, S/MIME is used. Multicast invitations are typically only authenticated, whereas unicast responses are encrypted.
IPsec secures the data (IP packets) of the deployed overlay. This prevents packets from non-overlay components from interfering with an overlay, and ensures that overlay components can trust network-level packets, such as routing and ICMP messages.
Figure 8. Multi-layered overlays allows dynamic re-mapping, supporting fault tolerance
Consider our first example, of a base network on which various overlays are deployed (Figure 3). The X-Bone can deploy stacked overlays, such as three ring networks on the base network, and a star on one of those rings (Figure 8). When a fault is detected in one ring, the star can be remapped to a different ring. The challenge is deploying multiple rings that are known not to share physical resources. The X-Bone's layering provides a level of indirection to IP addressing in the star overlay, which allows it to be renumbered with respect to the base network, without renumbering within the star (the virtual network equivalent of virtual memory paging). It is this principle that enables DynaBone.
The DynaBone extends the X-Bone architecture to deploy a layered set of inner overlays (innerlays) together with a feedback and distribution proactive/reactive multiplexer (PRM), inside an outer overlay (outerlay). The result is a system of overlays that endures DDOS attacks, because an attack on any individual network can result in its being disconnected without substantially affecting the overall connectivity of the group (Figure 9). When the innerlays of a DynaBone are attacked, its PRM shifts traffic to the unaffected overlays (Figure 10).
Figure 9. DynaBone architecture, PRM weights towards simpler innerlays
Figure 10. Reacting to DDOS attack, PRM shifts traffic away from affected innerlay
There are two components of the DynaBone architecture: its use of layered overlays, and its feedback-based multiplexer (PRM). The layered overlays are based on a unique capability of the X-Bone system, to provide recursive overlay structure. The PRM enables traffic from the outerlay to be dynamically directed at the innerlays best able to provide network service.
Figure 11. Detail of the PRM
The PRM consists of a multiplexer, demultiplexer, monitor, and proactive/reactive control components (Figure 11). The multiplexer distributes outgoing traffic among the innerlays, and selects a specific innerlay per-packet or per-connection, or may eventually include FEC-style replication of packets across multiple innerlays at once [1]. It adds labeling information if needed, e.g., to reestablish order at the receiver or to coordinate FEC extraction. The demultiplexer gathers incoming packets, and may reestablish order, drop duplicates, or extract data from the FEC encoding.
The multiplexer distributes traffic based on a combination of tunnel management and interface to existing bandwidth reservation and allocation mechanisms. The tunnel management selects which tunnel is used, on a per-packet, per-connection, or other mechanism (to be determined). Bandwidth allocation is provided by a combination of external interfaces to ALTQ to limit bandwidth on particular interfaces, RSVP to reserve bandwidth on paths, and RSVP extensions to reserve bandwidths over tunnels (the latter, provided they are extended by ongoing efforts to handle layered tunnels) [4][20][25].
The monitor coordinates the analysis of the status of individual innerlays.
It may emit heartbeat messages itself, or invoke external mechanisms, such as
pathchar, (circle) to track the performance of innerlays.
All three components are coordinated by a proactive/reactive controller (dark
diamond in the figure). This controller determines how to configure the multiplexer
to distribute traffic, incorporating information on attacks from external detectors
(hexagon), as well as incorporating performance information from external systems
(circle) and the monitor.
The DynaBone's deployment of alternate parallel concurrent innerlays relies on the variety of network protocols and security algorithms available. Recent measurements on 700 MHz PCs running FreeBSD 4.2 indicate a substantial variation in the bandwidth and latency between conventional IP (far left), encryption (middle-left group), and authentication (middle-right group) (Figure 12 and Figure 13). For completeness, IP compression is also shown (far right). In addition to these objective performance metrics, there are subjective security strength metrics as well; stronger algorithms are shown to the right in each group.
Figure 12. Bandwidth comparison of various IPsec algorithms
Figure 13. Latency comparison of various IPsec algorithms
This shows that there is substantial diversity in the deployed IPsec algorithms. There is also diversity in available software for network configuration and management. There are a number of IP multicast (e.g., dense-mode PIM, sparse-mode PIM, DVMRP) and interior routing protocols (e.g., RIP, RIPv2, linkstate). There are also a number of versions and implementations of various network services, such as DNS (ISC Bind 4.x, 8.x, 9.x, non-ISC versions, etc.). Recent CERT advisories have shown that some DDOS attacks are highly specific to particular tools or even versions of these tools. Maintaining parallel overlays with alternates provides the opportunity to disable compromised versions dynamically.
DDOS attacks come in many forms, focusing on protocols, particular implementations of protocols, specific hosts, subnets, or entire networks. Firewalls are effective at controlling attacks on specific ports of machines, based on the source IP address. However, as network services become more sophisticated, it becomes more challenging for a firewall to filter traffic on port information alone. DynaBone's parallel innerlays provide opportunities for more effective traffic control - both for entirely virtual deployments, and where multiple physical networks exist. In a virtual system, the parallel innerlays allow firewalls to filter on destination IP address alone, rather than requiring potentially complicated parsing to filter on TCP or UDP port numbers (e.g., when the IP packet has options). In networks with multiply-connected components, DynaBone helps automate the deployment of parallel overlays running on the separate infrastructure.
DynaBone provides a way to combine individual network DDOS defenses into a coherent whole that is stronger than its parts. In most DDOS systems, the use of COTS components increases the likelihood of successful attack; in DynaBone, COTS results in a larger variety of innerlay configurations, strengthening the resulting deployed outerlays. In fact, by their nature, even a single winning DDOS solution can be the target of likely attack; DynaBone reduces this liability, and encourages the combined deployment of diverse solutions. DynaBone also allows the use of weaker DDOS defenses; in combination, a sufficiently large set of such defenses itself provides a strong defense, because it would fail only when all component innerlays were compromised at once.
DynaBone extends the notion of air-gap defense and firewalls to isolate individual innerlays when attacked. Firewalls are limited air-gap defenses, where particular ports or address combinations are blocked from services at a site (host or router). DynaBone extends that capability to disconnect whole networks, including their networking protocols and services, when they are attacked. When its multiple innerlays are deployed virtually (over tunnels), they enable more efficient firewalling based solely on IP addresses (of the innerlay that is disconnected) - rather than requiring complicated transport packet parsing. When multiple innerlays can be configured over physically distinct interfaces and paths, they afford the same protection as multihoming, but with automated configuration and control.
DynaBone is a distributed service that protects against distributed attack. It enables continuous operation when attacks are successful. By disconnecting the affected innerlays, it has the added benefit of negatively affecting the attacker. E.g., when DES is used for an encryption attack, all DES traffic is disabled (DES-based innerlays are shutdown). The result is that the attackers traffic has no network over which to travel, effectively pushing the attack back to the source. If the attacker uses a DES-based network, that network will suffer decreased service. However, at the same time, DynaBone overlays shunt traffic to non-DES innerlays, restoring service dynamically.
DynaBone supports graceful degradation, because an attack on each overlay compromises
only a portion of the outerlay's traffic. Recovery is provided passively, in
the use of a set of innerlays. Restoration is provided by detecting the DDOS
attack, identifying it with particular innerlays, and shunting traffic away
from those innerlays. DynaBone supports multiple kinds of reconstitution - where
traffic migrates to unaffected innerlays, services can migrate, and defensive
tactics (e.g., honeypots, traffic hiding) can be employed.
DynaBone provides these benefits to existing applications, without modifying
or augmenting existing operating systems or network protocols. Finally, DynaBone
extends X-Bone's coalition support, where these capabilities can be deployed
across 'administrative domain' boundaries.
[1] Bestavros, A., Kim, G., "TCP Boston: A Fragmentation-tolerant TCP Protocol for ATM Networks," Proceedings of Infocom'97, Kobe, Japan, April 1997.
[2] Braden, R., ed. "Requirements for Internet Hosts -- Application and Support," RFC-1123, Oct. 1989.
[3] Cert, "Denial-of-Service Developments," CERT Advisory CA-2000-01, Jan. 3, 2000. http://www.cert.org/
[4] Cho, K., "Managing Traffic with ALTQ," In Proceedings of USENIX 1999 Annual Technical Conference: FREENIX Track, Monterey CA, June 1999.
[5] Deception Tool Kit, http://www.all.net/dtk/
[6] Early, S., Anderson, R., "The XenoService - A Distributed Defeat for Distributed Denial of Service," Proc. IEEE Info. Survivability Workshop 2000, October 2000.
[7] Eriksson, H., "MBone: The Multicast Backbone," Communications of the ACM, Aug. 1994, pp.54-60.
[8] Fox, B., Gleeson, B., "Virtual Private Networks Identifier," RFC-2685, Sept. 1999.
[9] Gleeson, B., et al., "A Framework for IP Based Virtual Private Networks," (work in prog.), Feb. 1999.
[10] Hickman, Kipp, "The SSL Protocol," Netscape Communications Corp., Feb. 1995.
[11] KAME IPsec patches, http://www.kame.net
[12] Kent, S., Atkinson, R., "Security Architecture for the Internet Protocol," RFC-2401, Nov. 1998.
[13] Kung, H.T., Wang, S.Y., "TCP trunking: design, implementation and performance," Proc. ICNP 1999, Page(s): 222 -231.
[14] Lim, L., Gao, J., et al., " Customizable Virtual Private Network Service with QoS," Computer Networks (to appear 1Q2001).
[15] Perkins, C., "IP Encapsulation within IP," RFC-2003, Oct. 1996.
[16] Rodeh, O., Birman, K., Hayden, M., Dolev, D., "Dynamic Virtual Private Networks," TR98-1695, Dept. of Computer Science, Cornell University, Aug. 1998.
[17] Savage, S., Anderson, T., et al., "Detour: a Case for Informed Internet Routing and Transport," IEEE Micro, V19, N1, Jan. 1999, pp. 50-59.
[18] Scott, C., Wolfe, P., Erwin, M., Virtual Private Networks, O'Reilly & Assoc., Sebastapol, CA, 1998.
[19] Su, G., Yemini, Y., "Virtual Active Networks: Towards Multi-Edged Network Computing," Computer Networks (to appear 1Q2001).
[20] Terzis, A., Krawczyk, J., Wroclawski, J., Zhang, L., "RSVP Operation Over IP Tunnels," RFC-2746, Jan. 2000.
[21] Touch, J., "Dynamic Internet Overlay Deployment and Management Using the X-Bone," Computer Networks (1Q2001). Previously in Proc. ICNP 2000, pp. 59-68.
[22] Touch, J., Eggert, L., "Use of IPsec Transport Mode for Virtual Networks," (work in progress), Mar. 2000.
[23] Touch, J., Hotz, S., "The X-Bone," Proc. Global Internet Mini-Conference at Globecom, Nov. 1998.
[24] Traw, C., Smith, J., "Striping Within the Network Subsystem," IEEE Network, Vol. 9, No. 4, July/August 1995, pp. 22-29.
[25] Zhang, L., Deering, S., Estrin,
D., Shenker, S., and Zappala, D., "RSVP: A New Resource ReSerVation Protocol,"
IEEE Network, Sept. 1993.