Sustainable Development in Globally Distributed Systems

Fabian Bustamante 

Wednesdays@NICO Seminar, Noon, October 1 2008, Chambers Hall, Lower Level

Prof. Fabian Bustamante, Northwestern University

Abstract 

The past few years have witnessed a growing number of emergent globally-distributed systems. Examples of such services range from content distribution networks and peer-to-peer file sharing, to voice-over-IP to gaming. Most of these systems are built under the assumption that no information is available about the underlying network. Thus, each of them regularly and independently probes its environment as it attempts, for instance, to identify better paths, route around problematic links or preferentially connect nodes that are near each other. This approach has a number of clear drawbacks – from the obvious redundancy and the difficulties with reusing the gathered information in other applications, to its suboptimal results and unnecessary additional complexity. My work builds on the thesis that most distributed systems can reduce their aggregated control and administrative overhead by strategically reusing the view of the network gathered by other long-running, ubiquitous services such as content distribution networks and peer-to-peer systems. We have applied our ideas to identify indirect, high-performance paths in the Internet, provide relative network positioning and reduce the networking impact of P2P file sharing services like BitTorrent. We are currently exploring its potential as the basis for an early warning system for Internet problems that reuses P2P natural traffic. In this talk, I will discuss these efforts in the context of our approach to sustainable scalability in Internet-distributed systems.