johnh: research teaching PAPERS software other contact about

Census and Survey of the Visible Internet (extended)

John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevive Bartlett, and Joseph Bannister
USC/Information Sciences Institute

Abstract

Prior measurement studies of the Internet have explored traffic and topology, but have largely ignored edge hosts. While the number of Internet hosts is very large, and many are hidden behind firewalls or in private address space, there is much to be learned from examining the population of visible hosts, those with public unicast addresses that respond to messages. In this paper we introduce two new approaches to explore the visible Internet. Applying statistical population sampling, we use censuses to walk the entire Internet address space, and surveys to probe frequently a fraction of that space. We then use these tools to evaluate address usage, where we find that only 3.6% of allocated addresses are actually occupied by visible hosts, and that occupancy is unevenly distributed, with a quarter of responsive /24 address blocks (subnets) less than 5% full, and only 9% of blocks more than half full. We also show that only about 34 million (about 16% of responsive addresses) are very stable, while the rest are used intermittently, with a median occupancy of 81 minutes. Finally, we show that many firewalls are visible, measuring significant diversity in the distribution of firewalled block size. To our knowledge, we are the first to take a census of edge hosts in the visible Internet since 1982, to evaluate the accuracy of active probing for address census and survey, and to quantify these aspects of the Internet.

Availability

This paper is available in several formats: abstract web page with pointers and cites, PDF, paper copies can be obtained by mail to the authors. Copyright terms for this paper appear below.

Reference

Heidemann08a: John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevive Bartlett, and Joseph Bannister. Census and Survey of the Visible Internet (extended). Technical Report ISI-TR-2008-649, USC/Information Sciences Institute, February, 2008. <http://www.isi.edu/~johnh/PAPERS/Heidemann08a.html>.

@techreport{Heidemann08a,
	author = "John Heidemann and Yuri Pradkin and Ramesh
                         Govindan and Christos Papadopoulos and 
                         Genevive Bartlett and Joseph Bannister",
	title = "Census and Survey of the Visible Internet (extended)",
	institution = "USC/Information Sciences Institute",
	year = "2008",
	number = "ISI-TR-2008-649",
	month = "February",
	keywords = "census, survey, internet address space",
	url = "http://www.isi.edu/~johnh/PAPERS/Heidemann08a.html",
	pdfurl = "http://www.isi.edu/~johnh/PAPERS/Heidemann08a.pdf",
	myorganization = "USC/Information Sciences Institute",
	copyrightholder = "authors",
}

Copyright

This paper is copyright © 2008 by its authors. Permission to make digital or hard copies of part or all of this work for personal use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that new copies bear this notice and the full citation on the first page. Abstracting with credit is permitted.

To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission of the authors.