We have also updated our web-based IPv4 address browser to provide information about to what organizations each address block is allocated. The map now visualizes the whois allocation data; we thank the five regional internet registries for sharing this data with us and authorizing this visualization.

Finally, our web-based IPv4 address browser now has better time travel, with nearly 30 different census from Dec. 2005 to Nov. 2010, and we continue to update the map regularly.

Data collection for this work is through the LANDER project, and the map browser improvements are due to AMITE, both supported by DHS. Video preparation was supported by these projects and NSF through the MADCAT project.

The paper “Selecting Representative IP Addresses for Internet Topology Studies” (available at http://www.isi.edu/~xunfan/research/Fan10a.pdf) was accepted to appear at the ACM Internet Measurement Conference 2010 in Melbourne, Australia.

From the abstract:

Although the Internet is widely used today, we have little information about the edge of the network. Decentralized management, firewalls, and sensitivity to probing prevent easy answers and make measurement difficult. Building on frequent ICMP probing of 1% of the Internet address space, we develop clustering and analysis methods to estimate how Internet addresses are used. We show that adjacent addresses often have similar characteristics and are used for similar purposes (61% of addresses we probe are consistent blocks of 64 neighbors or more). We then apply this block-level clustering to provide data to explore several open questions in how networks are managed. First, we provide information about how effectively network address blocks appear to be used, finding that a significant number of blocks are only lightly used (most addresses in about one-fifth of /24 blocks are in use less than 10% of the time), an important issue as the IPv4 address space nears full allocation. Second, we provide new measurements about dynamically managed address space, showing nearly 40% of /24 blocks appear to be dynamically allocated, and dynamic addressing is most widely used in countries more recent to the Internet (more than 80% in China, while less than 30% in the U.S.). Third, we distinguish blocks with low-bitrate last-hops and show that such blocks are often underutilized.

Citation: Xue Cai and John Heidemann. Understanding Block-level Address Usage in the Visible Internet. In Proceedings of the ACM SIGCOMM Conference , p. to appear. New Delhi, India, ACM. August, 2010. <http://www.isi.edu/~johnh/PAPERS/Cai10a.html>.

We just published a new technical report “Selecting Representative IP Addresses for Internet Topology Studies” (available at ftp://ftp.isi.edu/isi-pubs/tr-666.pdf) .

screenshot of browsing RTTs in the Internet

FIND ME: To locate any host on the map, click in the IP address address box (at the top right) and type in a hostname. A pushpin will appear at that address, with a bubble indicating the hostname and IP address, and the map will scroll to the location. No more manually finding addresses!

MULTIPLE VIEWS allow users to flip between different data types, census dates, source locations:

1. DATA TYPES: We now plot round-trip times in addition to prior ping responsiveness. See how far away the Internet is! (At least from our probing sites.)
2. CENSUS DATES: We currently plot five datasets from Nov 2006 to June 2009. Travel through time to see the Internet of yesteryear!
3. SOURCE LOCATIONS: We collect data from two different locations: Los Angeles and Colorado State University, to help understand if we have observation bias. See the Internet from sea level, or a mile high!

To select different views, click the +-sign on the right of the screen and pick from the menus.

Data collection for this work is through the LANDER project http://www.isi.edu/ant/lander/, and the visualization improvements are due to AMITE http://www.isi.edu/ant/amite/, both supported by DHS.  We thank OpenLayers.org for the customizable front-end.

The abstract summarizes the tech report:

Although the Internet is widely used today, there are few sound estimates of network demographics. Decentralized network management means questions about Internet use cannot be answered by a central authority, and firewalls and sensitivity to probing means that active measurements must be done carefully and validated against known data. Building on frequent ICMP probing of 1% of the Internet address space, we develop a clustering algorithm to estimate how Internet addresses are used. We show that adjacent addresses often have similar characteristics and are used for similar purposes (61% of addresses we probe are consistent blocks of 64 neighbors or more). We then apply this block-level clustering to provide data to explore several open questions in how networks are managed. First, the nearing full allocation of IPv4 addresses makes it increasingly important to estimate the costs of better management of the IPv4 space as a component of an IPv6 transition. We provide about how effectively network addresses blocks appear to be used, finding that a significant number of blocks are only lightly used (about one-fifth of /24 blocks ha
ve most addresses in use less than 10% of the time). Second, we provide new measurements about dynamically managed address space, showing nearly 40% of /24 blocks appear to be dynamically allocated, and dynamic addressing is most widely used in countries more recently to the Internet (more than 80% in China, while less then 30% in the U.S.).

Xue Cai and John Heidemann. Understanding Address Usage in the Visible Internet. Technical Report N. ISI-TR-2009-656, USC/Information Sciences Institute, February, 2009. http://www.isi.edu/~johnh/PAPERS/Cai09b.html

The article provides a nice summary of the issues, but it reaches a conclusion that is stronger supported by the study.  The subhead of the article is “A survey shows that addresses are not running out as quickly as we’d thought”, and the article draws the conclusion: “the problem [of IPv4 address exhaustion] may not be as bad as many fear.”

The article’s conclusion, I think, overly simplifies matters—it is only true if the “better things we should be doing in managing the IPv4 address space” are free. The Internet Census we carried out supports the opportunity for better IPv4 address space management.  But an open question is the cost of such management.  Historically, with plentiful IPv4 addresses, IPv4 management costs have been small, but potential better IPv4 management will likely be much more costly.  This cost of ongoing IPv4 management needs to be weighed against the costs of one-time conversion cost to IPv6 coupled followed lower IPv6 management costs.

To me, one exciting conclusion from the Internet Census we carried out is that we now have data that allows us to start evaluating these trade-offs.  The answer may be more careful IPv4 gets us a few years, or that the cost of more careful IPv4 makes IPv6 an obvious choice.  In either case, resolving this transition is important for the Internet community.

A preprint is available at http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html, and an extended version is available as an updated technical report at http://www.isi.edu/~johnh/PAPERS/Heidemann08a.html.

Citation: John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, pp. 169-182. Vouliagmeni, Greece, ACM. October, 2008. http://doi.acm.org/10.1145/1452520.1452542

We have three recent updates, a new TECHNICAL REPORT and a BROWSABLE INTERNET ADDRESS MAP, and a PROJECT BLOG.

We have have released a new TECHNICAL REPORT describing the methodology,  ISI-TR-2008-649 at <http://www.isi.edu/~johnh/PAPERS/Heidemann08a.html>.
This report should completely supersede our previous report (#640), adding:

• evaluation in ping accuracy, both absolutely and relative to TCP probing
• estimation of error in our evaluations of hosts and server counts
• validation of our approach to firewall detection
• significant improvements in organization and presentation

We have also put up a BROWSABLE INTERNET ADDRESS MAP at <http://www.isi.edu/ant/address/browse/>.

With the Google maps engine, this map lets you zoom from an overview to any part of the address space, including showing individual hosts (permuted for anonymization).

Finally, we now have a PROJECT BLOG to allow folks to track future developments: <http://www.isi.edu/ant/blog/>.  We plan to do all future announcements via the blog rather than with general e-mail messages, so folks can opt-in to what they want to hear.

We welcome any comments about the map or technical report, either to our group mailing list (ant, then at isi.edu), or to individuals.

-ANT folks (John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, Joseph Bannister)