USC/ISI ANT Datasets and Dataset Formats

ANT provides a number of datasets in different formats.

Getting Datasets

See our separate datasets requests page for steps to take to get access to our data,

Alternatively go straight to our full list of our datasets, or see below for the categories of data that we have and the formats they are in.

Dataset Categories

Our datasets come from several different categories of data types:

Address Space Allocation Data

Address Space Allocation Data contains Internet addresses that have some properties that characterize Internet topology (for example, addresses that respond with different codes, or that appear to be dynamic, etc.). The IP addresses in this dataset are not typically anonymized because they are determined from measurement traffic and not actual sender-receiver communications and so are not associated with specific individuals. This data can be used to better understand the Internet topology and address usage.

Specific sub-categories of address space allocation data include:

• Internet address censuses (format): we ping every allocated IP address one time over about two months and report the results. A census is useful for studying address allocation policies. We have collected 2 censuses every 2-4 months since 2006. (ex: internet_address_survey_it39w-20110322 was the first census after all IPv4 blocks were allocated.)

• Internet address surveys (format): we pick 1-2% of the /24 blocks in allocated Internet address space and ping every address in those blocks many times (approximately every 11 minutes) for up to two weeks. An address survey is useful for studying address usage (for example, static vs. dynamic allocation). We have collected 2 surveys every 2-4 months since 2006. (Example: internet_address_survey_reprobing_it43j-20111012 is our first survey from Japan.)

• IP Address Hitlists (format): A hitlist provides a representative for each /24 IPv4 address block that is judged mostly likely to reply to pings. Our goal is to provide representatives that are responsive, complete and stable: that is, addresses that respond to pings and traceroutes (with high probability), that cover every allocated IPv4 /24 prefix, and that do not change much over time. Our hitlists are derived from our census data. Hitlists are useful input for Internet topology studies. We have generated hitlists every 2-4 months since 2009. (Example: internet_address_hitlist_it29w-20091102 was our first production hitlist.)

• IP Address History (format): New starting 2011-09, the IP address history contains all IPv4 addresses who ever responded to 16 prior IPv4 address censuses, along with a scores for each address about its likelihood to respond to future pings. Address histories are useful for doing customized selection of IP addresses for topology and reliability studies. (Ex: internet_address_history_it43w-20110913 was our first address history.)

• AS-to-Organization maps: we relate ASes (from BGP routing) to organizations (companies). (Example: as_to_org_mapping_inferred_truth-20100507)

We also have some Internet Topology data: Internet topology data is created by a program that tries to map the Internet. The program is able to determine which routers are capable of talking to other routers. Internet topology data only shows router connectivity within the Internet core and to external enterprise borders; it does not contain any identifiable information or internal enterprise topology information. This dataset can be used for worm outbreak modeling and simulation, worm containment and countermeasures, zombie distribution for DDoS attacks, vulnerability assessments, longitudinal studies of the evolution of Internet topology and address distribution, Internet topology and address map inference. (Example: internet_router_map_planetlab-20030412).

Traffic-related data

Much of our data is IP Packet Headers: These datasets are comprised of headers of traffic data, containing information such as anonymized source and destination IP addresses and other IP and transport (e.g., TCP, UDP, ICMP, SCTP) header fields. No packet contents are included. Depending on the specific dataset, this category of data can be used for characterization of typical Internet traffic, or of traffic anomalies such as DDoS attacks, port scans, or worm outbreaks.

• general anonymized packet headers: collections of packet headers from regional network access links (Example: lander_sample-20080903)

• anonymized attack traffic (packet headers): collections of packet headers containing known denial-of-service attacks taken from a regional network access link. (Example: attack-tcpsyn-20061106.)

• artificial attacks over real background traffic: we have generated a set of artificial attacks of varying intensity overlayed on real network background traffic (example: UniformAttack_Traces_Generated20070821-20041202)

We also have traffic flow data: Network traffic can represented as flows between two endpoints. This dataset contains traffic flow information, which includes a variety of attributes such as source and destination IP address, source and destination port, protocol type, and packet and byte counts. This data can be in different formats generated by a range of different collection tools such as netflow, IPFIX, and argus. IP addresses in these files are anonymized on a per-dataset or per-time interval basis. These datasets are useful for research such as network economics and accounting, network planning, analysis, security, denial of service attacks, network monitoring, as well as traffic visualization.

• general anonymized flows: collections of network traffic flow data, derived from packet headers at regional network access links (Example: long_flows_D1_2_days-20090605)

E-mail-related Data

We expect to have data related to Unsolicited bulk e-mail (UBE) or spam constitutes a significant fraction of all e-mail connection attempts and routinely frustrates users, consumes resources, and serves as an infection vector for malicious software. The collection and analysis of datasets in this category enable a wide range of research including: characterizing spam trends, detecting bots and development of spam mitigation algorithms. These datasets may include spam logs collected at individual organizations, reputation lists data such as those provided by Spamhaus, SORBS, and others, and e-mails, including both headers and contents, captured at spam traps or otherwise specifically identified as spam. It may include IP addresses or e-mail addresses of suspected spammers and potentially known spam e-mail message contents. Datasets in this category may be anonymized.

Anycast Enumeration Data

Anycast enumeration datasets contain active probing information to DNS anycast services such as root DNS. Typically probes are made from many vantage points with the goal to enumerate all anycast nodes in the service. Anycast enumeration datasets are useful to understand the operational status and geographic reach of anycast services and nodes.

For detail of the dataset, please refer to the description page.

Internet Outage Data

Datasets in this category record information about Internet outages--address blocks that become unreachable. Typically outages are inferred from active probing. It may include /24 block-level outages over time, or lists of inferred outages that affect larger parts of the Internet. Outage data can be useful to understand Internet reliability.

For detail of the dataset, please refer to the description page.

Other types of data

• other paper-specific datasets: We have several other datasets specific to papers we have published, including p2p traffic detection, TCP SYNs, etc.

In addition, we document the formats of our datasets here:

• packet headers and attacks datasets: either tcpdump (pcap) format, or Endace ERF format
  • to read pcap: use wireshark or tcpdump. Some traces have vlan tags and will not read tcpdump with filtering without including a vlan option.
  • to read ERF: search for "ERF to pcap" (here) or use libtrace or wireshark's ERF module to read
• Internet address census/surveys: 24-byte binary format
• Internet router topology maps: format
• IP address hitlist format
• Long-lived Internet flow data: format

If you would like to request any of the datasets above, please send email to