Internet2 IPv6 Single Stream Land Speed Record

5.11 Gbps Geneva - US - Geneva

For submission to the Internet2 LSR

On January the 19th, 2005 Caltech and CERN transferred 357 GBytes in ten minutes over a distance of 14134 kms between Geneva - Chicago - Geneva through the CERN production network and the LHCnet network, using a single IPv6 stream.

Network setup

The following picture depicts the physical connections of the devices involved: physical map

The following picture depicts the logical connections (layer 3 connections):
logial map
The black lines are 10G ethernet connections, the blue ones are OC192.

Result

1 TCP/IPv6  streams  - Standard linux kernel 2.6.8  - Jumbo frames (9000 Bytes):

Tool used Throughput Duration iperf output TcpDump
iperf 5.11 Gbps
 10 minutes (600s) iperf_output The TCPdump file generated by this transfer is too large.
iperf 4.49 Gbps* 1 minute (60s)
 iperf_output TCPdump_file (206 MBytes of headers generated in 1 min); decoded

* the CPUs couldn't sustain the 5.11Gbps while running tcpdump

Bandwidth utilization graph of the LHCnet link

bandwidth

The link was used in both directions. The normal CERN production traffic to and from the US didn't affect the performance of the IPv6 stream. In the LHCnet backbone the production traffic is protected against bulk transfers with DiffServ QoS.

Internet2 Land speed record submission

    Geneva - Chicago - Geneva: 14134 (Distance measured with virtual GPS)
    Record submitted (IPv6 Single Strem class, and eventually Multi Stream): 7.222474E+16 bit-meter/sec (72224.74 Terabit-meter/sec)
    Rules and current records: http://lsr.internet2.edu/

Traceroute

emartell@W30gva> traceroute6 2001:1458:E000:2001::31
traceroute to 2001:1458:E000:2001::31 (2001:1458:e000:2001::31) from 2001:1458:ef11::30, 30 hops max, 16 byte packets
 1  2001:1458:ef11::62 (2001:1458:ef11::62)  0.323 ms  0.153 ms  0.144 ms
 2  2001:1458:ef10::56 (2001:1458:ef10::56)  116.434 ms  117.162 ms  116.857 ms
 3  ar5-chicago-datatag.ipv6.cern.ch (2001:1458:e100:1000::5:21)  116.527 ms  116.472 ms  116.466 ms
 4  cernh5-s1-0-0.ipv6.cern.ch (2001:1458:e000:1000::5:1)  232.595 ms  232.578 ms  232.606 ms
 5  2001:1458:e000:2001::31 (2001:1458:e000:2001::31)  232.426 ms  232.408 ms  232.45 ms

Ping

emartell@W30gva> ping6 2001:1458:E000:2001::31
PING 2001:1458:E000:2001::31(2001:1458:e000:2001::31) 56 data bytes
64 bytes from 2001:1458:e000:2001::31: icmp_seq=1 ttl=60 time=232 ms
64 bytes from 2001:1458:e000:2001::31: icmp_seq=2 ttl=60 time=232 ms
64 bytes from 2001:1458:e000:2001::31: icmp_seq=3 ttl=60 time=232 ms
64 bytes from 2001:1458:e000:2001::31: icmp_seq=4 ttl=60 time=232 ms
64 bytes from 2001:1458:e000:2001::31: icmp_seq=5 ttl=60 time=232 ms

End Systems used

w30gva and w31gva in the picture above were two servers with:

- Dual Opteron 250 (2.4 Ghz)
- Network adapter: Neterion (S2io) Xframe 10 Gigabit Ethernet Adapter
- Linux Kernel 2.6.8

End Systems configuration

PCI bus performance settings:

    setpci -d 17d5:5831 0x62.W=0x4f
    setpci -d 17d5:5831 LATENCY_TIMER=F8

Network interfaces settings:

    ifconfig eth3 txqueuelen 50000
    ifconfig eth3 mtu 9000

TCP settings:

    /proc/sys/net/ipv4/tcp_rmem = 65535 65536 10000000
    /proc/sys/net/ipv4/tcp_wmem = 65535 65536 10000000
    /proc/sys/net/ipv4/tcp_mem = 65535 65536 10000000
    /proc/sys/net/core/rmem_max = 536870912
    /proc/sys/net/core/rmem_default = 65535
    /proc/sys/net/core/wmem_max = 536870912
    /proc/sys/net/core/wmem_default = 65535
    /proc/sys/net/core/optmem_max = 536870912
    /proc/sys/net/core/netdev_max_backlog = 30000

Kernel settings:

    kernel 2.6.8 .config file

Routers and Switches

- r04gva: Cisco 7609 with Sup720, IOS 12.2(17d)SXB1
- r04chi: Cisco 7609 with Sup720, IOS 12.2(18)SXD2
- cernh5: Juniper T320, JunOS 6.2R3.10
- ar5-chicago: Juniper T320, JunOS 6.3R1.4

Contact persons

California Institue of technology (Caltech)
 Harvey Newman <newman@hep.caltech.edu>
Steven Low slow@caltech.edu
Julian Bunn <julian@cacr.caltech.edu>
Dan Nae <Dan.Nae@cern.ch>
Suresh Singh <suresh@cacr.caltech.edu>
Yang Xia <yxia@hep.caltech.edu>
Xun Su <xsu@hep.caltech.edu>
Sylvain Ravot <ravot@caltech.edu>
CERN
 David Foster <david.foster@cern.ch>
Olivier Martin <olivier.martin@cern.ch>
Paolo Moroni <paolo.moroni@cern.ch> 
Edoardo Martelli <edoardo.martelli@cern.ch>
Sverre Jarp (CERN OpenLab) <sverre.jarp@cern.ch>
Andreas Hirstius (CERN OpenLab) <andreas.hirstius@cern.ch>
National Center for Data Mining  (NCDM)
 Bob Grossman <grossman@uic.edu>
Michal Sabala <mike@lac.uic.edu>


Principal Investigators

caltech  cern


Projects

datatag  openlab

Major Sponsors

cisco  neterion  doe  nsf  eu

Supporting partners

NCDM
The National Center for Data Mining at the University of Illinois at Chicago recently launched the Teraflow Testbed, a high performance tesbed for mining remote and distributed data. The testbed consists of computer clusters in Chicago, Amsterdam, Geneva, Tokyo, and Kingston connected by 1 Gbps and 10 Gbps networks. Several different data sets are available, including astronomical data from the Sloan Digital Sky Survey, highway sensor data from the Gateway Project, and mass spec data from the Chicago Bio-informatics Consortium.
Edoardo Martelli - CERN, 1st February 2005