BEGIN:VCALENDAR
PRODID:-//Microsoft Corporation//Outlook MIMEDIR//EN
VERSION:1.0
BEGIN:VEVENT
DTSTART:20121113T230000Z
DTEND:20121113T233000Z
LOCATION:255-EF
DESCRIPTION;ENCODING=QUOTED-PRINTABLE:ABSTRACT: Hadoop Distributed File System (HDFS) acts as primary storage of Hadoop and has been adopted by reputed organizations (Facebook, Yahoo! etc.) due to its portability and fault-tolerance. The existing implementation of HDFS uses Java-socket interface for communication which delivers suboptimal performance in terms of latency and throughput. For data-intensive applications, network performance becomes key component as the amount of data being stored and replicated to HDFS, increases. In this paper, we present a novel design of HDFS using Remote Direct Memory Access (RDMA) over InfiniBand via JNI interfaces. Experimental results show that, for 5GB HDFS file writes, the new design reduces the communication time by 87% and 30% over 1Gigabit Ethernet (1GigE) and IP-over-InfiniBand (IPoIB), respectively, on QDR platform (32Gbps). For HBase, the Put operation performance is improved by 20% with our new design. To the best of our knowledge, this is the first design of HDFS over InfiniBand networks.
SUMMARY:High Performance RDMA-Based Design of HDFS over InfiniBand
PRIORITY:3
END:VEVENT
END:VCALENDAR