I came across Dave Hunt's Blog - http://davidhunt.ie/wp/?p=232, talking on using older generation of InfiniBand (10Gb) for the home. Think about your home entertainment system, Blu-ray streaming from a storage box to your PC and other usage models…
When it comes to Ethernet and 10Gbs, the price is in the sky, but you can get the same speed with InfiniBand for pennies. Dave built a system that passes over 700MB/sec throughput between his PCs at home for under $150! As Dave says, that’s like a full CD’s worth of data every second…
From Dave blog – “So, I now have an InfiniBand Fabric working at home, with over 7 gigabit throughput between PCs. The stuff of high-end datacenters in my back room. The main thing is that you don’t need a switch, so a PC to PC 10-gigabit link CAN be achieved for under $150! Here’s the breakdown: 2 x Mellanox MHEA28-XTC InfiniBand HCA’s @ $34.99 + shipping = $113 (from eBay), 1 x 3m Molex SFF-8470 InfiniBand cable include shipping = $29. Total: $142”.
Next will be for me to bring InfiniBand into my home…
It's important to read the edits in the second paragraph of the original blog article. A couple of 7gbps netperf readings don't accurately reflect the real-world throughput. I'm struggling to get past 200MB/sec with disk subsystems, pci-express lane allocation, protocol selection, etc. IPoIB seems to max out at about 125MB/sec which is what you'd most likely use for a home network. SRP is the fastest, but point to point only. So, there's limitations. Might be good to add this information to the above...
ReplyDeleteYou definitely need the right storage system to get the higher bandwidth - but now it is the disks that you use, not the network as the bottleneck. As for IPoIB - in my datacenter I can get around 2GB/sec bandwidth with the newer InfiniBand adapters that include the Ethernet stateless offloads. With the older versions, I heard folks getting around 1.3GB/sec with latest drivers and probably a stronger CPU.
ReplyDeleteAgreed. I need to bite the bullet and upgrade that 6 year old motherboard/processor in the linux box. I was hoping to do this really cheaply, maybe I can get a donation from somewhere! :)
ReplyDeleteI got the speed up to about 900MB/sec using a ramdisk: http://davidhunt.ie/wp/?p=2324
ReplyDeletePCI-express slot choice is critial :)