HPC interconnects: InfiniBand versus Intel Omni-Path

This is a hot topic nowadays, definitely in the HPC world. The leading InfiniBand solution versus the new proprietary product from Intel - Omni-Path. One can find massive information on what Omni-Path is (well, it was already announced at the 2015 supercomputing conference) but interesting enough, Intel did not release any applications performance so far. You can find some low level benchmarks such as network latency and bandwidth, but those numbers are actually similar between the two solutions, and actually do not reflect the main architectural differences between InfiniBand and Omni-Path. Just few days ago, Mellanox published an interesting article covering in details the main differences between InfiniBand and Omni-Path, and for the first time release some application performance comparisons.  Not surprising (still, Omni-Path is based on the QLogic True-Scale architecture) I guess, InfiniBand shows much higher applications performance even on a small cluster size - getting to 60% of performance advantage. It is an interesting article to read, and seems that InfiniBand will remain the top solution for HPC systems.
You can find the article at - http://www.hpcwire.com/2016/04/12/interconnect-offloading-versus-onloading/.

Co-Processors - Intel Knights Landing

In the past I wrote on NVIDIA GPUs, and on the GPU-Direct technology that enables a direct network communication between the NVIDIA GPU and the cluster interconnect (for example InfiniBand). Since then, there were several enhancement to the GPU-Direct technology, and I will try to cover that in future posts.

In this post I would like to give my view on the upcoming Intel Knights Landing Xeon Phi co-processor. Knights Landing is the next generation Xeon Phi after Knight Corner. Knights Corner was not a success story to say the least, and Intel aims to gain some attraction with the next generation Knight Landing.

Knights Landing has 72 cores, and more important it is a bootable device, so one can use Knights Landing as the main CPU in a server platform. This is a nice capability, as one can build/use a single CPU (Knight Landing) boards for example.

On the connectivity side, Knights Landing will be provided in two versions (packages) - KNL and KNL-F. KNL is the Knights Landing CPU, with two PCI-Express x16 and one PCI-Express x4 interfaces. The two x16 can be connected to any network solution. KNL-F is a packaged Knights Landing with two Intel Omni-Path ASICs, connected via PCI-Express x16 each to the KNL ASIC. There will be PCI-Express x4 connectivity out of the KNL-F package for management options.

KNL has being tested with InfiniBand and works great using the Open Fabrics OFED distribution, or the Mellanox OFED distribution. It has being implemented in several sites already.

KNL-F usage is questionable. It makes sense to Intel to try and lock users to their propriety interconnect product (which is no more than a new version of the QLogic TrueScale product), but why would one want to be locked down to OmniPath? Especially as Omni-Path requires the KNL to spend many expensive cores cycles to manage and operate the Omni-Path network (which means loss of KNL performance)?  TrueScale was not used with GPUs in the past due to its overhead, and Omni-Path, which is based on the same TrueScale architecture, is no better.

UCX – a Unified Communication X Framework for High Performance Communications

UCX is a new, an open-source, production grade, set of network APIs and their implementations for high performance communications for high-performance computing and data-centric applications.  UCX solves the problem of moving data memory across multiple type of memories (DRAM, accelerator memories, etc.) and multiple transports (e.g. InfiniBand, uGNI, Shared Memory, CUDA, etc. ), while minimizing latency, and maximizing bandwidth and message rate. The new communications framework supports all communication libraries (MPI, PGAS etc.) and enable a closer connection to the underlying hardware. A new community of supporters has been established behind the UCX efforts, which includes key participants from the HPC industry, laboratories and academia.  

 

UCX ensures there is very little software overhead in the communication path, allowing for near native-level hardware performance. To ensure the production quality of UCX, it was be co-designed, maintained, tested and used in production environments by leading proponents of the research community. In addition, UCX community is working on the UCX specification which will enable further extension of the UCX effort.

 

 As an open-source project licensed under the BSD-3 license, UCX is open for contributions from anyone in the industry. More information can be found at  www.openucx.com.

Intel OmniPath network - what happened to the performance numbers?

Intel has been talking on their new network products for high performance computing for quite some time. OmniPath is a new network based on the old technology Intel acquired from QLogic (aka TrueScale). As Intel officially announced OmniPath in the last supercomputing conference, I was waiting to see any real performance data using this network, but nothing was actually published so far (SC'15 was in Nov, March 2016 is around the corner...).

So, for now all I can do is to read what Intel publishes. But even this information is being changed over time. In July 2015 Intel published a chart claiming OmniPath do deliver 160 million messages per second, which is a very impressive number (and higher versus what InfiniBand can do today). But, but....wait, in the recent Linley Group data center conference (February 2016) Intel presented a slide that actually talks on a complete different set of numbers - they actually projected that the OmniPath message rate will be between 75 and 86 million, and said to actually measure 108 million (which is higher than what they have projected originally...). What happened to the 160 million number? 108 million is much lower versus InfiniBand, and is not that interesting to say the least.


 
 Intel July 2015 - 160 million messages per second

 

 Intel February 2016 - 108 million messages per second

Making treatment affordable and more accurate using supercomputers

A great example on how supercomputers can help improving our life. A new supercomputer in Denmark, dedicated to life science, will help analyzing the growing amounts of data we collect, to find better health treatments, safer, more accurate. And it is all based on standard technologies. Happy holidays!

https://www.youtube.com/watch?feature=player_detailpage&v=enJ4SEt2tDw&list=UUEMsq3TXw7EdVekkRBzW8nw

InfiniBand Performance Over Intel Haswell – First Numbers

As Intel just released the new CPU platform - official name Intel Xeon E5-2600 v3, code name “Haswell”, wanted to share some of our performance testing. Forr start, we tested the simple InfiniBand bandwidth and latency benchmarks one can find as part of the InfiniBand software distribution. We measured around 6.4 Giga Byte per second bandwidth and latency of close to 0.6 micro second. You can see the full graphs below. More to come J

Remote Direct Memory Access - RoCE versus iWARP

Remote Direct Memory Access (RDMA) is the technology that that allows server-to-server data communication to go directly to the user space (aka application) memory without any CPU involvement. RDMA technology delivers faster performance for large data transfers while reducing CPU utilization or overhead. It is a technology used in many applications segments – database, storage, cloud and of course HPC. All of the MPIs include support for RDMA for the rendezvous protocol.

There are three communications standards for RDMA – InfiniBand (the de-facto solution for HPC), RoCE and iWARP. The latter two are over Ethernet. RoCE has being standardized by the IBTA organization, and iWARP by the IETF.

iWARP solutions are being sold by Intel (due to the acquisition of NetEffect) and Chelsio. RoCE solutions are being sold by Mellanox, Emulex and others. The major issues of iWARP are performance and scalability. With iWARP, the data needs to pass through multiple protocols before it can hit the wire and therefore the performance iWARP delivers is not in par with RoCE (not to mention InfiniBand). The major RoCE limitation was with support over layer 3, but this has been solved with the new specification that is about to be released for RoCE v2.

Last week Intel announced their new Ethernet NICs (“Fortville”). No iWARP support is listed for these new NICs, and this leaves Intel without RDMA capability for their Ethernet NICs. Seems that the iWARP camp is shrinking… well…  there is a RoCE reason for it…

Supercomputer Simulations Help Gain Insight into New Cancer Treatment Technology

A recent release from the Texas Advanced Computing Center (TACC) sheds light on one of the research programs that is being supported, or better say enabled, by the TACC powerful supercomputer, one of the fastest machines in the world. Using supercomputer simulations on TACC's “Lonestar”  system, researchers are able to model radiation in a magnetic field, which will facilitate the safe use of the MRI-linac and enable more effective cancer treatment.

The research is being done by the MD Anderson Cancer Center in Houston.  According to the team working on it, the new solution they develop unites radiation therapy and magnetic resonance imaging (MRI), allowing physicians to view the cancer tumor in real-time and in high detail during treatment. It also permits physicians to adapt the radiation treatment during the procedure, sparing healthy tissue and reducing side effects.

To develop the system, the MD team utilize the TACC supercomputer to ran complex simulations. A great use for the supercomputing power. TACC system was build using the most flexible architecture of a cluster, a combination of CPUs and co-processors and InfiniBand for the connectivity. A great example of a standard based system and an example on why there is no reason to use proprietary products for supercomputers. You can read more on TACC systems at https://www.tacc.utexas.edu/resources/hpc. I enjoy using them too.

GPUDirect RDMA in Action

Last year I wrote about the release of the GPUDirect RDMA technology. Simply saying, this is the technology that enables direct communications between GPUs over the network (RDMA capable network) which translates into much higher performance for applications using GPUs – high performance applications, data analytics, gaming  etc. basically any application that run over more than a single GPU. If in the past every data movement from the GPU had to go through the CPU memory, with GPUDirect RDMA it is not the case anymore. The data will go directly from the GPU memory to the RDMA capable network (for example InfiniBand) – data latency is being reduced by more than 70%, data throughput is being increased by 5-6X and the CPU bottleneck is being eliminated.

The University of Cambridge and the HPC Advisory Council have released performance information of GPUDirect RDMA with one of their application on a nearly 100 servers system. The application is HOOMD-blue - a general-purpose molecular dynamics simulation created by the university of Michigan, that can be used over GPUs. Each server includes two GPUs and two InfiniBand RDMA adapters – so each GPU can connect directly to the network instead of going through the CPU and the QPI interface (a pair of GPU and network adapter is located on the same PCI-Express root complex). Bottom line, the GPUDirect RDMA technology enabled Cambridge to increase HOOMD-blue performance by 2X over the given system. Same system, same hardware, setting on GPUDirect RDMA, twice the performance….  Impressive.

New MPI (Message Passing Interface) Solution for High-Performance Applications

A good source for high-performance computing information is the HPC Advisory Council (www.hpcadvisorycouncil.com). Of course one can find many news sites - insideHPC, HPCWire and others, but the HPC Council is a good option if you want to get more in the details, to learn on new technologies and solutions being developed etc. One of the recent publications was a case study on STAR-CCM+ application (CFD) - http://www.hpcadvisorycouncil.com/pdf/STAR-CCM_Analysis_Intel_E5_2680_V2.pdf.

The publication included for the first time some performance information on a new MPI solution – called HPC-X. In the world of high-performance computing, MPI is one of the most used parallel communications library. There are some open source solutions, such as MPICH, MVAPICH, OpenMPI and commercial options – Platform MPI (formally known as HP MPI, now owned by IBM) and Intel MPI. HPC-X is a new solution from Mellanox which seems to be based on Open MPI plus various accelerations. In the past I did cover new releases of both MVAPICH and OpenMPI as these are the two most used solutions by us so far.

The combination of open source base and support does make HPC-X an very interesting solution for any HPC system. Of course, it must perform, as this is the most important item…  according to the new publication, HPC-X does provide a performance advantage over the other commercial options, up to around 20% at 32 server nodes cluster (dual socket servers). It is definitely a good start, and am looking forward to see further reports on HPC-X. Meanwhile we do plan to download and try ourselves. 

Cluster Topologies - Dragonfly

In the past I reviewed two clusters topologies - Fat Tree (CLOS) and Torus. The dragonfly is a hierarchical topology with the following properties: several groups are connected together using all to all links (i.e. each group has at least one link directly to each other group), the topology inside each group can be any topology, it requires non-minimal global adaptive routing and advanced congestion look ahead for efficient operation. Simply saying, a dragonfly topology is a two level (at least) topology where at the top level groups of switches are connected in a full graph. The internal structure of the groups may vary and be constructed as full graph, fat tree, torus, mesh, dragonfly and so on.

While many described dragonfly as a topology with one hop between groups, actually it is not correct and in many cases the network traffic will go over several hops before getting to the destination. The key to make dragonfly topology effective is to allow some pairs of end-nodes to communicate on a non-minimal route. It is the only way to distribute random group to group traffic. This represents a significant difference from other topologies. To support such routing a dragonfly system needs to utilize adaptive routing and to only send traffic on the longer paths only if congestion is impacting the minimal (hops) paths.

While dragonfly goal was to enable a higher bisectional bandwidth compared to torus topology (or similar) while reducing the overall costs (mainly cable lengths), in practice dragonfly does not provide such an advantage - nor performance or costs over fat tree topology for example. On the other hand it does add some sort of complexity with the need to enable adaptive routing. So far we prefer to use the fat tree option – either with a full bisectional bandwidth configuration or with some oversubscription options – depends on the targeted applications.

Here Comes The 100 Gigabit Per Second…

For some of the compute demanding applications, data throughput is a critical element in getting the needed performance. Throughput is of course important for storage performance and scalability, for checkpointing and other cases, which are do relevant to most applications. Today the fastest solution we can find is InfiniBand FDR. It enables bandwidth of 56 gigabit per second. Taking the overhead off, we are left with around 54 gigabit per second for actual data movement. Other options are QDR or 40 gigabit Ethernet. Ethernet is not what we use for our HPC systems. Too much performance overhead. 

One can claim that there are 100 gigabit ports on some Ethernet switches, but these are for network aggregation, not to the server. These ports actually use 10 lanes of 10 gigabit each. Less the desired 4 lanes approach.

We did see some announcements for real 100 gigabit HPC networks. The first InfiniBand 100 gigabit switch was announced back in June – not just higher throughput but also lower latency – so win on both sides. While no indications yet on when the 100 gigabit InfiniBand adapter will be out, the switch announcement hints that we are getting close to the 100 gigabit times.

The higher the bandwidth (typically) the higher the message rate. With InfiniBand FDR we already saw much higher message rate versus all the QDR options in the market – either from Mellanox or from Intel. The increase in message rate was greater than the bandwidth difference – therefore also due to the new architecture of the latest InfiniBand FDR adapters. We do base all of our system nowadays on FDR. Waiting for EDR….    

GPU Direct RDMA is Finally Out!

GPUDirect RDMA is the newest technology for GPU to GPU communications over the InfiniBand interconnect. GPUDirect RDMA enables a direct data transfer from the GPU memory over the InfiniBand network via PCI Pier-to-Pier (P2P). This capability introduced in the NVIDIA Kepler-class GPUs, CUDA 5.0 and the Mellanox InfiniBand solutions.

The importance of this capability is with bypassing the CPU for GPU communications (who needs the CPU…..), therefore a dramatic increase in performance. Finally after long time of waiting, the two companies mentioned above have demonstrated the new capability in the recent ISC’13 conference. Prof. Dhabaleswar K. (DK) Panda, Hari Subramoni and Sreeram Potluri from the Ohio State University presented at the HPC Advisory Council their first results with the GPU Direct RDMA – 70% reduction in latency! You can see the entire presentation at http://www.hpcadvisorycouncil.com/events/2013/European-Workshop/presentations/9_OSU.pdf. Seems that GE Intelligent Platforms already using the new technology - http://www.militaryaerospace.com/whitepapers/2013/03/gpudirect_-rdma.html, which is a great example of how the new capability can make our life better (or faster…). You can also read more on http://docs.nvidia.com/cuda/gpudirect-rdma/index.html.



In the graph: latency improvement presented by DK Panda

Amazon’s HPC cloud: not for HPC!

I came across an interesting article on Amazon HPC Cloud. As was reported recently, Cycle Computing built a 50,000-core Amazon cluster for Schrödinger, which makes simulation software for use in pharmaceutical and biotechnology research. Amazon and Cycle Computing made lot of noise around the HPC capability of EC2, how great it is for HPC applications and that Schrödinger is a great partner. 

When Schrödinger actually tried to run their simulations on the cloud, the results were not that great. Amazon EC2 architecture slows down HPC applications that require decent amount of communication between the servers, even at small scale. Schrödinger President Ramy Farid mentioned that they have successfully run parallel jobs on Amazon eight-core boxes, but when we tried anything more than that, they got terrible performance. Farid was using Amazon’s eight-core server instances, so running a job on 16 cores simultaneously required two eight-core machines. “Slow interconnect speeds between separate machines does become a serious issue”.

It is known that Amazon EC2 is not a good place for HPC application, and it will not change unless they actually build the right solution. But instead of the taking the right steps, Cycle Computing CEO Jason Stowe decided that the application is the fault…. and the application tested is an example for only 1% of the HPC applications and Amazon care for the other 99%. Jason, wake up! The applications tested are a good indication for many other HPC applications. Don’t blame the application or the user, blame yourself for building a lame solution. Deepak Singh, Amazon’s principal product manager for EC2 had also smart things to say - “We’re interested in figuring out from our customers what they want to run, and then deliver those capabilities to them. There are certain specialized applications that require very specialized hardware. It’s like one person running it in some secret national laboratory.” Deepak, you need to wake up too and to stop with these marketing responses. If you want to host HPC applications, don’t call every example a “secret national laboratory” and stop calling standard solutions that everyone can buy from any server manufacture, such as InfiniBand, “a very specialized hardware”.

Amazon is clearly not connected to its users, or potential users, and until they do try to understand what we need, the best thing is to avoid them. There are much better solutions for HPC clouds out there.

My Pick for the Best High-Performance Computing Conferences March-May 2012

There are many HPC related conferences and workshops one can choose to attend. Therefore, a conference that covers multiple topics and combines real hands-on with technical sessions is the one I prefer to take part and attend. I have tried to find the best HPC conferences in the next 3 months:

March: The HPC Advisory Council Switzerland Conference, March 13-15 (http://www.hpcadvisorycouncil.com/events/2012/Switzerland-Workshop/index.php). The location is in the beautiful city of Lugano, but more important is the agenda of course… the conference will cover all the major developments, and will include hands-on sessions. Definitely worth the travel!

April: Two options to pick - 2012 High Performance Computing Linux for Wall Street in New York or the IDC HPC User Forum in Richmond, VA. It is more sessions and opinions rather than technical focused. Both are decent options if you have time…

May: Two options again – the NVIDIA GPU Technology Conference in San Jose, CA or the IEEE International Parallel & Distributed Processing Symposium (IPDPS) in Shanghai, China. IPDPS is of course more technical and covers more subjects – so the better option.

June: The International Supercomputing Conference in Germany, no doubt…

New MPIs released – Open MPI 1.4.5 and MVAPICH2 1.8

In a very close timing, both the Open MPI group and the MVAPICH team released new versions of their open source MPIs. The Open MPI Team announced the release of Open MPI version 1.4.5. This release is mainly a bug fix release over the v1.4.4 release. Version 1.4.5 can be downloaded from the main Open MPI web site and it contains an improve management of the registration cache, a fix for SLURM cpus-per-task allocation, as well as some other bug fixes.

The MVAPICH team announced the release of MVAPICH2 1.8a2 and OSU Micro-Benchmarks (OMB) 3.5.1. The new features include support for collective communication from GPU buffers, non-contiguous datatype support in point-to-point and collective  communication from GPU buffers, efficient GPU-GPU transfers within a node using CUDA IPC, adjust shared-memory communication block size at runtime, enable XRC by default at configure time, new shared memory design for enhanced intra-node small message performance and SLURM integration with mpiexec.mpirun_rsh to use SLURM allocated hosts without specifying a hostfile. For downloading MVAPICH2 1.8a2, OMB 3.5.1, associated user guide,

quick start guide, and accessing the SVN, you can check http://mvapich.cse.ohio-state.edu.

Congrats for the teams for the new releases.

InfiniBand for the Home in Less Than $150 (10Gb Networking on the Cheap)

I came across Dave Hunt's Blog - http://davidhunt.ie/wp/?p=232, talking on using older generation of InfiniBand (10Gb) for the home. Think about your home entertainment system, Blu-ray streaming from a storage box to your PC and other usage models…

When it comes to Ethernet and 10Gbs, the price is in the sky, but you can get the same speed with InfiniBand for pennies.  Dave built a system that passes over 700MB/sec throughput between his PCs at home for under $150! As Dave says, that’s like a full CD’s worth of data every second…

From Dave blog – “So, I now have an InfiniBand Fabric working at home, with over 7 gigabit throughput between PCs. The stuff of high-end datacenters in my back room. The main thing is that you don’t need a switch, so a PC to PC 10-gigabit link CAN be achieved for under $150! Here’s the breakdown: 2 x Mellanox MHEA28-XTC InfiniBand HCA’s @ $34.99 + shipping = $113 (from eBay), 1 x 3m Molex SFF-8470 InfiniBand cable include shipping = $29. Total: $142”.

Next will be for me to bring InfiniBand into my home…

PCI-Express 3.0 is Finally Here…

Long time since my last post… You know how it is – new academic year, lots of preparations, going to Supercomputing 2011 in the freezing Seattle…

One of the new technologies that I was really waiting for is PCI Express 3.0. PCI Express 2.0 was released in 2008 and is the bottleneck whenever you use faster than 20Gb/s network - such as InfiniBand. PCI Express 2.0 was released in 2008, and it is about time to get the new generation out.

While sources said that the official release of PCI Express 3.0 will be in the March 2012 time frame, the first systems based on PCI Express 3.0 (and InfiniBand FDR!!) are already out there. If you were at SC’11, or if you monitor the news from the TOP500 list, you could hear (or read) on the new systems. One of them is the Carter supercomputer in Purdue University, which is said to be the US fastest campus supercomputer. Carter systems was ranked 54th on the November TOP500.org list and was built using the latest technologies from Intel, HP and Mellanox, including not-yet-released Xenon E-5 "Sandy Bridge" Intel processors, HP Proliant servers and the already-released InfiniBand FDR.

The folks from Purdue claim that "Carter is running twice as fast as the supercomputer we were using and is using only half of the nodes. That will allow us to scale our models for better forecasts." So higher performance at a lower operational cost. Great deal…

New MPIs Version Announced – Open MPI and MVAPICH (OSU)

This week, with one day difference, new MPI versions of Open MPI and MVAPICH were announced. On Thursday, the Open MPI team announced the release of Open MPI version 1.4.4. This release is mainly a bug fix release over the previous v1.4.3 release. The team strongly recommends that all users upgrade to version 1.4.4 if possible.

A Day after, On Friday, Prof. Dhabaleswar Panda from Ohio State University announced on behalf of the MVAPICH team the release of MVAPICH2-1.7 and OSU Micro-Benchmarks (OMB) 3.4. You can check both Open MPI and OSU websites for details on the new releases.

As part of the release, Prof. Dhabaleswar Panda has provided some performance results, and according to him, MVAPICH2 1.7 is being made available with OFED 1.5.4 and it continues to deliver excellent performance. OpenFabrics/Gen2 on Westmere quad-core (2.53 GHz) with PCIe-Gen2 and Mellanox ConnectX2-QDR (Two-sided Operations) provides 1.64 microsec one-way latency (4 bytes), 3394 MB/sec unidirectional bandwidth and 6537 MB/sec bidirectional bandwidth. QLogic InfiniPath Support on Westmere quad-core (2.53 GHz) with PCIe-Gen2 and QLogic-QDR (Two-sided Operations) provides 1.70 microsec one-way latency (4 bytes), 3265 MB/sec unidirectional bandwidth and 4228 MB/sec bidirectional bandwidth. Prof. Dhabaleswar Panda results clearly indicate that if you go with InfiniBand, Mellanox ConnectX-2 provides lower latency and much higher throughput. Clearly the performance winner.

Fibre Channel at Dead End, Better to Invest in Other Technologies

One of the most used solutions for storage connectivity is Fibre Chanel. Personally for high performance computing I prefer to use Lustre, but in my organization you can find enterprise class systems with enterprise storage networks, which were mainly Fibre Channel based.

As we evaluate every technology before we acquire new systems, I recently reviewed the options for an enterprise class storage solution. A quick history background: While Fibre Channel was created back than for general usage, it has become a storage networking solution. Fibre Channel is standardized in the T11 Technical Committee of the InterNational Committee for Information Technology Standards (INCITS).

In the early 2000s, Fibre Channel speed was at 2Gb/s, Ethernet was just getting into 1Gb/s and InfiniBand was not there. Having the storage capacity at 2x the communication capacity was a good reason for the Fibre Channel adoption. In the mid 2000s, Fibre Channel was at 4Gb/s, Ethernet at 1GigE and InfiniBand was moving into 20Gb/s. Nowadays, Fibre Channel is at 8Gb/s, Ethernet at 10Gb/s and InfiniBand at 56Gb/s. The next speed bump of Fibre Channel is 16Gb/s, Ethernet 40Gb/s and InfiniBand 100Gb/s. With this faster evolution of Ethernet and InfiniBand compared to Fibre Channel, there is clearly no reason to use Fibre Channel anymore and any investment in it for future deployments is clearly a mistake.

If we consider latency as another factor, the appearance of SSDs does not help Fibre Channel as well. The latency benefit of SSD is eliminated when Fibre Channel is being used, and the only options for SSD based storage are Ethernet or InfiniBand.

FCoE (Fibre Channel over Ethernet) is not the solution, and it seems to be just another try of the Fibre Channel vendors to extend the life time of the dying Fibre Channel storage. There is no reason to continue and use Fibre Channel. You better invest in getting Ethernet (iSCSI for example) or InfiniBand storage for your next system – higher throughput, lower latency, and better economics. I also believe that iSCSI deployments are on the increase while Fibre Channel on the decrease. Yet another proof point.