Open Source Gets Another Hit? OpenFOAM acquired by SGI

One of the driving forces behind HPC clusters is the open source or open community. I said that multiple time before. In particular, the existence of open source applications helps extending the reach of HPC to more places and creates the community to progress the development and take advantage of higher scaling and new hardware products. One of the more interesting open source applications is or was OpenFOAM. OpenFOAM is computational fluid dynamic (CFD) simulation software that has become a good alternative to the commercial solution, and a good tool for universities research activities.

Couple of weeks ago, OpenFOAM was acquired by SGI and the entire OpenCFD team has joined SGI as full-time employees. SGI said that they would provide “the market's first fully integrated CFD solution, where all the hardware and software work together” which is the same concept we saw in other software-hardware acquisitions. While SGI claims to continue and distribute free versions of OpenFOAM, my fear is that OpenFOAM will repeat previous similar cases, where over time the free versions become useless, and the application becomes yet another commercial option. The open source, open community, OpenFOAM software was a great thing, and served as the base for several derivatives for other simulations cases. We will need to wait and see what will be the future of it.  

Clustering Topologies – Torus

The torus network topology is a graph that consists of a mesh and additionally wrap-around edges. A torus topology implies that each switch is being connected in multiple directions to other switches. 3D torus means that the communication takes places in 6 different directions: X+, X-, Y+, Y-, Z+, Z-. Each switch junction is connected to 6 other switches, and the graph of the connections resembles a tri-dimensional matrix. In the case of 2D torus there are 4 connections, and in cases of higher dimensions (4D, 5D, 6D) there will be more connections between the switch junctions.

 The torus configuration makes it easier to support future system extensions, and it is a good fit for applications where the majority of the communications are local and do not spread across the entire system (for example LQCD). On the other hand, Torus is an oversubscribed fabric, and the worse case latency between two arbitrary nodes is much higher compared to a Fat Tree topology. 

In general, for low-mid size systems, fat-tree is my preferred option. Even for large node count, I would go with fat tree, but this is where economics plays its role and for the right applications torus can be the better, more cost effective option.

Clustering Topologies – Overview – and Fat Trees

Building a high-performance computing cluster is not a too much complicated task today. The variety of open source tools and commodity building blocks have made the task of building HPC systems an easier one compared to years ago. One item that one needs to decide when coming to build HPC cluster is its topology, or in other words, how to connect the servers together (the network topology). There are several topologies that are more common than others – Fat-Tree, Mesh, Torus, Hypercube, Butterfly, Dragonfly etc.

If the system size in discussion fits a single network switch, than the decision is very simple… but when you need to use multiple network switches, the topology decision need to be made. Fat-tree, hypercube, and the *fly topologies can be design as oversubscribed or non-oversubscribed topologies, while Torus is typically an oversubscribed topology.

I would say that Fat-Tree is probably the easiest solution to use. Fat-trees are a class of network topologies that were shown to scale their performance with the networking resources. The evolution of fat-trees started with a single common root node. A tree in which each switch is serving communication between all its branches is shown in the figure below. Fat-trees use higher bandwidth links when approaching the tree roots (the fat-tree name stems from the fact that links grow fatter when approaching the root). In order to make the fat tree concept a feasible concept for large scale networks, a family of fat-trees named k-ary-n-tree was defined. A k-ary-n-tree has N levels made out of KN-1 switches, each having K ports going down the tree and K going up the tree. This topology trades off the practical hardware aspects by using switches of the same bandwidth and port count for all levels of the tree. Fat trees probably provide the lowest latency between any given compute pairs compared to other topologies, and if you are not hitting the issues of spanning trees (for example if you use InfiniBand), fat trees are very easy to build. In my scale of clusters, fat tree is also the most economical solution. If you need help to design your own cluster topology, drop me a note. In my future blogs I will discuss some other topologies.

Math Libraries – Overview

I would say that the most used Math libraries are Intel Math Kernel Library (Intel MKL), AMD Core Math Library (ACML), and in particular of BLAS there is the GOTO one.

Intel MKL latest release is the 10.3 release. In includes support for BLAS, LAPACK, Trust Region Solver, ScaLAPACK and Cluster FFT. If you are using Intel CPUs, MKL is a library that you would like to use. Mathematical domains supported by Intel MKL are sparse Linear Algebra—sparse BLAS, sparse format converters, PARDISO direct sparse solver, iterative sparse solvers and pre-conditioners; fast fourier transforms, LINPACK benchmark, vector math library and statistics functions - vector random number generators, summary statistics library. The first place to start is http://software.intel.com/en-us/articles/intel-mkl/.

If you have AMD based systems, AMD ACML can provide higher performance. ACML provides a free set of math routines. ACML consists of the following main components: A full implementation of Level 1, 2 and 3 basic linear algebra subroutines (BLAS), a full suite of linear algebra (LAPACK) routines, a suite of fast fourier transforms (FFTs) in both single, double, single-complex and double-complex data types and a random number generators in both single- and double-precision. For downloading ACML (latest version is 5.0) - http://developer.amd.com/libraries/acml/downloads/pages/default.aspx.

For BLAS, GOTO was one of the leading solutions. You can find the latest release on TACC web - http://www.tacc.utexas.edu/tacc-projects/gotoblas2/. GOTO has moved from TACC to Microsoft and it seems that unfortunately he stopped supporting or developing the GOTO BLAS library. If you have any news on this front, drop a comment.

The End of Bash Scripts?

Bash is a command processor, typically run as text commands, allowing the user to type commands which cause actions – start programs, check files, check machine status etc. In the HPC world, Bash scripts have become one of the main tools of administrators and users to help maintain, administer, and create clustered systems. The basis for these scripts is often shared in books, articles, and on public mailing lists. Tutorial on Bash programming can be found at: http://www.linuxdoc.org/HOWTO/Bash-Prog-Intro-HOWTO.html.

As described in details on ClusterMonkey by Douglas Eadline (http://www.clustermonkey.net), there is a battle now over Bash scripts in a Kansas courtroom. A Linux cluster vendor, Atipa Technologies is claiming all the Bash scripts they shipped to customers contain trade secrets and were stolen by former employees. Should this issue be decided in Atipa's favor, the fundamental idea of shared and open software could be blanketed by the simple claim of trade secrets. You can find the rest of the details (and Doug listed each one of them…) on Doug’s web site.

One of the driving forces behind HPC Clusters is the open source or open community. I hope that it stays like that moving forward. Open source does not mean free stuff, but it means the capability to share and develop together.The end of Bash scripts? I hope not...