Clustering Topologies – Overview – and Fat Trees

Building a high-performance computing cluster is not a too much complicated task today. The variety of open source tools and commodity building blocks have made the task of building HPC systems an easier one compared to years ago. One item that one needs to decide when coming to build HPC cluster is its topology, or in other words, how to connect the servers together (the network topology). There are several topologies that are more common than others – Fat-Tree, Mesh, Torus, Hypercube, Butterfly, Dragonfly etc.

If the system size in discussion fits a single network switch, than the decision is very simple… but when you need to use multiple network switches, the topology decision need to be made. Fat-tree, hypercube, and the *fly topologies can be design as oversubscribed or non-oversubscribed topologies, while Torus is typically an oversubscribed topology.

I would say that Fat-Tree is probably the easiest solution to use. Fat-trees are a class of network topologies that were shown to scale their performance with the networking resources. The evolution of fat-trees started with a single common root node. A tree in which each switch is serving communication between all its branches is shown in the figure below. Fat-trees use higher bandwidth links when approaching the tree roots (the fat-tree name stems from the fact that links grow fatter when approaching the root). In order to make the fat tree concept a feasible concept for large scale networks, a family of fat-trees named k-ary-n-tree was defined. A k-ary-n-tree has N levels made out of KN-1 switches, each having K ports going down the tree and K going up the tree. This topology trades off the practical hardware aspects by using switches of the same bandwidth and port count for all levels of the tree. Fat trees probably provide the lowest latency between any given compute pairs compared to other topologies, and if you are not hitting the issues of spanning trees (for example if you use InfiniBand), fat trees are very easy to build. In my scale of clusters, fat tree is also the most economical solution. If you need help to design your own cluster topology, drop me a note. In my future blogs I will discuss some other topologies.