HERO, funded by the German Research Foundation (DFG) and the Ministry of Science and Culture (MWK) of the State of Lower Saxony, is a multi-purpose cluster designed to meet the needs of compute-intensive and data-driven research projects in the main areas of

• Quantum Chemistry and Quantum Dynamics,
• Theoretical Physics,
• the neurosciences (including hearing research),
• Oceanic and Marine Research
• biodiversity, and
• Computer Science

Like its sister cluster FLOW, HERO is operated by the IT Services of the University of Oldenburg. The system is used by more than 20 research groups from the Faculty of Mathematics and Science, and a couple of research groups from the Department of Computing Science at the School of Computing Science, Business Administration, Economics and Law.

Hardware Overview

• 150 compute nodes (1,800 CPU cores, 19.1 TFlop/s theoretical peak, 4.1 TB main memory)
  • 130 ‘standard’ nodes
    • IBM System x iDataPlex dx360 M3 server (12 cores, 24 GB DDR3 RAM, SATA-II 1 TB HDD)
    • Intel Xeon Processor X5650 (“Westmere-EP”, 6 cores, 2.66 GHz, 12 MB cache, max. memory speed 1333 MHz, QPI 6.4 GT/s, TDP 95 W)
  • 20 "big" nodes
    • IBM System x iDataPlex dx360 M3 server (12 cores, 48 GB DDR3 RAM) with a storage expansion unit (8 SAS 300 GB HDDs, 15k RPM, 6 Gbps, configured as RAID-0 with a gross capacity of 2.4 TB)
    • The CPU is the same as in the “standard” nodes
    • The “big” nodes are intended for jobs that require large amounts of memory or high I/O performance (for reading and writing local scratch files), or both.
• Shared-Memory Component (120 cores, 1.3 TFlop/s theoretical peak, 640 GB main memory)
  • SGI Altix UV 100 system (20 sockets, 640 GB DDR3 RAM, SAS 300 GB system disk) with an additional storage unit (20 SAS 600 GB HDDs, 15k RPM, 6 Gbps, configured as RAID-0 with a gross capacity of 12 TB)
  • Intel Xeon Processor X7542 (“Nehalem-EX”, 6 cores, 2.66 GHz, 18 MB cache, max. memory speed 1066 MHz, QPI 5.86 GT/s, TDP 130 W)
  • Proprietary NUMAlink 5 interconnect, which outperforms InfiniBand in terms of both latency and bandwidth
  • The Altix UV 100 system is designed for jobs with extreme memory requirements or for highly communication-intensive jobs (or for jobs with a combination of both requirements). As some of these jobs are also very I/O-intensive, the system is equipped with a high-performance RAID (Level 0) for fast read and write access to local scratch files.
• Some of the components from the older GOLEM Beowulf cluster, which has been succeeded by HERO, have been integrated into the latter. This adds a further 57 compute nodes (AMD Opteron dual-core and quad-core CPUs, totalling 288 cores, 600 GB of main memory, and a theoretical peak performance of approximately 1.9 TFlops) to HERO.
• Management and Login Nodes
  • 2 master nodes in an active/passive high-availability (HA) configuration
    • IBM System x3550 M3 server (8 cores, 24 GB DDR3 RAM, storage: 4 SAS 300 GB HDDs, 10k RPM, 6 Gbps, configured as RAID-10)
    • Intel Xeon Processor E5520 (“Westmere-EP”, 4 cores, 2.4 GHz, 12 MB cache, max. memory speed 1066 MHz, QPI 5.86 GT/s, TDP 80 W)
    • The master nodes are shared between HERO and its sister cluster, FLOW, and run all vital cluster services (node provisioning, DHCP, DNS, LDAP, NFS, Job Management System, etc.). They also provide monitoring functions for both clusters (with automated alerting). Monitoring covers hardware components (health status of all servers, temperature, power consumption, etc.) as well as basic cluster services (with automated restart should a service fail).
  • 2 login nodes for user access to the system, software development (programming environment), and job submission and control
    • IBM System x3550 M3 server (8 cores, 24 GB DDR3 RAM, storage: 2 SAS 146 GB HDDs, 10k RPM, 6 Gbps, configured as RAID-1)
    • The CPU is the same as in the master nodes
• Internal networks
  • Node Interconnect: Gigabit Ethernet (“MPI network”) with a 10 Gb Ethernet backbone network (non-blocking islands comprising 96 nodes)
  • Secondly, a physically separate Gigabit Ethernet (“base network”) for vital cluster services (node provisioning, DHCP, DNS, LDAP, NFS, Job Management System, etc.)
  • 10Gb Ethernet backbone network connecting the management and login nodes, the storage system, and the Gigabit Ethernet (MPI and base network) leaf switches
  • A dedicated IPMI network for hardware monitoring and control, including access to the VGA console (KVM functionality), enabling full remote management of the cluster
• Storage System
  • Enterprise-class scalable NAS cluster (manufacturer: EMC Isilon), 180 TB raw capacity, 130 TB net capacity (for the chosen redundancy level), IOPS NFS/CIFS (SpecSFS 2008) 18075 / 32279, InfiniBand backend network, two Gigabit Ethernet and two 10Gb Ethernet front-end ports per storage node.
  • The storage system is shared between HERO and its sister cluster, FLOW. Disk space is allocated to the two clusters depending on the proportion of the storage system’s hardware that was funded from the FLOW and HERO project budgets, respectively.

System Software and Middleware

• All cluster nodes are running Scientific Linux
• Cluster management: Bright Cluster Manager
• Commercial compilers, debuggers and profilers, and performance libraries: Intel Cluster Studio, PGI
• Workload management: Sun Grid Engine (SGE)

Selected applications running on HERO

(Due to licensing restrictions, some applications are only accessible to specific users or research groups.)

• Quantum chemistry packages: Gaussian 09, MOLCAS, MOLPRO, VASP
• LEDA – a C++ class library for efficient data types and algorithms
• MATLAB, including the Parallel Computing Toolbox
• FVCOM – The Unstructured Grid Finite Volume Coastal Ocean Model

Pictures