FLOW is one of the largest HPC clusters in Europe dedicated exclusively to wind energy research and is operated by the IT Services of the University of Oldenburg. It is funded by the Federal Ministry for the Environment, Nature Conservation and Nuclear Safety (BMU) through a programme aimed at increasing the share of renewables in Germany’s total electricity generation. The users of the cluster are

• the Complex Fluid and System Dynamics project group at the Fraunhofer IWES Institute.

The cluster is used for challenging computational fluid dynamics (CFD) calculations across a wide range of (spatiotemporal) scales, from individual blades and wind turbines, to the simulation of wind farms, to mesoscale and weather models.

Hardware Overview

• 193 compute nodes (2,288 CPU cores, 24.3 TFlop/s theoretical peak, 6.2 TB main memory)
  • 122 ‘small-memory’ nodes (diskless)
    • IBM System x iDataPlex dx360 M3 server (12 cores, 24 GB DDR3 RAM)
    • Intel Xeon Processor X5650 (“Westmere-EP”, 6 cores, 2.66 GHz, 12 MB cache, max. memory speed 1333 MHz, QPI 6.4 GT/s, TDP 95 W)
  • 64 “large-memory” nodes (diskless)
    • IBM System x iDataPlex dx360 M3 server (12 cores, 48 GB DDR3 RAM)
    • The CPU is the same as in the “small-memory” nodes
  • 7 ‘special’ nodes (e.g., for data pre- and post-processing, and other purposes)
    • IBM System x3550 M2 server (8 cores, 32 GB DDR3 RAM, 147 GB SAS HDD)
    • Intel Xeon Processor E5520 (“Nehalem-EP”, 4 cores, 2.26 GHz, 8 MB cache, max. memory speed 1066 MHz, QPI 5.86 GT/s, TDP 80 W)
• Management and Login Nodes
  • 2 master nodes in an active/passive high-availability (HA) configuration
    • IBM System x3550 M3 server (8 cores, 24 GB DDR3 RAM, storage: 4 SAS 300 GB HDDs, 10k RPM, 6 Gbps, configured as RAID-10)
    • Intel Xeon Processor E5520 (“Westmere-EP”, 4 cores, 2.4 GHz, 12 MB cache, max. memory speed 1066 MHz, QPI 5.86 GT/s, TDP 80 W)
    • The master nodes are shared between FLOW and its sister cluster, HERO, and run all vital cluster services (node provisioning, DHCP, DNS, LDAP, NFS, Job Management System, etc.). They also provide monitoring functions for both clusters (with automated alerting). Monitoring covers hardware components (health status of all servers, temperature, power consumption, etc.) as well as basic cluster services (with automated restart should a service fail).
  • 2 login nodes for user access to the system, software development (programming environment), and job submission and control
    • IBM System x3550 M3 server (8 cores, 24 GB DDR3 RAM, storage: 2 SAS 146 GB HDDs, 10k RPM, 6 Gbps, configured as RAID-1)
    • The CPU is the same as in the master nodes
• Internal networks
  • Node Interconnect: fully non-blocking QDR InfiniBand (Mellanox ConnectX-2 HCAs, Mellanox IS5200 216-port switch)
  • 10Gb Ethernet backbone network connecting the management and login nodes, the storage system, and the Gigabit Ethernet (see below) leaf switches
  • Gigabit Ethernet for vital cluster services (node provisioning, DHCP, DNS, LDAP, NFS, Job Management System, etc.)
  • Dedicated IPMI network for hardware monitoring and control, including access to the VGA console (KVM functionality), enabling full remote management of the cluster
• Storage System
  • Enterprise-class scalable NAS cluster (manufacturer: EMC Isilon), 180 TB raw capacity, 130 TB net capacity (for the chosen redundancy level), IOPS NFS/CIFS (SpecSFS 2008) 18075 / 32279, InfiniBand backend network, two Gigabit Ethernet and two 10Gb Ethernet front-end ports per storage node.
  • The storage system is shared between FLOW and its sister cluster, HERO. Disk space is allocated to the two clusters depending on the proportion of the storage system’s hardware that was funded from the FLOW and HERO project budgets, respectively.

System Software and Middleware

• All cluster nodes are running Scientific Linux
• Cluster management: Bright Cluster Manager
• Commercial compilers, debuggers and profilers, and performance libraries: Intel Cluster Studio, NAG, PGI
• Workload management: Sun Grid Engine (SGE)

Selected applications running on FLOW

• ANSYS CFX – a high-performance, general-purpose fluid dynamics programme
• NekTar – a Navier–Stokes solver
• OpenFOAM, a free, open-source CFD software package that incorporates a variety of methods and features to solve a wide range of problems, from chemical reactions, turbulence and heat transfer to solid dynamics and electromagnetics
• PALM – A Parallelised Large-Eddy Simulation Model for Atmospheric and Oceanic Flows
• WRF – The Weather Research & Forecasting Model

Pictures