AFiD: an open-source code for DNS of turbulent flows

Most of the simulations conducted in my research group are performed using a second order finite difference code with fractional time integration. Other add-ons to the basic simulations are the addition of a Lagrangian second-phase in the flow, and the use of a multiple resolution scheme to decouple the grids where scalars and momentum are discretized.

The numerical code used is written in Fortran 90, and has a MPI parallelization. The MPI domain decomposition is a 2D decomposition, i.e. in pencils. It uses parallel HDF5 for output, and FFTW/Lapack libraries for some mathematical operations. Simulations ran using this code won in 2012 the Wim van Niewpoort award for most efficient use of HPC computing time in the Netherlands. It has run on up to 10k cores in IBM (Fermi@Cineca), Cray (Hermit@HLRS) and Bull (Curie@TGCC) systems, and was shown to scale up to 64k cores on Curie Thin Nodes. The Rayleigh-Bénard version of the code, based on the library 2DECOMP was open sourced under the name of AFiD, and is available for download. Details of the numerical schemes of the code and the computational algorithms are detailed in this reference. Thanks to help from people at nVidia, a GPU (CUDA) version of the semi-implicit code is also available.

Due to the flexibility of second-order finite differences, it is easy to extend the code to include objects in the flow through the immersed boundary method. Currently, it is possible to not only include fixed objects such as rough boundaries, but also full fluid-structure interaction, including both weak and strong coupled fluid-structure interaction, and an immersed boundary formulation based on moving least squares, capable of simulating objects which deform and move due to fluid forces instead of due to a prescribed movement.

A recent add-on was the simulation of electro-kinetic flows. However, the near-wall region of these flows require substantial resolution, and we are working on implementing wall functions to allow us to extend the parameter space which is currently possible to simulate.

Relevant articles

  • A multiple-resolution strategy for Direct Numerical Simulation of scalar turbulence. R. Ostilla Mónico, Y. T. Yang, E. P. van der Poel, D. Lohse, R. Verzicco Journal of Computational Physics, 301, 308-321 (2015).. [arxiv] [JCP]
  • A parallel interaction potential approach coupled with the immersed boundary method for fully resolved simulations of deformable interfaces and membranes. V. Spandan, V. Meschini, R. Ostilla-Mónico, D. Lohse, G. Querzoli, M. D. de Tullio, R. Verzicco J. Comp. Phys, 348, 567-590 (2017). [JCP] [arxiv]
  • Controlling turbulent drag across electrolytes using electric fields. R. Ostilla-Mónico, A. Lee Faraday Discussions, 199, 159-173 (2017). [RSC] [arxiv]
  • AFiD-GPU: a versatile Navier-Stokes Solver for Wall-Bounded Turbulent Flows on GPU Clusters. X. Zhu, E. Phillips, V. Spandan, J. Donners, G. Ruetsch, J. Romero, R. Ostilla-Mónico, Y. Yang, D. Lohse, R. Verzicco, M. Fatica, R.J.A.M. Stevens Comp. Phys. Comm., 229, 199-210 (2018). [arXiv] [CPC]