Keynote Talk: Massively Parallel Numerical Simulation of Hydrodynamics and Transfers in a Polydispersed Reactive Gas-Particle Fluidized Bed at Industrial Scale with a Very Fine Mesh, over One Billion of Cells

Dense particle-laden reactive flows are encountered in a wide range of industrial applications such as coal/biomass combustion (CO₂ capture / chemical looping), catalytic polymerization, uranium fluorination or medicine drugs production. The development of their numerical modeling with particular focus on the hydrodynamics of fluidized beds has been a very active topic of research over the last three decades. From a physical point of view, the key challenges are about accounting for complex phenomenon occurring in reactive multi-scale flows: turbulence-particle fluctuating kinetic energy, fluid-particles interactions, interfacial transfers of mass, momentum and energy as well as chemical reactions… On top of these modelling aspects, this work primarily focuses on numerical challenges such as numerical schemes, mesh refinement, partitioning, highly parallel computing (HPC), data and post-processing.

Igci et al. [1] pointed out that the numerical simulation of fluidized beds is very expensive (CPU time), both at industrial and academic scales, in particular due to the 3D unsteady solid structures of small size (clusters). Predicting accurately the macroscopic flow properties such as bed expansion or solid flow rate in these systems requires a detailed description of these small-scale structures. To accurately describe and predict the formation of these small clusters, a very fine mesh having million/billion of cells is required. Therefore, massively parallel computations are required and are accessible thanks to recent progresses in supercomputer capabilities. However, this induces complex issues in terms of geometry and mesh generation, data management (I/O and storage), computation time and HPC.

The purpose of the work is the numerical simulation of an industrial fluidized bed whose dimensions are 30 m height and 5 m in diameter. The reactor contains 100 tonnes of large particles (1600 µm) and 4 injections (located at 0.6 m height) of very fine particles (80 µm) which are subjected to an exothermic reaction. Therefore, separate enthalpy transport equations are computed for gas and both particle species with local reactive source terms, heat transfers between gas and particles and dispersion by fluctuant velocity. To sum up the overall modeling, simulations are including particle drag, turbulent and kinetic model with four-way coupling, for gas and particles, respectively. Finally, a polydispersed collision model describes large and fine particle-particle interactions (Batrak et al. [2]).

Nowadays it is possible to perform realistic 3D simulations of industrial geometries using an unsteady Eulerian multi-fluid approach for turbulent polydisperse reactive particle mixtures (Hamidouche [3], Fede et al. [4]). This approach is implemented in the massively parallelized code NEPTUNE_CFD. This software is a multiphase flow solver developed in the framework of the NEPTUNE project, financially supported by CEA, EDF, IRSN and Framatome. The main numerical characteristics of this software are (i) unstructured meshes with all types of cell, (ii) non-matching meshes and/or rotating meshes, (iii) “cell-center” type finite-volume method, (iv) calculation of co-localized gradients with reconstruction methods and (v) distributed-memory parallelism by domain splitting (mesh partitioning and MPI parallelization). The algorithm core is based on an elliptic fractional step method using iterative linear solvers or direct matrix inversion (Méchitoua et al. [5]). The algorithm accounts for density variation as a function of pressure and enthalpy during each time step. The pressure is resolved by a parallel multigrid solver. NEPTUNE_CFD scalability has already been proven on this kind of configuration up to 2,560 cores (Hamidouche et al. [3]). Previous runs have been performed using up to 4,000 cores with linear speed-up and perfect efficiency as long as each core handles at least 25,000 cells.

The key novelties of this work are a highly refined meshing and the use of the whole new CALMIP supercomputer, Olympe, to perform a reactive polydispersed simulation. The spatial discretization consisted in an unstructured mesh of 1,002,355,456 hexahedron cells. The characteristic volume of these cells is about 1.25*10^-7 m³ that must be compared to a total reactor volume of 600 m³.

This massively parallel unsteady simulation of a 3D industrial reactive polydispersed fluidized bed with such a refined mesh and using more than 13,000 cores is a worldwide premiere. NEPTUNE_CFD ran on the 362 nodes of Olympe, i.e. 13,032 cores (SkyLake 6140) and relied on Intel compiler and IntelMPI library. The numerical simulation used 5 millions of CPU hours over 15 days (elapsed time) and 120 TB of generated raw data. The physical time simulated is about 15.5s. A special attention was given on massively parallel computation performances of NEPTUNE_CFD in terms of profiling and code efficiency, obtained speed-up, MPI communications and mesh partitioning.

The use of the latest generation of supercomputers coupled with the massively parallel CFD software NEPTUNE_CFD allows performing computations that could not be considered hitherto. Obtained results have a yet unmatched accuracy which will allow for fine understanding of complex flows encountered at industrial scale. These results can be considered as reference results, converged and mesh independent for an industrial geometry. They will be used to develop and evaluate sub-grid modeling approaches to enable low cost simulations while accounting for small scale phenomenon. This is the first time that highly resolved simulations can be used to develop such models at an industrial scale.

REFERENCES

[1] Igci, Y. & Sundaresan, S., Verification of filtered two-fluid models for gas-particle flows in risers
AIChE Journal, Wiley Subscription Services, Inc., A Wiley Company, 2011 , 57 , 2691-2707.

[2] Batrak O., Patino G., Simonin O., Flour I., Le Guevel T. & Perez E., Unlike particles size collision model in 3D unsteady polydispersed simulation of circulating fluidised bed, Circulating Fluidized Bed Technology VIII, Kefa Cen Ed., International Academic Publishers / Beijing Word Publishing Corporation, pp.370–378, 2005.

[3] Hamidouche Z., Masi E., Fede P., Ansart R., Neau H., Hemati M. & Simonin O., Numerical Simulation of Multiphase Reactive Flows, in: Advances in Chemical Engineering, volume 52, 2018, pp. 51–124. doi:10.1016/bs. ache.2018.01.003.

[4] Fede P., Simonin O. & Ingram, A., 3D numerical simulation of a lab-scale pressurized dense fluidized bed focussing on the effect of the particle-particle restitution coefficient and particle--wall boundary conditions, Chemical Engineering Science, 2016, 142, 215-235.

[5] Méchitoua N., Boucker M., Laviéville J., Hérard J.M., Pigny S., & Serre G., An unstructured finite volume solver for two-phase water/vapour flows modeling based on elliptic oriented fractional step method. NURETH 10, Séoul, Corée du Sud, 2003.