Running NekRS as a Standalone Application

Standalone Simulations

As part of Cardinal's build process, the nekrs executable used to run standalone NekRS cases is compiled and placed in the $NEKRS_HOME/bin directory. This directory also contains other scripts used to simplify the use of the nekrs executable. To use these scripts to run standalone NekRS cases, we recommend adding this location to your path:

export PATH=$NEKRS_HOME/bin:$PATH

Then, you can run any standalone NekRS case simply by having built Cardinal - no need to separately build and compile NekRS. For instance, try running the ethier example that ships with NekRS:

cd contrib/nekRS/examples/ethier
nrsmpi ethier 4

And that's it! No need to separately compile NekRS.

Thinly-Wrapped Simulations

To access this tutorial:

cd cardinal/tutorials/standalone

To contrast with the previous example, you can achieve the same "standalone" calculations via Cardinal, which you might be interested in to leverage Cardinal's postprocessing and data I/O features. Some useful features include:

• Query the solution, evaluate heat balances and pressure drops, or evaluate solution convergence

• Providing one-way coupling to other MOOSE applications, such as for transporting scalars based on NekRS's velocity solution or for projecting NekRS turbulent viscosity closure terms onto another MOOSE application's mesh

• Project the NekRS solution onto other discretization schemes, such as a subchannel discretization, or onto other MOOSE applications, such as for providing closures

• Automatically convert nondimensional NekRS solutions into dimensional form

• Because the MOOSE framework supports many different output formats, obtain a representation of the NekRS solution in Exodus, VTK, CSV, and other formats.

Instead of running a NekRS input with the nekrs executable, you can instead create a "thin" wrapper input file that runs NekRS as a MOOSE application (but allowing usage of the postprocessing and data I/O features of Cardinal). For wrapped applications, NekRS will continue to write its own field file output during the simulation as specified by settings in the .par file.

To run NekRS via MOOSE, without any physics coupling, Cardinal simply replaces calls to MOOSE solve methods with NekRS solve methods available through an API. There are no data transfers to/from NekRS. A thinly-wrapped simulation uses:

1. NekRSMesh: create a "mirror" of the NekRS mesh, which can optionally be used to interpolate a high-order NekRS solution into a lower-order mesh (in any MOOSE-supported format).

2. NekRSProblem: allow MOOSE to run NekRS

For this tutorial, we will use the turbPipe example that ships with the NekRS repository as an example case. This case models turbulent flow in a cylindrical pipe (cast in non-dimensional form). The domain consists of a pipe of diameter 1 and length 20 with flow in the direction. The NekRS mesh is shown in Figure 1. Note that the visualization of NekRS's mesh in Paraview draws lines connecting the GLL points, and not the actual element edges.

Figure 1: NekRS mesh, with lines connecting GLL points and sideset IDs.

The NekRS input files are exactly the same that would be used to run this model as a standalone case. These input files include:

• turbPipe.re2: NekRS mesh

• turbPipe.par: High-level settings for the solver, boundary condition mappings to sidesets, and the equations to solve

• turbPipe.udf: User-defined C++ functions for on-line postprocessing and model setup

• turbPipe.oudf: User-defined OCCA kernels for boundary conditions and source terms

This particular input also uses an optional turbPipe.usr file for setting up other parts of the model using the Nek5000 backend. We refer a discussion of these files to the NekRS documentation website.

Instead of running this input directly with the NekRS scripts like we did in Standalone Simulations , we instead wrap the NekRS simulation as a MOOSE application. The Cardinal input file is shown below; this is not the simplest file that we need to run NekRS, but we add extra features to be described shortly.

[Mesh<<<{"href": "../syntax/Mesh/index.html"}>>>]
  type = NekRSMesh
  volume = true
  order = SECOND
[]

[Problem<<<{"href": "../syntax/Problem/index.html"}>>>]
  type = NekRSProblem
  casename<<<{"description": "Case name for the NekRS input files; this is <case> in <case>.par, <case>.udf, <case>.oudf, and <case>.re2."}>>> = 'turbPipe'

  [FieldTransfers<<<{"href": "../syntax/Problem/FieldTransfers/index.html"}>>>]
    [pressure]
      type = NekFieldVariable<<<{"description": "Reads/writes volumetric field data between NekRS and MOOSE."}>>>
      direction<<<{"description": "Direction in which to send data"}>>> = from_nek
    []
    [velocity_x]
      type = NekFieldVariable<<<{"description": "Reads/writes volumetric field data between NekRS and MOOSE."}>>>
      direction<<<{"description": "Direction in which to send data"}>>> = from_nek
    []
    [velocity_y]
      type = NekFieldVariable<<<{"description": "Reads/writes volumetric field data between NekRS and MOOSE."}>>>
      direction<<<{"description": "Direction in which to send data"}>>> = from_nek
    []
    [velocity_z]
      type = NekFieldVariable<<<{"description": "Reads/writes volumetric field data between NekRS and MOOSE."}>>>
      direction<<<{"description": "Direction in which to send data"}>>> = from_nek
    []
  []
[]

[Postprocessors<<<{"href": "../syntax/Postprocessors/index.html"}>>>]
  [outlet_p]
    type = NekSideAverage<<<{"description": "Average of a field over a boundary of the NekRS mesh", "href": "../source/postprocessors/NekSideAverage.html"}>>>
    boundary<<<{"description": "Boundary ID(s) for which to compute the postprocessor"}>>> = '2'
    field<<<{"description": "Field to apply this object to"}>>> = pressure
  []
  [inlet_p]
    type = NekSideAverage<<<{"description": "Average of a field over a boundary of the NekRS mesh", "href": "../source/postprocessors/NekSideAverage.html"}>>>
    boundary<<<{"description": "Boundary ID(s) for which to compute the postprocessor"}>>> = '1'
    field<<<{"description": "Field to apply this object to"}>>> = pressure
  []
  [mdot]
    type = NekMassFluxWeightedSideIntegral<<<{"description": "Mass flux weighted integral of a field over a boundary of the NekRS mesh", "href": "../source/postprocessors/NekMassFluxWeightedSideIntegral.html"}>>>
    boundary<<<{"description": "Boundary ID(s) for which to compute the postprocessor"}>>> = '1'
    field<<<{"description": "Field to apply this object to"}>>> = unity
  []

  # subtracts the two pressure postprocessors
  [dP]
    type = DifferencePostprocessor<<<{"description": "Computes the difference between two postprocessors", "href": "../source/postprocessors/DifferencePostprocessor.html"}>>>
    value1<<<{"description": "First value"}>>> = outlet_p
    value2<<<{"description": "Second value"}>>> = inlet_p
  []
[]

[UserObjects<<<{"href": "../syntax/UserObjects/index.html"}>>>]
  [axial_bins]
    type = LayeredBin<<<{"description": "Creates a unique spatial bin for layers in a specified direction", "href": "../source/userobjects/LayeredBin.html"}>>>
    direction<<<{"description": "The direction of the layers (x, y, or z)"}>>> = z
    num_layers<<<{"description": "The number of layers between the bounding box of the domain"}>>> = 20
  []
  [radial_bins]
    type = RadialBin<<<{"description": "Creates spatial bins for layers in the radial direction", "href": "../source/userobjects/RadialBin.html"}>>>
    vertical_axis<<<{"description": "The vertical axis about which to compute the radial coordinate (x, y, or z)"}>>> = z
    rmax<<<{"description": "Outer radius"}>>> = 0.5
    nr<<<{"description": "The number of layers in the radial direction"}>>> = 12
    growth_r<<<{"description": "The ratio of radial sizes of successive rings of elements"}>>> = 0.9
  []
  [volume_averages]
    type = NekBinnedVolumeAverage<<<{"description": "Compute the spatially-binned volume average of a field over the NekRS mesh", "href": "../source/userobjects/NekBinnedVolumeAverage.html"}>>>
    bins<<<{"description": "Userobjects providing a spatial bin given a point"}>>> = 'radial_bins axial_bins'
    field<<<{"description": "Field to apply this object to"}>>> = velocity_z
    map_space_by_qp<<<{"description": "Whether to map the NekRS spatial domain to a bin according to the element centroids (true) or quadrature point locations (false)."}>>> = true
  []
[]

[AuxVariables<<<{"href": "../syntax/AuxVariables/index.html"}>>>]
  [volume_averages]
    family<<<{"description": "Specifies the family of FE shape functions to use for this variable"}>>> = MONOMIAL
    order<<<{"description": "Specifies the order of the FE shape function to use for this variable (additional orders not listed are allowed)"}>>> = CONSTANT
  []
[]

[AuxKernels<<<{"href": "../syntax/AuxKernels/index.html"}>>>]
  [volume_averages]
    type = SpatialUserObjectAux<<<{"description": "Populates an auxiliary variable with a spatial value returned from a UserObject spatialValue method.", "href": "../source/auxkernels/SpatialUserObjectAux.html"}>>>
    variable<<<{"description": "The name of the variable that this object applies to"}>>> = volume_averages
    user_object<<<{"description": "The UserObject UserObject to get values from.  Note that the UserObject _must_ implement the spatialValue() virtual function!"}>>> = volume_averages
  []
[]

[MultiApps<<<{"href": "../syntax/MultiApps/index.html"}>>>]
  [sub]
    type = TransientMultiApp<<<{"description": "MultiApp for performing coupled simulations with the parent and sub-application both progressing in time.", "href": "../source/multiapps/TransientMultiApp.html"}>>>
    input_files<<<{"description": "The input file for each App.  If this parameter only contains one input file it will be used for all of the Apps.  When using 'positions_from_file' it is also admissable to provide one input_file per file."}>>> = sub.i
    execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = timestep_end
  []
[]

[Transfers<<<{"href": "../syntax/Transfers/index.html"}>>>]
  [uo_to_sub]
    type = MultiAppGeneralFieldUserObjectTransfer<<<{"description": "Transfers user object spatial evaluations from an origin app onto a variable in the target application.", "href": "../source/transfers/MultiAppGeneralFieldUserObjectTransfer.html"}>>>
    to_multi_app<<<{"description": "The name of the MultiApp to transfer the data to"}>>> = sub
    source_user_object<<<{"description": "The UserObject you want to transfer values from. It must implement the SpatialValue() class routine"}>>> = volume_averages
    variable<<<{"description": "The auxiliary variable to store the transferred values in."}>>> = avg_velocity
  []
[]

[Executioner<<<{"href": "../syntax/Executioner/index.html"}>>>]
  type = Transient

  [TimeStepper<<<{"href": "../syntax/Executioner/TimeStepper/index.html"}>>>]
    type = NekTimeStepper
  []
[]

[Outputs<<<{"href": "../syntax/Outputs/index.html"}>>>]
  exodus<<<{"description": "Output the results using the default settings for Exodus output."}>>> = true
  csv<<<{"description": "Output the scalar variable and postprocessors to a *.csv file using the default CSV output."}>>> = true

  # this hides these values from the screen for neater output
  hide<<<{"description": "A list of the variables and postprocessors that should NOT be output to the Exodus file (may include Variables, ScalarVariables, and Postprocessor names)."}>>> = 'outlet_p inlet_p'
[]

The essential blocks in the input file are:

• Mesh: creates a lower-order mirror of the NekRS mesh

• Problem: replaces MOOSE finite element solves with NekRS solves. NekFieldVariable objects are added in order to read from the NekRS internal solution fields and write onto the NekRSMesh for viewing.

• Executioner: controls the time stepping according to the settings in the NekRS input files

• Outputs: outputs any results that have been projected onto the NekRSMesh to the specified format.

This input file is run with:

mpiexec -np 4 cardinal-opt -i nek.i

which will run with 4 MPI ranks. This will create a number of output files:

• nek_out.e shows the NekRS solution mapped to a MOOSE mesh

• nek_out_sub0. shows the result of a postprocessing operation, mapped to a different MOOSE mesh

• nek_out.csv shows the CSV postprocessor values

• turbPipe0.f<n> are the NekRS output files, where <n> is an integer representing output step index in NekRS

When running this tutorial, the NekRS output file is the nek_out.e file, while the output of the sub-application is the nek_out_sub0.e file.

Now that you know how to run, let's describe the rest of the contents in the nek.i input file. This file adds a few additional postprocessing operations to compute:

• pressure drop, computed by subtracting the inlet average pressure from the outlet average pressure with two NekSideAverage postprocessors and the DifferencePostprocessor

• mass flowrate, computed with a NekMassFluxWeightedSideIntegral postprocessor

[Postprocessors<<<{"href": "../syntax/Postprocessors/index.html"}>>>]
  [outlet_p]
    type = NekSideAverage<<<{"description": "Average of a field over a boundary of the NekRS mesh", "href": "../source/postprocessors/NekSideAverage.html"}>>>
    boundary<<<{"description": "Boundary ID(s) for which to compute the postprocessor"}>>> = '2'
    field<<<{"description": "Field to apply this object to"}>>> = pressure
  []
  [inlet_p]
    type = NekSideAverage<<<{"description": "Average of a field over a boundary of the NekRS mesh", "href": "../source/postprocessors/NekSideAverage.html"}>>>
    boundary<<<{"description": "Boundary ID(s) for which to compute the postprocessor"}>>> = '1'
    field<<<{"description": "Field to apply this object to"}>>> = pressure
  []
  [mdot]
    type = NekMassFluxWeightedSideIntegral<<<{"description": "Mass flux weighted integral of a field over a boundary of the NekRS mesh", "href": "../source/postprocessors/NekMassFluxWeightedSideIntegral.html"}>>>
    boundary<<<{"description": "Boundary ID(s) for which to compute the postprocessor"}>>> = '1'
    field<<<{"description": "Field to apply this object to"}>>> = unity
  []

  # subtracts the two pressure postprocessors
  [dP]
    type = DifferencePostprocessor<<<{"description": "Computes the difference between two postprocessors", "href": "../source/postprocessors/DifferencePostprocessor.html"}>>>
    value1<<<{"description": "First value"}>>> = outlet_p
    value2<<<{"description": "Second value"}>>> = inlet_p
  []
[]

This will print to the screen for each time step the values of these postprocessors, which can be useful for evaluating solution progression:

+----------------+----------------+----------------+
| time           | dP             | mdot           |
+----------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
|   6.000000e-03 |  -3.334281e+02 |  -7.854005e-01 |
|   1.200000e-02 |   1.637766e+02 |  -7.853985e-01 |
|   1.800000e-02 |   1.540674e+00 |  -7.853985e-01 |
+----------------+----------------+----------------+

By setting csv = true in the output block, we will also write these postprocessors into a CSV format, which is convenient for script-based postprocessing operations:

time,dP,mdot
0,0,0
0.006,-333.42814336273,-0.78540045786239
0.012,163.77659967808,-0.78539846245391
0.018,1.540674311383,-0.78539846245391

By using NekFieldVariable objects in the [FieldTransfers] block, we write the NekRS solution for pressure and velocity (which for this example has degrees of freedom per element, since polynomialOrder = 7 in turbPipe.par) onto second-order Lagrange auxiliary variables named P and velocity_x, velocity_y, and velocity_z. You can then apply any MOOSE object to those variables, such as postprocessors, userobjects, auxiliary kernels, and so on. You can also transfer these variables to another MOOSE application if you want to couple NekRS to MOOSE without feedback - such as for using Nek's velocity to transport a passive scalar in another MOOSE application.

The axial velocity computed by NekRS, as well as the velocity interpolated onto the mesh mirror, are shown in Figure 2.

Figure 2: NekRS computed axial velocity (top) and the velocity interpolated onto the NekRSMesh mirror (bottom)

We can also apply several userobjects directly to the NekRS solution for a number of postprocessing operations. Below, we perform a volume average of in 12 radial bins discretized into 20 axial layers.

[UserObjects<<<{"href": "../syntax/UserObjects/index.html"}>>>]
  [axial_bins]
    type = LayeredBin<<<{"description": "Creates a unique spatial bin for layers in a specified direction", "href": "../source/userobjects/LayeredBin.html"}>>>
    direction<<<{"description": "The direction of the layers (x, y, or z)"}>>> = z
    num_layers<<<{"description": "The number of layers between the bounding box of the domain"}>>> = 20
  []
  [radial_bins]
    type = RadialBin<<<{"description": "Creates spatial bins for layers in the radial direction", "href": "../source/userobjects/RadialBin.html"}>>>
    vertical_axis<<<{"description": "The vertical axis about which to compute the radial coordinate (x, y, or z)"}>>> = z
    rmax<<<{"description": "Outer radius"}>>> = 0.5
    nr<<<{"description": "The number of layers in the radial direction"}>>> = 12
    growth_r<<<{"description": "The ratio of radial sizes of successive rings of elements"}>>> = 0.9
  []
  [volume_averages]
    type = NekBinnedVolumeAverage<<<{"description": "Compute the spatially-binned volume average of a field over the NekRS mesh", "href": "../source/userobjects/NekBinnedVolumeAverage.html"}>>>
    bins<<<{"description": "Userobjects providing a spatial bin given a point"}>>> = 'radial_bins axial_bins'
    field<<<{"description": "Field to apply this object to"}>>> = velocity_z
    map_space_by_qp<<<{"description": "Whether to map the NekRS spatial domain to a bin according to the element centroids (true) or quadrature point locations (false)."}>>> = true
  []
[]

If we want to view the output of this averaging on the NekRSMesh, we could visualize it with a SpatialUserObjectAux.

[AuxVariables<<<{"href": "../syntax/AuxVariables/index.html"}>>>]
  [volume_averages]
    family<<<{"description": "Specifies the family of FE shape functions to use for this variable"}>>> = MONOMIAL
    order<<<{"description": "Specifies the order of the FE shape function to use for this variable (additional orders not listed are allowed)"}>>> = CONSTANT
  []
[]

[AuxKernels<<<{"href": "../syntax/AuxKernels/index.html"}>>>]
  [volume_averages]
    type = SpatialUserObjectAux<<<{"description": "Populates an auxiliary variable with a spatial value returned from a UserObject spatialValue method.", "href": "../source/auxkernels/SpatialUserObjectAux.html"}>>>
    variable<<<{"description": "The name of the variable that this object applies to"}>>> = volume_averages
    user_object<<<{"description": "The UserObject UserObject to get values from.  Note that the UserObject _must_ implement the spatialValue() virtual function!"}>>> = volume_averages
  []
[]

The result of the volume averaging operation is shown in Figure 3. Because the NekRS mesh elements don't fall nicely into the specified bins, we actually can only see the bin averages that the mesh mirror elements "hit" (according to their centroid). This is obviously non-ideal because the underlying form of the NekRS mesh is distorting the visualization of the volume average (even though the NekRS mesh element layout doesn't affect the actual averaging and the userobject stores the values of all 12 radial bins, even if they can't be seen).

Figure 3: Representation of the volume_averages binned averaging on the NekRS mesh mirror

Instead, we can leverage MOOSE's MultiApp system to transfer the user object to a sub-application with a different mesh than what is used in NekRS. Then we can visualize the averaging operation perfectly without concern for the fact that the NekRS mesh doesn't have elements that fall nicely into the 12 radial bins. To do this, we create a sub-application with mesh elements exactly matching the user object binning and turn the solve off by setting solve = false, so that this input file only serves to receive data onto a different mesh.

[Mesh<<<{"href": "../syntax/Mesh/index.html"}>>>]
  [disc]
    type = AnnularMeshGenerator<<<{"description": "For rmin>0: creates an annular mesh of QUAD4 elements. For rmin=0: creates a disc mesh of QUAD4 and TRI3 elements. Boundary sidesets are created at rmax and rmin, and given these names. If dmin!=0 and dmax!=360, a sector of an annulus or disc is created. In this case boundary sidesets are also created at dmin and dmax, and given these names", "href": "../source/meshgenerators/AnnularMeshGenerator.html"}>>>
    rmax<<<{"description": "Outer radius"}>>> = 0.5
    rmin<<<{"description": "Inner radius.  If rmin=0 then a disc mesh (with no central hole) will be created."}>>> = 0.0
    nr<<<{"description": "Number of elements in the radial direction"}>>> = 12
    nt<<<{"description": "Number of elements in the angular direction"}>>> = 24
    growth_r<<<{"description": "The ratio of radial sizes of successive rings of elements"}>>> = 0.9
  []
  [extrude]
    type = AdvancedExtruderGenerator<<<{"description": "Extrudes a 1D mesh into 2D, or a 2D mesh into 3D, can have a variable height for each elevation, variable number of layers within each elevation, variable growth factors of axial element sizes within each elevation and remap subdomain_ids, boundary_ids and element extra integers within each elevation as well as interface boundaries between neighboring elevation layers.", "href": "../source/meshgenerators/AdvancedExtruderGenerator.html"}>>>
    input<<<{"description": "The mesh to extrude"}>>> = disc
    heights<<<{"description": "The height of each elevation"}>>> = '20.0'
    num_layers<<<{"description": "The number of layers for each elevation - must be num_elevations in length!"}>>> = '20'
    direction<<<{"description": "A vector that points in the direction to extrude (note, this will be normalized internally - so don't worry about it here)"}>>> = '0 0 1'
  []
[]

[Problem<<<{"href": "../syntax/Problem/index.html"}>>>]
  solve = false
  type = FEProblem
[]

[AuxVariables<<<{"href": "../syntax/AuxVariables/index.html"}>>>]
  [avg_velocity]
    family<<<{"description": "Specifies the family of FE shape functions to use for this variable"}>>> = MONOMIAL
    order<<<{"description": "Specifies the order of the FE shape function to use for this variable (additional orders not listed are allowed)"}>>> = CONSTANT
  []
[]

[Executioner<<<{"href": "../syntax/Executioner/index.html"}>>>]
  type = Transient
[]

[Outputs<<<{"href": "../syntax/Outputs/index.html"}>>>]
  exodus<<<{"description": "Output the results using the default settings for Exodus output."}>>> = true
[]

Then we transfer the volume_averages user object to the sub-application.

[MultiApps<<<{"href": "../syntax/MultiApps/index.html"}>>>]
  [sub]
    type = TransientMultiApp<<<{"description": "MultiApp for performing coupled simulations with the parent and sub-application both progressing in time.", "href": "../source/multiapps/TransientMultiApp.html"}>>>
    input_files<<<{"description": "The input file for each App.  If this parameter only contains one input file it will be used for all of the Apps.  When using 'positions_from_file' it is also admissable to provide one input_file per file."}>>> = sub.i
    execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = timestep_end
  []
[]

[Transfers<<<{"href": "../syntax/Transfers/index.html"}>>>]
  [uo_to_sub]
    type = MultiAppGeneralFieldUserObjectTransfer<<<{"description": "Transfers user object spatial evaluations from an origin app onto a variable in the target application.", "href": "../source/transfers/MultiAppGeneralFieldUserObjectTransfer.html"}>>>
    to_multi_app<<<{"description": "The name of the MultiApp to transfer the data to"}>>> = sub
    source_user_object<<<{"description": "The UserObject you want to transfer values from. It must implement the SpatialValue() class routine"}>>> = volume_averages
    variable<<<{"description": "The auxiliary variable to store the transferred values in."}>>> = avg_velocity
  []
[]

The user object received on the sub-application is shown in Figure 4, which exactly represents the 12 radial averaging bins.

Figure 4: Representation of the volume_averages binned exactly as computed by user object

A few examples of other postprocessors that may be of use to NekRS simulations include:

• ElementL2Error, which computes the L norm of a variable relative to a provided function, useful for solution verification

• FindValueOnLine, which finds the point at which a specified value of a variable occurs, which might be used for evaluating a boundary layer thickness

• LinearCombinationPostprocessor, which can be used to combine postprocessors together in a general expression , where are coefficients, are postprocessors, and is a constant additive factor. This can be used to compute the temperature rise in a domain by subtracting a postprocessor that computes the inlet temperature from a postprocessor that computes the outlet temperature.

• PercentChangePostprocessor which computes the percent change between two successive time steps for assessing convergence.

• TimeExtremeValue, which provides the maximum/minimum value of a variable over the course of an entire simulation, such as for evaluating peak stress in an oscillating system

Please consult the MOOSE documentation for a full list of available postprocessors.