NekRSProblem

Nondimensional Solution

NekRS is most often solved in nondimensional form, such that all solution variables are of order unity by normalizing by problem-specific characteristic scales. However, most other MOOSE applications use dimensional units. When transferring field data to/from NekRS or when postprocessing the NekRS solution, it is important for the NekRS solution to match the dimensional solution of the coupled MOOSE application. For physical intuition, it is also often helpful in many cases to visualize and interpret a NekRS solution in dimensional form. This class automatically performs these conversions between dimensional and non-dimensional form for you, as well as dimensionalizes the various Nek postpocessors in Cardinal.

If your NekRS input files are in nondimensional form, you must set nondimensional = true and provide the various characteristic scales that were used to set up the NekRS inputs. Cardinal assumes that the NekRS inputs were nondimensionalized with the following:

(1)

(2)

(3)

(4)

(5)

where superscripts indicate nondimensional quantities. U_ref is used to specify , T_ref is used to specify , dT_ref is used to specify , L_ref is used to specify , rho_0 is used to specify , and Cp_0 is used to specify (which does not appear above, but is necessary for scaling a volumetric heat source). Finally, the mesh mirror must be in the same units as used in the coupled MOOSE application, so the scaling parameter on NekRSMesh must be set to dimensionalize the nondimensional .re2 mesh. In other words, scaling must be set to .

For example, suppose your NekRS input is in non-dimensional form. Applying a NekVolumeAverage postprocessor to temperature would be used to evaluate a volume average of temperature.

[Postprocessors]
  [avg_T]
    type = NekVolumeAverage
    field = temperature
  []
[]

If the NekRS inputs are properly non-dimensionalized and the correct scales are provided to this class, then temperature is non-dimensionalized according to Eq. (3) and volume is non-dimensionalized according to Eq. (4), or

(6)

The NekVolumeAverage postprocessor is then computed directly on the NekRS solution (in non-dimensional form) to give

(7)

where is the value of the postprocessor in non-dimensional form. Before returning the value of the postprocessor, Eq. (7) is dimensionalized by applying the scales in Eq. (3) and Eq. (6) to give the

(8)

where is the value of the postprocessor in dimensional form (which is what is actually returned by the postprocessor). So when using the nondimensional = true feature, all postprocessors and solution fields extracted from NekRS are represented in dimensional form in Cardinal.

Outputting the NekRS Solution

This class enables extracting a number of fields from the NekRS solution onto the NekRSMesh mesh mirror in addition to any fields that are already interpolated onto the mesh mirror for purposes of multiphysics coupling to another MOOSE application. This feature can be used for:

• Quick visualization of the NekRS solution without needing to rely on NekRS's custom output format

• Postprocessing the NekRS solution using MOOSE's systems

• Providing one-way coupling to other MOOSE applications, such as for transporting scalars based on NekRS's velocity solution or for projecting NekRS turbulent viscosity closure terms onto another MOOSE application's mesh

This output feature is used by specifying the fields to be output with the output parameter. Available output fields include:

• pressure (which creates a MOOSE variable named P)

• velocity (which creates MOOSE variables named vel_x, vel_y, and vel_z)

• temperature (which creates a MOOSE variable named temp)

• scalar01 (which creates a MOOSE variable named scalar01)

• scalar02 (which creates a MOOSE variable named scalar02)

• scalar03 (which creates a MOOSE variable named scalar03)

For NekRS simulations that are coupled to MOOSE, the temperature will already be output because it is used as part of the physics transfers.

When outputting data, the NekRS solution fields are interpolated from the GLL points onto the NekRSMesh, which will be either a first or second order representation of the NekRS mesh. This interpolation is performed using Vandermonde matrices. Therefore, the output solution is generally not an exact representation of NekRS's solution, for which the NekRS field files are still required to visualize fully. Because the MOOSE framework supports many different output formats, this is a convenient manner by which to obtain a representation of the NekRS solution in Exodus, VTK, CSV, and other formats.

If volume = true is specified on the NekRSMesh, the output solutions are represented over a volume mesh mirror. Otherwise, if volume = false, the solution is shown only on the boundaries specified with the boundary parameter.

For example, consider a CHT simulation where NekRS is coupled to MOOSE through a boundary. Normally, the NekRSMesh contains only the data that is used to couple NekRS to MOOSE - the NekRS wall temperature and the MOOSE heat flux. The input file below will interpolate the NekRS pressure and velocity solutions onto the mesh mirror. We set volume = true for the mesh mirror so that the pressure and velocity are represented over the volume for visualization. Because setting volume = true normally indicates that you want to couple NekRS to MOOSE by a volumetric heat source (which we don't just for visualization purposes), we set has_heat_source = false so that various error checks related to volume-based coupling to MOOSE are skipped.

[Problem]
  type = NekRSProblem
  casename = 'sfr_pin'
  output = 'pressure velocity'
  has_heat_source = false
[]

[Mesh]
  type = NekRSMesh
  order = SECOND
  boundary = '1'

  # we dont have any volume-based coupling with NekRS, but by specifying
  # volume here, we will extract pressure and velocity on
  # a volume mesh mirror
  volume = true
[]

[Executioner]
  type = Transient

  [TimeStepper]
    type = NekTimeStepper
  []
[]

[Outputs]
  exodus = true
  execute_on = 'final'
[]

For instance, Figure 1 shows the velocity from the NekRS field files (left) and interpolated onto a second-order mesh mirror (right). Because this particular example runs NekRS in a higher order than can be represented on a second-order mesh mirror, the interpolated velocity is clearly not an exact representation of the NekRS solution - only an interpolated version of the NekRS solution.

Figure 1: Velocity from the NekRS field files (left) and after interpolation onto a second order mesh mirror (right).

Outputting the Scratch Array

This class (and all other NekRS wrappings in Cardinal) allows you to write slots in the nrs->usrwrk scratch space array to NekRS field files. This can be useful for viewing the data sent from MOOSE to NekRS (for problem classes that involve multiphysics), as well as to visualize custom user usage of nrs->usrwrk, such as for fetching a wall distance computation from the Nek5000 backend. To write the scratch space to a field file, set usrwrk_output to an array with each "slot" in the nrs->usrwrk array that you want to write. Then, specify a filename prefix to use to name each field file.

In the example below, the first two "slots" in the nrs->usrwrk array will be written to field files on the same interval that NekRS writes its usual field files. These files will be named aaabrick0.f00001, etc. and cccbrick0.f00001, etc. Based on limitations in how NekRS writes its files, the fields written to these files will all be named temperature when visualized.

[Problem]
  type = NekRSStandaloneProblem
  casename = 'brick'
  usrwrk_output = '0 1'
  usrwrk_output_prefix = 'aaa ccc'
[]

Reducing CPU/GPU Data Transfers

As shown in External Solve , for every NekRS time step, data is passed into NekRS (which involves interpolating from a MOOSE mesh to a (usually) higher-order NekRS mesh and then copying this field from host to device) and data is passed out from NekRS (which involves copying this field from device to host and then interpolating from a (usually) higher-order NekRS mesh to a MOOSE mesh). When NekRS is run in standalone mode, data is only copied between the device and host when an output file is written.

If NekRS is run as a sub-application to a master application, and sub-cycling is used, a lot of these interpolations and Central Processing Unit (CPU)/GPU data transfers can be omitted. When you run a Nek-wrapped input file, timing estimates for these data transfers are printed to the screen. Even for NekRS meshes with a few million elements, these interpolations to/from NekRS typically require only fractions of a second. But for some use cases, you may want to reduce as many of these communciations as possible.

First, let's explain what MOOSE does in the usual master/sub coupling scheme, using boundary coupling with subcycling = true as an example. Suppose you have a master application with a time step size of 1 second, and run NekRS as a sub-application with a time step size of 0.4 seconds that executes at the end of the master application time step. The calculation procedure involves:

1. Solve the master application from to seconds.

2. Transfer a variable representing flux (and its integral) from the master application to the NekRS sub-application) at .

3. Interpolate the flux from the NekRSMesh onto NekRS's GLL points, then normalize it and copy it from host to device.

4. Run a NekRS time step from to seconds.

5. Copy the temperature from device to host and then interpolate NekRS's temperature from NekRS's GLL points to the NekRSMesh.

6. Even though the flux data hasn't changed, and even though the temperature data isn't going to be used by the master application yet, ExternalProblem::solve() performs data transfers in/out of NekRS for every NekRS time step. Regardless of whether that data has changed or is needed yet by MOOSE, repeat steps 3-5 two times - once for a time step size of 0.4 seconds, and again for a time step size of 0.2 seconds (for the NekRS sub-application to "catch up" to the master application's overall time step length of 1 second.

If NekRS is run with a time step times smaller than its master application, this structuring of ExternalProblem represents unnecessary interpolations and CPU to GPU copies of the flux, and unnecessary GPU to CPU copies of the temperature and interpolations. NekRSProblem contains features that allow you to turn off these extra transfers. However, MOOSE's MultiApp system is designed in such a way that sub-applications know very little about their master applications (and for good reason - such a design is what enables such flexible multiphysics couplings). So, the only way that NekRS can definitively know that a data transfer from a master application is the first data transfer after the flux data has been updated, we monitor the value of a dummy postprocessor sent by the master application to NekRS. In other words, we define a postprocessor in the master application that just has a value of 1.

[Postprocessors]
  [synchronize]
    type = Receiver
    default = 1
  []
[]

We define this postprocessor as a Receiver postprocessor, but we won't actually use it to receive anything from other applications. Instead, we set the default value to 1 in order to indicate "true". Then, at the same time that we send new flux values to NekRS, we also pass this postprocessor.

[Transfers]
  [synchronize_in]
    type = MultiAppPostprocessorTransfer
    to_postprocessor = transfer_in
    from_postprocessor = synchronize
    to_multi_app = nek
  []
[]

We then receive this postprocessor in the sub-application. This basically means that, when the flux data is new, the NekRS sub-application will receive a value of "true" from the master-application (through the lens of this postprocessor), and send the data into NekRS.

For data transfer out of NekRS, we determine when the temperatue data is ready for use by MOOSE by monitoring how close the sub-application is to the synchronization time to the master application.

All that is required to use this reduced communication feature are to define the dummy postprocessor in the master application, and transfer it to the sub-application. Then, set the following options in NekRSProblem, which here are shown minimizing the communication going in and out of NekRS.

[Problem]
  type = NekRSProblem
  casename = 'sfr_7pin'
  synchronization_interval = parent_app
[]

Flux Normalization

By default, NekRSProblem will lump all "receiving" sidesets in NekRS together for the purpose of flux normalization. For example, suppose we are coupling NekRS to MOOSE heat conduction through two boundaries. Suppose MOOSE heat conduction predicts the following heat flux integrals on each of these boundaries:

• MOOSE boundary 1: 100.0 W

• MOOSE boundary 2: 200.0 W

When this data gets mapped to NekRS's spectral element mesh, we need to renormalize to ensure that we conserve power. This renormalization is necessary because the NekRS mesh can be entirely different from the MOOSE mesh, and integrating a nodal field on an origin mesh from MOOSE (say, a 1-st order Lagrange interpolation) will give a different integral value than integrating the interpolated nodal values on the receiving NekRS mesh (say, a 7-th order Lagrange interpolation of GLL points). By default, NekRSProblem lumps all receiving sidesets in the NekRS model together for the sake of normalization. For example, suppose that once received on the NekRS mesh and integrated, that the received flux has values of:

• NekRS boundary 1: 102.0 W

• NekRS boundary 2: 199.0 W

By default, this class will renormalize the NekRS flux in order to match the total MOOSE flux (of 300.0 W) to give:

• NekRS boundary 1:

• NekRS boundary 2:

The total power entering the flux is preserved, but not the distribution among the sidesets. This is usually a very small error and is an acceptable approximation for many analyses. However, we do also support an option where the heat flux on each sideset is preserved with the conserve_flux_by_sideset option. When setting this parameter to true, the heat flux on each NekRS sideset will instead be evaluated as:

• NekRS boundary 1:

• NekRS boundary 2:

This option requires using a vector postprocessor (we typically recommend the VectorOfPostprocessors object) to send indiviudal boundary flux values to preserve. This more advanced option cannot be used if:

• Your sidesets in the coupled MOOSE App are not individually specified. For instance, if your heat conduction solver has one sideset that maps to two NekRS sidesets, there's no natural way to figure out what the heat flux is in those sub-sidesets to send a correct value to NekRS.

• Any nodes are shared among the NekRS sidesets, such as a node on a corner between two sidesets. When we renormalize the NekRS flux, nodes that are present on more than one sideset will get renormalized multiple times, so there is no guarantee that we can enforce the total flux.

Limiting Temperature

For many NekRS simulations, such as those with sharp interior corners, it is often of use to "clip" the temperature to prevent oscillations. The temperature extracted from NekRS can be clipped before being transferred to a coupled MOOSE application by providing either one of both of the min_T and max_T postprocessors. With these specified, the temperature written to the NekRSMesh is adjusted to the range . min_T and max_T should be given in dimensional units.