NekRSSeparateDomainProblem

This class allows for coupling of NekRS to a 1D T/H code such as SAM or THM. This coupling is performed using the "separate-domain" coupling strategy, where the coupling between codes is performed purely by updating boundary conditions between the domains. For most applications where there's a large flow loop, the 1D T/H code should model most of the loop, and NekRS should model only a portion of the loop that is of interest for CFD simulation.

This class must be used in conjunction with two other classes in Cardinal:

  1. NekRSMesh, which builds a mirror of the NekRS mesh in a MOOSE format so that all the usual Transfers understand how to send data into/out of NekRS. The settings on NekRSMesh also determine which coupling type (listed above) is used.

  2. NekTimeStepper, which allows NekRS to control its own time stepping.

Therefore, we recommend first reading the documentation for the above classes before proceeding here.

The coupling's setup is controlled using the coupling_type, which provides information for how the NekRS is coupled to the 1D T/H code, via NekRS's inlet, outlet, or 'inlet outlet'.

warningwarning

This class currently only supports dimensional solutions coming from NekRS. Nondimensional support is in progress.

Velocity, temperature, and scalar coupling

Figure 1 shows the coupling of velocity (V), temperature (T), and scalar01 (S01) for coupling_type = 'inlet outlet', where the 1D T/H code is coupled to NekRS's inlet and outlet boundaries.

Scalar coupling is allowed for up to 3 scalars (scalar01, scalar02, and scalar03) using the optional coupled_scalars parameter. Coupling scalars is useful when running a 1D T/H coupled to a NekRS RANS k-tau simulation with coupled passive scalar transport. For such a case, NekRS may use k and tau as scalar01 and scalar02, respectively, and the coupled passive scalar as scalar03. For coupling just scalar03, one would set coupled_scalars = scalar03.

Figure 1: Example boundary transfers for 'inlet outlet' coupling.

All inlet_XX are single values passed from the 1D T/H domain to the NekRS inlet boundary provided by inlet_boundary. For information on implementing these transferred values, refer to this section of the documentation.

All outlet_XX values are automated NekSideAverage postProcessors that are created on the NekRS boundary ID given by outlet_boundary. These postProcessors can then be transferred to the 1D T/H code using the MultiAppPostprocessorTransfer system.

Pressure coupling

Pressure information is transferred between NekRS and the 1D T/H code using a global pressure drop because the 1D T/H code only needs the overall pressure drop over the NekRS domain. inlet_P and outlet_P NekSideAverage postProcessors are created along with the pressure drop dP postProcessor. Figure 2 shows this process, which is performed no matter what coupling_type is given. dP can be transferred to the 1D T/H code using the MultiAppPostprocessorTransfer system.

Figure 2: Example pressure drop calculation over the NekRS domain.

Transferring data between NekRS and the 1D T/H code

A summary of postprocessors generated is shown in the following table for velocity, temperature, and scalar01 coupling. Postprocessors for scalar02 and scalar03 coupling follow the same pattern as scalar01. The postprocessor created depends on (1) the coupling_type given, (2) whether or not NekRS is solving for temperature and (3) the optional coupling_scalars provided.

Postprocessor createdcoupling_type = inletcoupling_type = outletcoupling_type = 'inlet outlet'
dP
inlet_V 
outlet_V 
inlet_T, if temperature solved , if temperature solved
outlet_T , if temperature solved, if temperature solved
inlet_S01, if scalar01 in coupled_scalars , if scalar01 in coupled_scalars
outlet_S01 , if scalar01 in coupled_scalars, if scalar01 in coupled_scalars

Example MultiApp transfers for velocity, temperature and scalar01 coupling are shown below for coupling_type = 'inlet outlet' and coupled_scalars = 'scalar01'. NekRS can run as either the MainApp or SubApp, but these example transfers assume NekRS is acting as the MainApp and the 1D_TH_code as the SubApp.


[Transfers]
  [dP_transfer]
    type = MultiAppPostprocessorTransfer
    to_multi_app = 1D_TH_code
    from_postprocessor = dP
    to_postprocessor   = fromNekRS_pressureDrop
  []
  [inlet_V_transfer]
    type = MultiAppPostprocessorTransfer
    from_multi_app = 1D_TH_code
    reduction_type = average
    from_postprocessor = toNekRS_velocity
    to_postprocessor   = inlet_V
  []
  [inlet_T_transfer]
    type = MultiAppPostprocessorTransfer
    from_multi_app = 1D_TH_code
    reduction_type = average
    from_postprocessor = toNekRS_temperature
    to_postprocessor   = inlet_T
  []
  [inlet_S01_transfer]
    type = MultiAppPostprocessorTransfer
    from_multi_app = 1D_TH_code
    reduction_type = average
    from_postprocessor = toNekRS_scalar01
    to_postprocessor   = inlet_S01
  []
  [outlet_V_transfer]
    type = MultiAppPostprocessorTransfer
    to_multi_app = 1D_TH_code
    from_postprocessor = outlet_V
    to_postprocessor   = fromNekRS_velocity
  []
  [outlet_T_transfer]
    type = MultiAppPostprocessorTransfer
    to_multi_app = 1D_TH_code
    from_postprocessor = outlet_T
    to_postprocessor   = fromNekRS_temperature
  []
  [outlet_S01_transfer]
    type = MultiAppPostprocessorTransfer
    to_multi_app = 1D_TH_code
    from_postprocessor = outlet_S01
    to_postprocessor   = fromNekRS_scalar01
  []
[]

Using transferred data for NekRS inlet boundary conditions

Data is "sent" into NekRS by writing into the nrs->usrwrk scratch space array, which NekRS makes available within the boundary condition functions in the .oudf file (on device, this array is technically called the nrs->o_usrwrk array). Table 1 shows the assignment of "slots" in the nrs->usrwrk scratch space array with quantities written by Cardinal. Because different quantities are written into Cardinal depending on the problem setup, if a particular slice is not needed for a case, it will be skipped. That is, the order of the various quantities is always the same in nrs->usrwrk, but with any unused slots cleared out.

Table 1: Quantities written into the scratch space array by Cardinal. In the last column, n indicates that the value is case-dependent depending on what features you have enabled.

SliceQuantityWhen Will It Be Created?How to Access in the .oudf File
0Inlet velocityif coupling_type includes inletbc->usrwrk[n * bc->fieldOffset + bc->idM]
1Inlet temperatureif coupling_type includes inlet and NekRS's case files include a temperature solvebc->usrwrk[n * bc->fieldOffset + bc->idM]
2Inlet scalar01if coupling_type includes inlet and coupled_scalars includes scalar01bc->usrwrk[n * bc->fieldOffset + bc->idM]
3Inlet scalar02if coupling_type includes inlet and coupled_scalars includes scalar02bc->usrwrk[n * bc->fieldOffset + bc->idM]
4Inlet scalar03if coupling_type includes inlet and coupled_scalars includes scalar03bc->usrwrk[n * bc->fieldOffset + bc->idM]

The total number of slots in the scratch space that are allocated by Cardinal is controlled with the n_usrwrk_slots parameter. If you need to use extra slices in nrs->usrwrk for other custom user actions, simply set n_usrwrk_slots to be greater than the number of slots strictly needed for coupling. At the start of your Cardinal simulation, a table will be printed to the screen to explicitly tell you what each slice in the scratch space holds. Any extra slots are noted as unused, and are free for non-coupling use.

For example, if your case couples NekRS to MOOSE with coupling_type = inlet and coupled_scalars = scalar02, but has a NekRS case without a temperature solve, the slices normally dedicated to storing inlet temperature and scalar01 are skipped. A table similar to the following would print out at the start of your simulation. You could use slices 2 onwards for custom purposes.

Listing 1: Table printed at start of Cardinal simulation that describes available scratch space for a case that couples NekRS to MOOSE via an inlet and with the second passive scalar, but without the first passive scalar or a temperature solve. A total of 7 slots are allocated by setting n_usrwrk_slots to 7

------------------------------------------------------------------------------------------------
| Quantity |           How to Access (.oudf)           |         How to Access (.udf)          |
------------------------------------------------------------------------------------------------
| velocity | bc->usrwrk[0 * bc->fieldOffset + bc->idM] | nrs->usrwrk[0 * nrs->fieldOffset + n] |
| scalar02 | bc->usrwrk[1 * bc->fieldOffset + bc->idM] | nrs->usrwrk[1 * nrs->fieldOffset + n] |
| unused   | bc->usrwrk[2 * bc->fieldOffset + bc->idM] | nrs->usrwrk[2 * nrs->fieldOffset + n] |
| unused   | bc->usrwrk[3 * bc->fieldOffset + bc->idM] | nrs->usrwrk[3 * nrs->fieldOffset + n] |
| unused   | bc->usrwrk[4 * bc->fieldOffset + bc->idM] | nrs->usrwrk[4 * nrs->fieldOffset + n] |
| unused   | bc->usrwrk[5 * bc->fieldOffset + bc->idM] | nrs->usrwrk[5 * nrs->fieldOffset + n] |
| unused   | bc->usrwrk[6 * bc->fieldOffset + bc->idM] | nrs->usrwrk[6 * nrs->fieldOffset + n] |
------------------------------------------------------------------------------------------------
warningwarning

Allocation of nrs->usrwrk and nrs->o_usrwrk is done automatically. If you attempt to run a NekRS input file that accesses bc->usrwrk in the .oudf file without a Cardinal executable (e.g. like nrsmpi case 4), then that scratch space will have to be manually allocated in the .udf file, or else your input will seg fault. This will not be typically encountered by most users, but if you really do want to run the NekRS input files intended for a Cardinal case with the NekRS executable (perhaps for debugging), we recommend simply replacing bc->usrwrk by a dummy value, such as bc->flux = 0.0 for the boundary heat flux use case. This just replaces a value that normally comes from MOOSE by a fixed value. All other aspects of the NekRS case files should not require modification.

In other words, the scratch space slots contain pointwise values for each (singly-valued) postprocessor value on all the GLL points on the corresponding boundary. So, the values in each "slot" correspond to the following postprocessors:

  • Slot with velocity: inlet_V postprocessor value

  • Slot with temperature: inlet_T postprocessor value

  • Slot with scalar01: inlet_S01 postprocessor value

  • Slot with scalar02: inlet_S02 postprocessor value

  • Slot with scalar03: inlet_S03 postprocessor value

This allows the user the freedom to choose what type of profile they want to implement at the inlet boundary, i.e. flat or fully-developed. Below are a few example implementations.

Example for a flat velocity(x) profile:

void velocityDirichletConditions(bcData *bc)
{
  bc->u = bc->usrwrk[bc->idM];
  bc->v = 0.0;
  bc->w = 0.0;
}

Example for a flat temperature and scalar profiles:

void scalarDirichletConditions(bcData *bc)
{
  if(bc->scalarId==0) bc->s  = bc->usrwrk[bc->idM + bc->fieldOffset];   // temperature
  if(bc->scalarId==1) bc->s  = bc->usrwrk[bc->idM + 2*bc->fieldOffset]; // scalar01
  if(bc->scalarId==2) bc->s  = bc->usrwrk[bc->idM + 3*bc->fieldOffset]; // scalar02
  if(bc->scalarId==3) bc->s  = bc->usrwrk[bc->idM + 4*bc->fieldOffset]; // scalar03
}

Example for fully-developed velocity(x) profile for laminar pipe flow:

void velocityDirichletConditions(bcData *bc)
{
  dfloat yi = bc->y;
  dfloat zi = bc->z;
  dfloat rsq  = yi*yi + zi*zi;
  dfloat Rsq  = 0.5*0.5; // Radius^2
  dfloat Uavg  = bc->usrwrk[bc->idM];

  bc->u = 2*Uavg*(1-rsq/Rsq);
  bc->v = 0.0;
  bc->w = 0.0;
}

Other Features

This class mainly facilitates data transfers to and from NekRS. A number of other features are implemented in order to enable nondimensional solutions, improved communication, and convenient solution modifications. These are described in this section.

Outputting the Scratch Array

This class (and all other NekRS wrappings in Cardinal) allows you to write slots in the nrs->usrwrk scratch space array to NekRS field files. This can be useful for viewing the data sent from MOOSE to NekRS (for problem classes that involve multiphysics), as well as to visualize custom user usage of nrs->usrwrk, such as for fetching a wall distance computation from the Nek5000 backend. To write the scratch space to a field file, set usrwrk_output to an array with each "slot" in the nrs->usrwrk array that you want to write. Then, specify a filename prefix to use to name each field file.

In the example below, the first two "slots" in the nrs->usrwrk array will be written to field files on the same interval that NekRS writes its usual field files. These files will be named aaabrick0.f00001, etc. and cccbrick0.f00001, etc. Based on limitations in how NekRS writes its files, the fields written to these files will all be named temperature when visualized.

[Problem]
  type = NekRSStandaloneProblem
  casename = 'brick'
  usrwrk_output = '0 1'
  usrwrk_output_prefix = 'aaa ccc'
[]
(test/tests/nek_file_output/usrwrk/nek.i)

Input Parameters

  • casenameCase name for the NekRS input files; this is in .par, .udf, .oudf, and .re2.

    C++ Type:std::string

    Controllable:No

    Description:Case name for the NekRS input files; this is in .par, .udf, .oudf, and .re2.

  • coupling_typeNekRS boundary types to couple to a 1-D T/H code

    C++ Type:MultiMooseEnum

    Options:inlet, outlet

    Controllable:No

    Description:NekRS boundary types to couple to a 1-D T/H code

  • inlet_boundaryNekRS inlet boundary ID

    C++ Type:std::vector<int>

    Controllable:No

    Description:NekRS inlet boundary ID

  • outlet_boundaryNekRS outlet boundary ID

    C++ Type:std::vector<int>

    Controllable:No

    Description:NekRS outlet boundary ID

Required Parameters

  • Cp_01Heat capacity parameter value for non-dimensional solution

    Default:1

    C++ Type:double

    Controllable:No

    Description:Heat capacity parameter value for non-dimensional solution

  • L_ref1Reference length scale value for non-dimensional solution

    Default:1

    C++ Type:double

    Controllable:No

    Description:Reference length scale value for non-dimensional solution

  • T_ref0Reference temperature value for non-dimensional solution

    Default:0

    C++ Type:double

    Controllable:No

    Description:Reference temperature value for non-dimensional solution

  • U_ref1Reference velocity value for non-dimensional solution

    Default:1

    C++ Type:double

    Controllable:No

    Description:Reference velocity value for non-dimensional solution

  • constant_interval1Constant interval (in units of number of time steps) with which to synchronize the NekRS solution

    Default:1

    C++ Type:unsigned int

    Controllable:No

    Description:Constant interval (in units of number of time steps) with which to synchronize the NekRS solution

  • coupled_scalarsNekRS scalars to couple to a 1-D T/H code

    C++ Type:MultiMooseEnum

    Options:scalar01, scalar02, scalar03

    Controllable:No

    Description:NekRS scalars to couple to a 1-D T/H code

  • dT_ref1Reference temperature range value for non-dimensional solution

    Default:1

    C++ Type:double

    Controllable:No

    Description:Reference temperature range value for non-dimensional solution

  • disable_fld_file_outputFalseWhether to turn off all NekRS field file output writing (for the usual field file output - this does not affect writing the usrwrk with 'usrwrk_output')

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Whether to turn off all NekRS field file output writing (for the usual field file output - this does not affect writing the usrwrk with 'usrwrk_output')

  • first_reserved_usrwrk_slot0Slice (zero-indexed) in nrs->usrwrk where Cardinal will begin reading/writing data; this can be used to shift the usrwrk slots reserved by Cardinal, so that you can use earlier slices for custom purposes

    Default:0

    C++ Type:unsigned int

    Controllable:No

    Description:Slice (zero-indexed) in nrs->usrwrk where Cardinal will begin reading/writing data; this can be used to shift the usrwrk slots reserved by Cardinal, so that you can use earlier slices for custom purposes

  • minimize_transfers_inFalseWhether to only synchronize nekRS for the direction TO_EXTERNAL_APP on multiapp synchronization steps

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Whether to only synchronize nekRS for the direction TO_EXTERNAL_APP on multiapp synchronization steps

  • minimize_transfers_outFalseWhether to only synchronize nekRS for the direction FROM_EXTERNAL_APP on multiapp synchronization steps

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Whether to only synchronize nekRS for the direction FROM_EXTERNAL_APP on multiapp synchronization steps

  • n_usrwrk_slots7Number of slots to allocate in nrs->usrwrk to hold fields either related to coupling (which will be populated by Cardinal), or other custom usages, such as a distance-to-wall calculation

    Default:7

    C++ Type:unsigned int

    Controllable:No

    Description:Number of slots to allocate in nrs->usrwrk to hold fields either related to coupling (which will be populated by Cardinal), or other custom usages, such as a distance-to-wall calculation

  • nondimensionalFalseWhether NekRS is solved in non-dimensional form

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Whether NekRS is solved in non-dimensional form

  • outputField(s) to output from NekRS onto the mesh mirror

    C++ Type:MultiMooseEnum

    Options:temperature, pressure, velocity, scalar01, scalar02, scalar03

    Controllable:No

    Description:Field(s) to output from NekRS onto the mesh mirror

  • regard_general_exceptions_as_errorsFalseIf we catch an exception during residual/Jacobian evaluaton for which we don't have specific handling, immediately error instead of allowing the time step to be cut

    Default:False

    C++ Type:bool

    Controllable:No

    Description:If we catch an exception during residual/Jacobian evaluaton for which we don't have specific handling, immediately error instead of allowing the time step to be cut

  • rho_01Density parameter value for non-dimensional solution

    Default:1

    C++ Type:double

    Controllable:No

    Description:Density parameter value for non-dimensional solution

  • skip_final_field_fileFalseBy default, we write a NekRS field file on the last time step; set this to true to disable

    Default:False

    C++ Type:bool

    Controllable:No

    Description:By default, we write a NekRS field file on the last time step; set this to true to disable

  • solveTrueWhether or not to actually solve the Nonlinear system. This is handy in the case that all you want to do is execute AuxKernels, Transfers, etc. without actually solving anything

    Default:True

    C++ Type:bool

    Controllable:Yes

    Description:Whether or not to actually solve the Nonlinear system. This is handy in the case that all you want to do is execute AuxKernels, Transfers, etc. without actually solving anything

  • synchronization_intervalconstantWhen to synchronize the NekRS solution with the mesh mirror. By default, the NekRS solution is mapped to/receives data from the mesh mirror for every time step.

    Default:constant

    C++ Type:MooseEnum

    Options:constant, parent_app

    Controllable:No

    Description:When to synchronize the NekRS solution with the mesh mirror. By default, the NekRS solution is mapped to/receives data from the mesh mirror for every time step.

  • usrwrk_outputUsrwrk slot(s) to output to NekRS field files; this can be used for viewing the quantities passed from MOOSE to NekRS after interpolation to the CFD mesh. Can also be used for any slots in usrwrk that are written by the user, but unused for coupling.

    C++ Type:std::vector<unsigned int>

    Controllable:No

    Description:Usrwrk slot(s) to output to NekRS field files; this can be used for viewing the quantities passed from MOOSE to NekRS after interpolation to the CFD mesh. Can also be used for any slots in usrwrk that are written by the user, but unused for coupling.

  • usrwrk_output_prefixString prefix to use for naming the field file(s); only the first three characters are used in the name based on limitations in NekRS

    C++ Type:std::vector<std::string>

    Controllable:No

    Description:String prefix to use for naming the field file(s); only the first three characters are used in the name based on limitations in NekRS

  • write_fld_filesFalseWhether to write NekRS field file output from Cardinal. If true, this will disable any output writing by NekRS itself, and instead produce output files with names a01...a99pin, b01...b99pin, etc.

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Whether to write NekRS field file output from Cardinal. If true, this will disable any output writing by NekRS itself, and instead produce output files with names a01...a99pin, b01...b99pin, etc.

Optional Parameters

  • allow_initial_conditions_with_restartFalseTrue to allow the user to specify initial conditions when restarting. Initial conditions can override any restarted field

    Default:False

    C++ Type:bool

    Controllable:No

    Description:True to allow the user to specify initial conditions when restarting. Initial conditions can override any restarted field

  • restart_file_baseFile base name used for restart (e.g. / or /LATEST to grab the latest file available)

    C++ Type:FileNameNoExtension

    Controllable:No

    Description:File base name used for restart (e.g. / or /LATEST to grab the latest file available)

Restart Parameters

  • control_tagsAdds user-defined labels for accessing object parameters via control logic.

    C++ Type:std::vector<std::string>

    Controllable:No

    Description:Adds user-defined labels for accessing object parameters via control logic.

  • default_ghostingFalseWhether or not to use libMesh's default amount of algebraic and geometric ghosting

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Whether or not to use libMesh's default amount of algebraic and geometric ghosting

  • enableTrueSet the enabled status of the MooseObject.

    Default:True

    C++ Type:bool

    Controllable:No

    Description:Set the enabled status of the MooseObject.

Advanced Parameters

  • kernel_coverage_block_listList of subdomains for kernel coverage check. The meaning of this list is controlled by the parameter 'kernel_coverage_check' (whether this is the list of subdomains to be checked, not to be checked or not taken into account).

    C++ Type:std::vector<SubdomainName>

    Controllable:No

    Description:List of subdomains for kernel coverage check. The meaning of this list is controlled by the parameter 'kernel_coverage_check' (whether this is the list of subdomains to be checked, not to be checked or not taken into account).

  • material_coverage_block_listList of subdomains for material coverage check. The meaning of this list is controlled by the parameter 'material_coverage_check' (whether this is the list of subdomains to be checked, not to be checked or not taken into account).

    C++ Type:std::vector<SubdomainName>

    Controllable:No

    Description:List of subdomains for material coverage check. The meaning of this list is controlled by the parameter 'material_coverage_check' (whether this is the list of subdomains to be checked, not to be checked or not taken into account).

  • material_coverage_checkTRUEControls, if and how a material subdomain coverage check is performed. With 'TRUE' or 'ON' all subdomains are checked (the default). Setting 'FALSE' or 'OFF' will disable the check for all subdomains. To exclude a predefined set of subdomains 'SKIP_LIST' is to be used, while the subdomains to skip are to be defined in the parameter 'material_coverage_block_list'. To limit the check to a list of subdomains, 'ONLY_LIST' is to be used (again, using the parameter 'material_coverage_block_list').

    Default:TRUE

    C++ Type:MooseEnum

    Options:FALSE, TRUE, OFF, ON, SKIP_LIST, ONLY_LIST

    Controllable:No

    Description:Controls, if and how a material subdomain coverage check is performed. With 'TRUE' or 'ON' all subdomains are checked (the default). Setting 'FALSE' or 'OFF' will disable the check for all subdomains. To exclude a predefined set of subdomains 'SKIP_LIST' is to be used, while the subdomains to skip are to be defined in the parameter 'material_coverage_block_list'. To limit the check to a list of subdomains, 'ONLY_LIST' is to be used (again, using the parameter 'material_coverage_block_list').

Simulation Checks Parameters

  • parallel_barrier_messagingFalseDisplays messaging from parallel barrier notifications when executing or transferring to/from Multiapps (default: false)

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Displays messaging from parallel barrier notifications when executing or transferring to/from Multiapps (default: false)

  • verbose_multiappsFalseSet to True to enable verbose screen printing related to MultiApps

    Default:False

    C++ Type:bool

    Controllable:No

    Description:Set to True to enable verbose screen printing related to MultiApps

  • verbose_setupfalseSet to 'true' to have the problem report on any object created. Set to 'extra' to also display all parameters.

    Default:false

    C++ Type:MooseEnum

    Options:false, true, extra

    Controllable:No

    Description:Set to 'true' to have the problem report on any object created. Set to 'extra' to also display all parameters.

Verbosity Parameters

  • show_invalid_solution_consoleTrueSet to true to show the invalid solution occurance summary in console

    Default:True

    C++ Type:bool

    Controllable:No

    Description:Set to true to show the invalid solution occurance summary in console

Solution Validity Control Parameters

Input Files