MOM6 Development
Marshall Ward
2022-10-18
Presentation is available at
CI Testing
| Test | Description |
|---|---|
| Mem Layout | Symmetric vs Nonsymmetric |
| OpenMP | With threads |
| CPU Layout | 1 vs 2 CPUs |
| Rotation | 90° Domain Turn |
| NaN Init | Aggressive initialization |
| Regression | Answer changes |
Also tests for documentation, MacOS support, etc
Autoconf
.testing is based on Autoconf; let's try it:
# Build FMS
cd ac/deps
make -j
# Create ./configure
cd ../
autoreconf -if
# Build MOM6
cd ..
mkdir build
cd build
../ac/configure
make -jReproducibility
Numerical results must reproduce:
Even floating point!
Verification is based on ocean.stats:
Step, Day, Truncs, Energy/Mass, Maximum CFL, Mean sea level, Total Mass, Frac Mass Err
[days] [m2 s-2] [Nondim] [m] [kg] [Nondim]
0, 0.000, 0, En 3.3361965169932362E-01, CFL 0.01092, SL -0.0000E+00, Mass 3.95904E+18, Me 0.00E+00
144, 1.000, 0, En 3.3113616621947212E-01, CFL 0.01064, SL -0.0000E+00, Mass 3.95904E+18, Me -4.25E-18Energy/Mass is machine-precision and reproducible
Numerical "style"
No ambiguous ordering
- Bad:
a + b + c - OK:
(a + b) + c
- Bad:
Avoid intrinsics
sum()is ambiguous; what is the order?- use
reproducing_sum() - Avoid
sin(),exp(), evena**b
See my FortranCon talk for more about this
Bit Repro Debugging
In MOM_input:
DEBUG=TrueThis enables checksums in the output:
u-point: mean= 4.5198826930130449E-02 min= -2.2679844164834517E-03 max= 3.6386233175123922E-01 u Beginning of step_MOM [uv]
u-point: c= 320978 sw= 316992 se= 316992 nw= 316992 ne= 316992 u Beginning of step_MOM [uv]
v-point: mean= 0.0000000000000000E+00 min= 0.0000000000000000E+00 max= 0.0000000000000000E+00 v Beginning of step_MOM [uv]
v-point: c= 0 sw= 0 se= 0 nw= 0 ne= 0 v Beginning of step_MOM [uv]
h-point: mean= 1.0000000000000000E+03 min= 2.0022236368437586E+02 max= 1.7997776363156240E+03 Beginning of step_MOM h
h-point: c= 272640 sw= 274800 se= 274800 nw= 274740 ne= 274740 Beginning of step_MOM hSend to stderr (expect a lot of output!)
(See also DIAG_AS_CHKSUM)
Debug Syntax
h-point:Diagnostic is on a center point
mean,min,maxCalculated over the compute domain (no halos)
c=Bit checksum over compute domain
sw=,se=,nw=,ne=Checksum after domain shifted into halos
Call Tree
In MOM_unit:
VERBOSITY=9Produces call tree (and other info):
NOTE: callTree: ---> step_MOM(), MOM.F90
NOTE: callTree: loop DT cycles (step_MOM) n= 1
NOTE: callTree: o done with set_viscous_BBL (step_MOM)
NOTE: callTree: ---> step_MOM_dyn_split_RK2(), MOM_dynamics_split_RK2.F90
NOTE: callTree: o done with PressureForce (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with vertvisc_coef (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity[BT_cont] (step_MOM_dyn_split_RK2)
NOTE: callTree: ---> btstep(), MOM_barotropic.F90
NOTE: btstep is using a dynamic barotropic timestep of 7.500000E+01 seconds, max 8.529637E+01.
NOTE: callTree: <--- btstep()
NOTE: callTree: o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with btcalc[BT_cont_BT_thick] (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with horizontal_viscosity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree: ---> btstep(), MOM_barotropic.F90
NOTE: callTree: <--- btstep()
NOTE: callTree: o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree: <--- step_MOM_dyn_split_RK2()
NOTE: callTree: o finished step_MOM_dyn_split (step_MOM)
NOTE: callTree: o finished thickness_diffuse (step_MOM)
NOTE: callTree: <--- DT cycles (step_MOM)
NOTE: callTree: loop DT cycles (step_MOM) n= 2
NOTE: callTree: ---> step_MOM_dyn_split_RK2(), MOM_dynamics_split_RK2.F90
NOTE: callTree: o done with PressureForce (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with vertvisc_coef (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity[BT_cont] (step_MOM_dyn_split_RK2)
NOTE: callTree: ---> btstep(), MOM_barotropic.F90
NOTE: callTree: <--- btstep()
NOTE: callTree: o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with btcalc[BT_cont_BT_thick] (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with horizontal_viscosity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree: ---> btstep(), MOM_barotropic.F90
NOTE: callTree: <--- btstep()
NOTE: callTree: o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with CorAdCalc (step_MOM_dyn_split_RK2)
NOTE: callTree: <--- step_MOM_dyn_split_RK2()
NOTE: callTree: o finished step_MOM_dyn_split (step_MOM)
NOTE: callTree: o finished thickness_diffuse (step_MOM)
NOTE: callTree: <--- DT cycles (step_MOM)
....
NOTE: callTree: o calling extract_surface_state (step_MOM)
NOTE: callTree: ---> extract_surface_state(), MOM.F90
NOTE: callTree: <--- extract_surface_sfc_state()
NOTE: callTree: <--- step_MOM()
NOTE: callTree: <--- Main loop
Exercise
Check out fork and compile:
cd MOM6-examples/src/MOM6
git fetch
git checkout mysterious-crash
# You remember how to compile, right?Run Phillips 2-layer:
cd ../../ocean_only/Phillips_2layer
../../build/ocean_only/MOM6What do you see?
E.1 Output
Segmentation fault, no symbols:
MOM Day 0.000 0: En 3.336197E-01, MaxCFL 0.01092, Mass 3.959040000000E+18
Total Energy: 43B25478EC07BAB8 1.3208135458636902E+18
Total Mass: 3.9590400000000000E+18, Change: 0.0000000000000000E+00 Error: 0.00000E+00 ( 0.0E+00)
NOTE: callTree: <--- finish_MOM_initialization()
NOTE: callTree: ---> step_MOM(), MOM.F90
NOTE: callTree: loop DT cycles (step_MOM) n= 1
NOTE: callTree: o done with set_viscous_BBL (step_MOM)
NOTE: callTree: ---> step_MOM_dyn_split_RK2(), MOM_dynamics_split_RK2.F90
NOTE: callTree: o done with PressureForce (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with vertvisc_coef (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity[BT_cont] (step_MOM_dyn_split_RK2)
NOTE: callTree: ---> btstep(), MOM_barotropic.F90
NOTE: btstep is using a dynamic barotropic timestep of 7.500000E+01 seconds, max 8.529637E+01.
NOTE: callTree: <--- btstep()
NOTE: callTree: o done with vertvisc (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with continuity (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with btcalc[BT_cont_BT_thick] (step_MOM_dyn_split_RK2)
NOTE: callTree: o done with horizontal_viscosity (step_MOM_dyn_split_RK2)
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7fe847a6f9ff in ???
#1 0x55893a0b6dad in ???
#2 0x55893a086ee6 in ???
#3 0x55893a439281 in ???
#4 0x55893a39b4a0 in ???
#5 0x558939e76d0e in ???
#6 0x7fe847a5a28f in ???
#7 0x7fe847a5a349 in ???
#8 0x558939e76d44 in _start
at ../sysdeps/x86_64/start.S:115
#9 0xffffffffffffffff in ???
Segmentation fault (core dumped)
E.1 Hint
Rebuild with DEBUG:
$ make clean
$ make DEBUG=1 -jThis restores symbols:
-gSymbol tables
-fbacktraceEnable traceback (the report at the end)
Now check the symbols
Exercise
Another fork to check out:
cd MOM6-examples/src/MOM6
git fetch
git checkout demo-layout-issueRun with 1 and 4 CPUs:
cd ../../ocean_only/Phillips_2layer
mpirun -np 1 ../../build/ocean_only/MOM6
cp ocean.stats ocean.stats.p1
mpirun -np 4 ../../build/ocean_only/MOM6
cp ocean.stats ocean.stats.p4
diff ocean.stats.p1 ocean.stats.p4Do they agree?
Style Matters
From Griffies and Hallberg (2000):
This paper discusses a NUMERICAL CLOSURE,motivated from the ideas of
Smagorinsky, for use with a
bi
harmonic
operator.
The result is a highly
scale -selective,state- dependent
friction operator for use in
eddy -permitting geophysical fluid models.Would you write like this?
Syntax Style
Indent properly!
Respect line length (<=120 characters)
- <80 is even better!
snake_caseSpace around operators:
if (a == 0)No space in function args:
call f(a, b, c=0)
Array Style
This is bad:
u = 0.This is OK:
u(:,:,:) = 0.- But only for assignment and initialization
Explicit loops:
do j = js, je ; do i = is, ie u(i,j) = v(i,j) + w(i,j)**2 enddo ; enddoIndex convention:
ijon centers,IJon edges
Why Style?
- Consistent patterns: faster to read, faster to write
- Reduce the size of git commits
MOM6 Development Marshall Ward 2022-10-18 TODO