WAM crashes when writing output, fort.30

AIFS, IFS and OpenIFS OpenIFS
, ,
1

Hi 

I tried for the very first time to activate WAM output in OpenIFS cy40, but got a model crash that I don't understand. 

The crash happens on the last time step (step 119 of 120) and the message is

 THE WAVE MODEL IS CALLED FOR TWO-WAY INTERACTION BETWEEN WIND AND WAVES


  NORMS OF GLOBAL INPUT ATMOSPHERIC FIELDS :

  *Can only be compared if the log files

   are for processor:            1

   have the same number of processors:           39

   and have the same model decomposition: TWO D

  Neutral 10m U    -0.894164721618000       -17.7012133823277

   18.5800319486246            33716           1          39 F

                   HEX: BFEC9CFF5592C298  C031B382B8609D7F  4032947CF949FA52

  Neutral 10m V    -0.594902568418843       -22.7843575514328

   21.5759745558943            33716           1          39 F

                   HEX: BFE309711C754A6B  C036C8CBA80FC640  403593731188E4EF

  Air density        1.21361576350479       0.830975331069688

   1.72569257056212            33716           1          39 F

                   HEX: 3FF36AF85CE29C3D  3FEA975993D74444  3FFB9C6FD0183B4A

  w*                0.682200589046991       0.000000000000000E+000

   2.39370861946792            33716           1          39 F

                   HEX: 3FE5D496546899B0  0000000000000000  40032650B46624E5

  Sea ice fraction  -43.7506553350302       -999.000000000000

   1.00000000000000            33716           1          39 F

                   HEX: C045E015795942E8  C08F380000000000  3FF0000000000000



   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 19820105221500  !!!!!!!!!!!!

 FILE_TRANSFER: MOVE FILE fort.30 TO ./MPP19820101000118


   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 19820105223000  !!!!!!!!!!!!

 ******************************************

 *                                        *

 *   ERROR FOLLOWING CALL TO INQUIRE   *

 *   IN FILE_TRANSFER                     *

 *   COULD NOT FIND FILE

 ./fort.30



 *                                        *

 ******************************************

 ABOR1 CALLED

 !! WAVE MODEL HAS ABORTED !!!!


I have no idea where fort.30 is supposed to come from. Turning off WAM output does not actually help either. I seem to remember there was a problem with WAM writing restarts, but this error does not seem familiar. The namelist for OpenIFS is fort.4, and for WAM I thought it is wam_namelist (both enclosed). 

Does anyone have any idea what the problem could be? 

Best regards


Joakim 

fort-1.4

wam_namelist-1

2

Funnily enough I found and fixed this bug last week. I put a post on the forums with the fix here:

Wave model bug if wam output and restarts are both enabled

How's that for service?  (big grin)

3

Wow! That is spectacular service indeed! (thumbs up)
Reporting a bug after 7pm and having it solved within 2 hours is unheard of. Works like a charm now! 

But...when I do "cdo -t ecmwf -f nc copy MPP19820101000119 MPP19820101000119.nc" the variable names are all wrong. 
I'm guessing CDO is using the GRIB codes from some old IFS version or something. Do you know if I can tell CDO to use the proper GRIB codes to translate variable names? 
I now realise I also have this problem with runoff output from HTESSEL. 

Cheers

Joakim 

4

What happens without the "-t ecmwf"?  I think with that option cdo only recognises GRIB1 fields and not GRIB2. That might be the problem but I'm not that familiar with cdo.

Something else to make you aware of, WAM has a nasty feature when run in OpenIFS where it uses a 'call system()' to rename the fort.30 file to MPP<date stamp>. The subroutine gsfile calls file_transfer.F and will make this call. It's far from ideal. The system call will block execution and in my tests raises the memory requirements significantly.  I have logged an internal issue for this to be fixed in 43r3 but I won't make any changes for 40r1.

Cheers,  Glenn

5

Hi Glenn

Without "-t ecmwf" I only get names like "var130", i.e. "var" plus the grib code. I'll keep checking then and see if there's a good way to do this. 

Thanks for the heads up with the problem. Sounds like it might halt the model for a little bit during the final model step, especially if I'm running T511 or T1279. 
Could I simply comment this line out? The file renaming could be done in the post processing script afterwards instead. 

Cheers
Joakim 

6

Unfortunately I don't use cdo much and we use metview for handling wave model files. I suggest emailing servicedesk _at_ ecmwf to ask about grib codes for the wave model in cdo. That will get routed to user support and someone there might know.  I'll watch the internal issue and then we can post more information here and on the user guide.

As for the fort.30 issue, it may not be noticeable in a long run. If you disable the system call to 'mv fort30', check the code carefully. The wave model might reopen the file for the next output write time and overwrite the previous results or delete them. Not sure. If it was me, I'd live with it until it gets fixed in 43r3.

Cheers,  Glenn

7

Hi Glenn

I've fired off an email to service desk to see what they think. At some point, it would be awesome to have one script that takes the ICMGG and ICMSH files and returns netCDF files with correct variable names and regular Gaussian grid. 
ECHAM users can use the CDO "afterburner" which is really powerful for this kind of thing and converts the GRIB output to CMIP-compliant netCDF data. Having something similar for OpenIFS (shipped with ecCodes or something) would be fantastic. 

I'll wait and see what service desk says, and post any results here if I find some good general solution. 

Cheers
Joakim


PS "If it was me, I'd live with it until it gets fixed in 43r3" - Glenn Carver. This is a quote I might use at the next time I have to present the state of our coupled model... (wink) 


8

What's the CDO 'afterburner'? Never heard that before.

eccodes' grib_to_netcdf will give a cf compliant netcdf file. But what it doesn't do is the interpolation to a regular lat-lon grid, which I guess is that you are also referring to.  However, if you use ECMWF's metview, it will do interpolation and write to netcdf. So maybe what we need to look at is a containerized version of metview customized to do exactly as you describe. I would favour this approach because metview uses the ECMWF interpolation & spectral transform routines and would be consistent with what we do internally.  I think metview is available with most linux distros now but a containerized version specifically designed just to convert ICM* output to netcdf would be a nice app.  What do you think?

Glenn

p.s. yes, why not blame it all on me (big grin)


9

Hi Glenn

The CDO afterburner (invoked with "cdo after <namelist>") is basically a built-in standard post processing tool for ECHAM. It does SH→GG interpolations, hybrid→pressure interpolations, and also computes some derived variables (cloud radiative forcing, stream functions etc) and standard statistics (I think QBO etc. but I'm not sure). https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-7540002.15.2 (search for afterburner). 
So the user doesn't really have to understand anything about grib tables or Gaussian grids to get an end product with daily-mean fields on a regular lon/lat grid. 

I'm not familiar with metview, but perhaps I should take the time to learn a bit about it. It would be awesome if there was a simple metview script that takes all files like "ICMGGgu5a+?????? ICMSHgu5a+??????" and returns files like "gu5a_19920101_0000_19920115_0000_plevels.nc" and similar for other vertical coordinates. Preferably using OpenMP as much as possible, since the spectral→gp transforms for e.g. 1 month of 6-hourly TL1279 take a really long time! 

I can do this with CDO at the moment although I lose variable names for HTESSEL and WAM. But it might be tricky with cy43, because I don't think CDO can do spectral→gp transforms for cubic-icosahedral grids. CDO also has a couple of nasty problems, e.g. confusing "lsp" (large-scale precip) and "lnsp" (log surf pressure) since it does not have the correct GRIB tables. 
A containerized version of Metview would indeed be the best way to go, at least until the day when XIOS can re-grid spectral and reduced grid variables! 

Best regards
Joakim