Climatology quintiles timeout

Workshops, competitions and events AI Weather Quest
1

I am trying to use the retrieve_20yr_quintile_clim method from AI_WQ_package, but my script just hangs at that point and then times out with the following error:

  File "/usr/local/lib/python3.10/dist-packages/AI_WQ_package/retrieve_evaluation_data.py", line 89, in retrieve_20yr_quintile_clim
    session.retrbinary(f"RETR {remote_path}", f.write)
  File "/usr/lib/python3.10/ftplib.py", line 436, in retrbinary
    with self.transfercmd(cmd, rest) as conn:
  File "/usr/lib/python3.10/ftplib.py", line 393, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/usr/lib/python3.10/ftplib.py", line 359, in ntransfercmd
    resp = self.sendcmd(cmd)
  File "/usr/lib/python3.10/ftplib.py", line 281, in sendcmd
    return self.getresp()
  File "/usr/lib/python3.10/ftplib.py", line 252, in getresp
    raise error_temp(resp)
ftplib.error_temp: 421 Timeout

Is there anything I can do to remedy this situation? I’d rather use the provided quintiles than compute my own, but this is currently blocking me.

2

Hi Peter,

Bizarre. Could you please provide me with a copy of python command?

Kind regards,
Josh

3

Hi Joshua,

Here is the Python routine calling that function:

def get_quintile_clim(forecast_date, variable):
    """
    Retrieve the quintile climatology for a given forecast date and variable.

    Args:
        forecast_date (str): The forecast date in the format YYYYMMDD.
        variable (str): The variable to retrieve quintile climatology for.

    Returns:
        xr.DataArray: The quintile climatology.
    """

    fc_valid_date1, fc_valid_date2 = valid_dates(forecast_date)

    clim1 =  retrieve_evaluation_data.retrieve_20yr_quintile_clim(fc_valid_date1, variable, password=os.getenv('AIWQ_SUBMIT_PWD'))
    clim2 =  retrieve_evaluation_data.retrieve_20yr_quintile_clim(fc_valid_date2, variable, password=os.getenv('AIWQ_SUBMIT_PWD'))

    return clim1, clim2

def valid_dates(forecast_date):
    """
    Retrieve the valid dates for a given forecast date.

    Args:
        forecast_date (str): The forecast date in the format YYYYMMDD.

    Returns:
        tuple: A tuple containing the valid dates for the forecast. 
    """

    date_obj = datetime.strptime(forecast_date,"%Y%m%d") # get initial date as a date obj

    # add number of days to date object depending on lead time
    fc_valid_date_obj1 = date_obj + timedelta(days=4+(7*2)) # get to the next Monday then add number of weeks
    fc_valid_date_obj2 = date_obj + timedelta(days=4+(7*3))

    fc_valid_date1 = fc_valid_date_obj1.strftime("%Y%m%d") # convert date obj back to a string
    fc_valid_date2 = fc_valid_date_obj2.strftime("%Y%m%d") # convert date obj back to a string

    return fc_valid_date1, fc_valid_date2

The forecast date in this test was 2025-05-08

4

That’s odd.

I’ve just copied your code and performed the following command:
clim1, clim2 = get_quintile_clim('20250508','tas')

It successfully downloaded two climatology files. Can you check your password is set correctly?

Josh

5

I did double-check the password and it is correct…I suspect it is related to some network/security setting on the cluster I am running on, maybe something related to ftp? Trying on a different system does seem to work; I will try and pull all the climatology data ahead of time on that system so my real-time submissions are not affected by this issue.

6

There are climatology files to the end of 2025. I need to set-up automation of climatology upload. I will let you know once I do.

Good luck.

7

That seems to have worked, though it stumbled on the last Monday in 2025:

ftplib.error_perm: 550 Can't open /climatologies/2025/tas_20yrCLIM_WEEKLYMEAN_quintiles_20251229.nc: No such file or director

Maybe that one isn’t available yet?

8

Yes. This isn’t available.

Josh