"NameError: name 'reduce' is not defined" occurred when download ERA5-Land daily statistics

Software and Tools API
, ,
1

Hi. I am trying to download the daily ERA5-Land dataset by calling the application of daily statistics through the following scripts. It worked fine until yesterday. The same dataset could be downloaded smoothly.

However, using the same scripts today with slight revisions in variable names, etc., the NameError of "name 'reduce' is not defined" occurred (see following NameError). I suspect that this might be because the codes were compiled by python 3.9. While not intend to use python 2, I wonder what I should do to solve this problem and batch download daily statistics.

Scripts for downloading ERA5-Land dataset

# -*- coding: utf-8 -*-
 
import cdsapi
import requests
 
# CDS API script to use CDS service to retrieve daily ERA5* variables and iterate over
# all months in the specified years.
 
# Requires:
# 1) the CDS API to be installed and working on your system
# 2) You have agreed to the ERA5 Licence (via the CDS web page)
# 3) Selection of required variable, daily statistic, etc
 
# Output:
# 1) separate netCDF file for chosen daily statistic/variable for each month
 
c = cdsapi.Client(timeout=300)
 
# Uncomment years as required
 
years =  ['2012']
 
# Retrieve all months for a given year.
 
months = ['10', '11', '12']
 
# For valid keywords, see Table 2 of:
# https://datastore.copernicus-climate.eu/documents/app-c3s-daily-era5-statistics/C3S_Application-Documentation_ERA5-daily-statistics-v2.pdf
 
# select your variable; name must be a valid ERA5 CDS API name.
var = "volumetric_soil_water_layer_2"
 
# Select the required statistic, valid names given in link above
stat = "daily_mean"
 
# Loop over years and months
 
for yr in years:
    for mn in months:
        result = c.service(
        "tool.toolbox.orchestrator.workflow",
        params={
             "realm": "c3s",                                  
             "project": "app-c3s-daily-era5-statistics",
             "version": "master",
             "kwargs": {
                 "dataset": "reanalysis-era5-land",          
                 "product_type": "reanalysis",
                 "variable": var,
                 "statistic": stat,
                 "year": yr,
                 "month": mn,
                 "time_zone": "UTC+00:0",
                 "frequency": "1-hourly",
                 "grid": "0.1/0.1",                          
                 "area": {"lat": [23, 90], "lon": [-180, 180]}
#
# Users can change the output grid resolution and selected area
#
#                "grid": "1.0/1.0",
#                "area":{"lat": [10, 60], "lon": [65, 140]}
 
                 },
        "workflow_name": "application"
        })
 
# set name of output file for each month (statistic, variable, year, month
 
        file_name = "D:/%s/" %(var) + "ERA5_Land_" + stat + "_" + var + "_" + yr + "_" + mn + ".nc"
 
        location=result[0]['location']
        res = requests.get(location, stream = True)
        print("Writing data to " + file_name)
        with open(file_name,'wb') as fh:
            for r in res.iter_content(chunk_size = 1024):
                fh.write(r)
        fh.close()

NameError: name ‘reduce’ is not defined

D:\JUNO\UCAS\JunoOwnWork\PermafrostPhenology\Codes\Juno\DownloadClimaticData\batch_download_scripts>C:/Users/AERO/Anaconda3/envs/AcaPheno/python.exe D:/JUNO/s/ERA5-Land_daily_download_vswl2_aero_5_2005.py
2022-12-21 21:24:15,577 INFO Welcome to the CDS
2022-12-21 21:24:15,577 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/tasks/services/tool/toolbox/orchestrator/workflow/clientid-34b6c140a8134ad282f83ae52e0bdd50
2022-12-21 21:24:16,169 INFO Request is queued
2022-12-21 21:24:17,525 INFO Request is running
2022-12-21 21:33:06,196 INFO Request is failed
2022-12-21 21:36:11,011 ERROR Message:
2022-12-21 21:36:11,017 ERROR Reason:  Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall
    resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context)
  File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow
    results = workflow_bare_func(**kwargs)
  File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 774, in application
    [daily_stat_concat, daily_stat], dim='lon'
  File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 557, in concat
    non_common_coords = list(reduce(lambda x, y: set(x)^set(y), coords))
NameError: name 'reduce' is not defined
 
2022-12-21 21:36:11,018 ERROR   Traceback (most recent call last):
2022-12-21 21:36:11,019 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 59, in handle_request
2022-12-21 21:36:11,019 ERROR       result = cached(context.method, proc, context, context.args, context.kwargs)
2022-12-21 21:36:11,019 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
2022-12-21 21:36:11,020 ERROR       result = proc(context, *context.args, **context.kwargs)
2022-12-21 21:36:11,020 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 124, in __call__
2022-12-21 21:36:11,021 ERROR       return p(*args, **kwargs)
2022-12-21 21:36:11,021 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 60, in __call__
2022-12-21 21:36:11,022 ERROR       return self.proc(context, *args, **kwargs)
2022-12-21 21:36:11,022 ERROR     File "/home/cds/cdsservices/services/workflow.py", line 35, in execute
2022-12-21 21:36:11,023 ERROR       raise exceptions.CDSException(True, True, logging + traceback, '', uri)
2022-12-21 21:36:11,024 ERROR   cdsclient.exceptions.CDSException: Traceback (most recent call last):
2022-12-21 21:36:11,024 ERROR     File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall
2022-12-21 21:36:11,024 ERROR       resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context)
2022-12-21 21:36:11,025 ERROR     File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow
2022-12-21 21:36:11,025 ERROR       results = workflow_bare_func(**kwargs)
2022-12-21 21:36:11,026 ERROR     File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 774, in application
2022-12-21 21:36:11,026 ERROR       [daily_stat_concat, daily_stat], dim='lon'
2022-12-21 21:36:11,027 ERROR     File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 557, in concat
2022-12-21 21:36:11,027 ERROR       non_common_coords = list(reduce(lambda x, y: set(x)^set(y), coords))
2022-12-21 21:36:11,027 ERROR   NameError: name 'reduce' is not defined
Traceback (most recent call last):
  File "D:\JUNO\ERA5-Land_daily_download_vswl2_aero_5_2005.py", line 47, in <module>
  File "C:\Users\AERO\Anaconda3\envs\AcaPheno\lib\site-packages\cdsapi\api.py", line 366, in service
    result = self._api(
  File "C:\Users\AERO\Anaconda3\envs\AcaPheno\lib\site-packages\cdsapi\api.py", line 506, in _api
    raise Exception(
Exception: . Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall
    resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context)
  File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow
    results = workflow_bare_func(**kwargs)
  File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 774, in application
    [daily_stat_concat, daily_stat], dim='lon'
  File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 557, in concat
    non_common_coords = list(reduce(lambda x, y: set(x)^set(y), coords))
NameError: name 'reduce' is not defined```
2

Please, have you solved the problems?

3

I also meet the same problem.  Have you solved the problems?

I just the API scripts listed in Retrieve daily ERA5/ERA5-Land data using the CDS API

It's ok when I downloaded "2m_temperature“

But when I want to download ”surface_net_solar_radiation“ ， I meet the same error 

as below:

" NameError: name 'reduce' is not defined "


4

Hi,

please be aware that not all ERA5-Land variables are available for download in the Daily statistics app. 

I suggest to check on the app for the list.


Thanks

5

Thank you for the reply! I just want to download "surface_net_solar_radiation" as shown in Tab2 in
https://datastore.copernicus-climate.eu/documents/app-c3s-daily-era5-statistics/C3S_Application-Documentation_ERA5-daily-statistics-v2.pdf
Is the list in this Tab2 not the list mentioned by you ?

6

Hi,

I am pretty sure that this problem is not strictly related to the variable we choose, as the same error occured when downloading '2m_temperature' for me. I tested the source code for application of "daily statistics calculated from ERA5 data" in Toolbox, and met the totally same NameError: name 'reduce' is not defined.

This error can be simple resolved in Toolbox by adding 'from functools import reduce' at the start of the script. To make it possible for bulk access, I wonder if it is possible for the ECMWF supportors to revise the source code?

7

I used the cdstoolbox on the webpage of Toolbox Editor directly and got the daily data. 

First use ct.catalogue.retrieve() (this can be generated on the download page) to get the data, then use the ct.climate.daily_mean(data) to calculate. 

A link with the result will be generated on the left of the page.

8

Thanks for your tips, and the proposed process works fine for me.

However, two drawbacks remain:

1) This can only retrieve daily data one by one, as the new selection limit allows for no more than a month at the time in a single request (Release of ERA5-Land-T data and new selection limit in place);

2) This cannot  automatically rename the daily data, making it hard to distinguish each downloaded file.

It will be much appreciated if you have any further suggestions.

9

1) In fact, I met the same problem and haven't solved;

2) I pip the cdstoolbox-remote locally and do some try but failed. I find that if I want to rename the data before downloading, I should revise the source code. 

    If renaming the file after downloading, I think maybe you can copy the code to your local environment with cdstoolbox-remote prepared and add a main function that can rename the file instantly after every downloading. And the loop can help do the batch processing. However, I'm not sure is it ok, because I haven't tried.

    

    


10

Thank  you for the reply!

So now this  problem can be resolved in cdstoolbox (I am not familiar with that), meanwhile can not be resolved in cdsapi?

I've tried to add 'from functools import reduce'  in my script using cdsapi,

but it can not work.

Should I try to make another script with cdstoolbox ?

11

I didn't resolve this problem in my local environment (i.e. importing cdsapi). So instead, I used the cdstoolbox. Still waiting for someone to radically resolve this.

I assumed that you add 'from functools import reduce' in your own script, which doesn't call reduce function. Instead, this revision should be applied to the source code of application 'Daily statistics calculated from ERA5 data' (https://cds.climate.copernicus.eu/cdsapp#!/software/app-c3s-daily-era5-statistics?tab=appcode).

To use the cdstoolbox, you can firstly copy the above codes, and then do slight changes to the original code, including:

1) Add 'from functools import reduce';

2) Delete line 7-36;

3) Uncomment line 612 and comment/delete line 613

4) (optional) Delete line 565-610, delete line 630-656. You can keep these lines if you want to choose download file by visible interface like this (https://cds.climate.copernicus.eu/cdsapp#!/software/app-c3s-daily-era5-statistics?tab=app) does.

5) (optional) If you keep lines in step 4), then skip step 5). Otherwise state variables for your download file in line 660-668. Or delete these lines, but state variables inside the function after the bracket.

Other (might be easier) ways to use the cdstoolbox to obtain daily data see the reply of One W , or see the C3S Climate Data Store Toolbox User Forum with keyword 'daily'.

12

Hi ...

I'm having the same ....   "NameError: name 'reduce' is not defined", ... somebody knows who to solve it ?? 

I added the at the top .. "from functools import reduce", but the error continues ... 


Any HELP please, .... Thanks in advance. 


I want to calculate daily precipitation ... below the code.

from functools import reduce
import cdsapi
c = cdsapi.Client()
MONTHS = [
 "01", "02", "03", "04", "05", "06",
 "07", "08", "09", "10", "11", "12"
]
for month in MONTHS:

 result = c.service(
 "tool.toolbox.orchestrator.workflow",
 params={
 "realm": "c3s",
"project": "app-c3s-daily-era5-statistics",
 "version": "master",
 "kwargs": {
 "dataset": "reanalysis-era5-single-levels",
 "product_type": "reanalysis",
 "variable": "total_precipitation",
 "pressure_level": "-",
 "statistic": "daily_mean",
 "year": "2020",
 "month": month,
 "time_zone": "UTC+00:0",
 "frequency": "1-hourly",
 "grid": "0.25/0.25",
 "area": {"lat": [-20, -10], "lon": [40, 30]}
 },
 "workflow_name": "application"
 })
 c.download(result)



13

Hi Luis,

Please see the posting comment on 29th March, 2023 by Kevin Marsh (Retrieve daily ERA5/ERA5-Land data using the CDS API - Copernicus User Support Forum - ECMWF Confluence Wiki).

I succeeded two months ago by simply modifying the line "realm": "c3s" to "realm": "user-apps".