Hello, I need to download a netcdf file from the cds servers directly to an AWS S3 storage location.
To do that I would usually have the following code:
import fsspec
import cdsapi
ct = cdsapi.Client(quiet=True)
file_to_download_path="simplecache::s3://"+aws_bucket_name+'/'+global_climate_monthly_aws_s3_prefix_nc + 'lat%s_%s_%s.nc' % (lat, year, m)
with fsspec.open(file_to_download_path, mode='wb') as f:
----do something that writes to file f
when I try to use the following command:
ct.retrieve(dataset, kwargs, f) it doesn't work because ct.retrieve expects a file name, not a file Instance. I need to open my file with fsspec.open otherwise it doesn't work to write directly to the AWS S3 cloud.
Any idea how I can have the ct.retrieve command write directly to an AWS S3 file.
I am desperately trying to avoid having to write the file locally and then upload it.
Thank you