Intake-ESM Integration based on #1218

[charles-turner-1]

Description

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (69f2019) to head (d26a52b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2690      +/-   ##
==========================================
+ Coverage   95.70%   95.72%   +0.02%     
==========================================
  Files         267      268       +1     
  Lines       15768    15858      +90     
==========================================
+ Hits        15090    15180      +90     
  Misses        678      678              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bouweandela]

Great to see progress on this @charles-turner-1!

[bouweandela]

Is there any documentation on where to find all potential facets that ESMValCore might accept & what they represent?

ESMValCore is quite flexible with what facets it accepts. We have a translation between some of 'our' facets and the official ones in the esmvalcore.esgf.facets module (this is the subset that we use to search for files on ESGF). A few facets are used by ESMValCore for specific purposes such as CMOR checks and fixes (off the top of my head that would be dataset, project, mip, short_name), but others are entirely free-form and only used for finding input files and defining the output file names using the paths described in the config-developer.yml file.

Our intake catalogs here on Gadi have a bunch of extra keys (facets) that I haven't mapped.

If these are completely determined by the other facets, you can add them automatically using the extra facets facility

[bouweandela]

Structure: I've put this in an intake submodule,

How about adding a new module called e.g. esmvalcore.data or esmvalcore.data_sources or something similar and adding it as a submodule there? We could also move the esmvalcore.local and esmvalcore.esgf modules there (does not have to be in this pull request). I foresee us adding multiple input data sources in the near future.

[charles-turner-1]

Thanks for the review Bouwe, super helpful! I've only had a skim so far, but I'll get those suggestions incorporated next week

[bouweandela]

I started working on adding some interface code that could be useful here too in #2765.

[charles-turner-1]

Cheers, I'll take a look when I get the chance! Gonna talk to Martin Durant (author of Intake) in ~10 days so hopefully this PR should pick up stone steam after then, I'll be working on this stuff more actively.

[charles-turner-1]

• Seems like the ordering of IntakeEsmDataSource.find_data is not stable, tests failing in CI (different outcome between runs...) are passing locally. (I'm working on this right now).
• I've had to modify the .gitignore to include the netCDF files in the intake-esm sample datastore. We can happily just move the sample data out to an object storage service/use a dataset already there if we like, which feels a bit cleaner? That or use a zarr formatted datastore potentially.

[charles-turner-1]

Few lines of coverage to fix, but I think this is mostly ready for review now!

[bouweandela]

Great to see this progressing @charles-turner-1! Would it be possible to add some configuration files for using this, e.g. with the catalogs at your own institute, Pangeo or DKRZ cloud storage?

[bouweandela]

I came across a CORDEX catalog at https://euro-cordex.s3.eu-central-1.amazonaws.com/catalog/CORDEX-CMIP5.json too: https://py-cordex.readthedocs.io/en/latest/notebooks/prudence.html#Investigate-regional-warming-in-Prudence-regions

[bouweandela]

Looks like you're making good progress here. Are you still planning to work on this more or do you need another review?

We can happily just move the sample data out to an object storage service/use a dataset already there if we like, which feels a bit cleaner? That or use a zarr formatted datastore potentially.

These NetCDF files are a bit big to add to the repository indeed. If you want to use NetCDF files that are available locally, you could build a catalog from https://github.com/ESMValGroup/ESMValTool_sample_data/, which is installed when running the tests.

It would be rather nice to also have a test with the Pangeo data stored in the cloud at Amazon or Google. Or would that be too slow because it's such a large dataset?

[charles-turner-1]

It shouldn't be too difficult to point this at one of the catalog's that we use in the intake-esm tests. I think they're relatively small.

I started doing some digging into the configs but got sidelined with something else - are there docs for how we want to configure this? Per system and/or use cloud catalogs?

I'll try to get this as close to wrapped up as I can over the rest of this week and/or next, it's dragged on far too long.