| title |
|---|
Preparations for participating in the tutorial |
This page includes some information on how to prepare for participating in this tutorial.
The prerequisites for the tutorial are listed on the [tutorials home page]({{ page.root}}[% index.md %}) and are also eproduced here:
Basic understanding of git (optional)
Basic understanding of your preferred command line interface (ie a bash terminal)
Access to CMIP data
Access to a suitable computing system (eg Jasmin, DLR machine)
Github account (optional) {: .prereq}
We typically use the command line to interact with ESMValTool. While most of us are likely to have experience with the command line, novices may want to work through this software carpentry unix shell course.
- Command line: https://swcarpentry.github.io/shell-novice/
Git is a distributed version-control system for tracking changes in source code during software development. It’s how we distribute, share, and manage the ESMValTool code.
To complete this tutorial and use ESMValTool, you will need access to data in a reasonable format. Some data will be provided, but there is simply too much data available for your tutors to make it all available directly.
ESMValTool may be run on multiple platforms, from your local machine to large computing clusters. The best option is to use a computing cluster with an Earth System Grid Federation (ESGF) node. The benefit of using a compute cluster with an ESGF node is that the Coupled Model Intercomparison Project (CMIP) is locally stored on disk and accessible directly by the tool. Similarly, observational data would also be available at these sites.
Here are a few options for compute clusters with ESGF nodes:
- CEDA-Jasmin (UK):
- DKRZ (Germany):
- ETHZ (Switzerland):
A full list of all ESGF nodes is available here.
If you're running on a computing cluster without an ESGF node, such as your local machine, or your institue machine, you will most likely have to make a local copy of the data that you need.
If neccesairy, data can be downloaded using the synda tool.
If you do not already have an account on JASMIN, then request an account as soon as possible. Please follow these instructions on how to create a Jasmin account
During the account creation, you will need an SSH key, which can be generated following these instructions
Here are some more instructions on how to get started with jasmin.
Also note that if you are working from home, JASMIN may not be directly accessible from your home. You may need to use ssh to connect to a machine in your institute and then on to JASMIN. Please test your connection before the tutorial.
Note that you have only created an account for the web-interface. To log into the jasmin machine and do work, you'll need to create a login account too using this page.
Please request access to the working groups:
Once you have access to the data archive on CEDA, make sure to link your CEDA and JASMIN accounts. This can be done by checking the link to CEDA box on your JASMIN profile page.
The linking may take a few hours to take effect and is necessary for you to access the BADC archives via JASMIN. Some CMIP5 data sets such as MIROC are not accessible by default and special permission has to be requested to access them via the CEDA catalogue page.
Log into jasmin-login:
ssh -X JASMIN-USERNAME@jasmin-login1.ceda.ac.uk
{: .language-bash}
Then log into the sci1 machine:
ssh -X jasmin-sci1
{: .language-bash}
Can you see the following locations:
ls /group_workspaces/jasmin4/esmeval/obsdata/Tier2
ls /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES
ls /badc/cmip6/data/CMIP6/CMIP/*/*/historical/r1i1p1f?/Omon/[ts]os/gn/latest/*.nc
{: .language-bash}
Note that the JASMIN is only open to certain locations (mostly universities, and research centres). You may need a VPN if you wish to connect from your home network.
Please request access to the working groups:
Once you have access to the data archive on CEDA, make sure to link your CEDA and JASMIN accounts. This can be done by checking the link to CEDA box on your JASMIN profile page.
The linking may take a few hours to take effect and is necessary for you to access the BADC archives via JASMIN. Some CMIP5 data sets such as MIROC are not accessible by default and special permission has to be requested to access them via the CEDA catalogue page.
Log into jasmin-login:
ssh -X JASMIN-USERNAME@jasmin-login1.ceda.ac.uk
{: .language-bash}
Then log into the sci1 machine:
ssh -X jasmin-sci1
{: .language-bash}
Can you see the following locations:
ls /group_workspaces/jasmin4/esmeval/obsdata/Tier2
ls /badc/cmip5/data/cmip5/output1/MOHC/HadGEM2-ES
ls /badc/cmip6/data/CMIP6/CMIP/*/*/historical/r1i1p1f?/Omon/[ts]os/gn/latest/*.nc
{: .language-bash}
Note that the JASMIN is only open to certain locations (mostly universities, and research centres). You may need a VPN if you wish to connect from your home network.
If you do not already have an account at the DKRZ, then register as soon as possible. You could find a short introduction on how to get started at DKRZ here.
There is also a user manual for Mistral which is DKRZ's current supercomputer.
To use the resources on DKRZ you have to join a project. One option is to join an existing project by logging into https://luv.dkrz.de/ with your account and select 'Join existing project'. Once you are accepted by the manager of your chosen project, your web account will be turned into a full LDAP account which will allow you to log into and use the DKRZ's resources. If you do not have access to an existing project, another option for you would be to apply for resources at DKRZ. Here are some instructions on how to apply for resources.
CMIP5 and CMIP6 data are available in these directories:
- CMIP5: /mnt/lustre01/work/kd0956/CMIP5/data/cmip5/output1/
- CMIP6: /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/CMIP/
Log into Mistral (DKRZ)
ssh -X user-account@mistral.dkrz.de
{: .language-bash}
Login nodes are for compiling and job submission only! For all other tasks, use one of the pre- and post-processing nodes
ssh -X <user-account>@mistralpp.dkrz.de
(see also this)
Data storage:
- Personal data: home directory (24GiB)
- Project data: /work//
- Temporary data: scratch directory on /scratch/*/ is automatically deleted after 14 days (15TiB) (Please use this directory for all your testing! Do not use the work directory for tests.) (see also this)
Running batch jobs: Info and examples on SLURM job scheduling system at DKRZ can be found here.
FIXME
If you are planning on running ESMValTool on your own machine, please make sure that you are able to download CMIP data and that you have several GB of space available to install conda & ESMVAlTool, but also enough to make a copy of some data.
You will also need to able to use:
- git
- conda
- synda
ESMValTool does not directly support Windows, but successful usage has been reported through the Windows Subsystem for Linux(WSL), available in Windows 10.
You don’t need a github account to participate in the tutorial. However, if you want to raise an issue, contribute to the discussions, or share your code, please create a github account.
To learn how to use github, please have a look at this introduction to github.
You may hear a few of the following phrases during the tutorial. Don’t be alarmed, they will make sense eventually.
Issues are github’s ticketing system. They allow users and developers to discuss problems, identify bugs, or to make suggestions. Each issue is assigned a number and will have it’s own page on github.
Here’s an explanation of the github issues.
Raising an issue is the act of creating a new issue. If you are asked to raise an issue, please follow any instructions that you are given, and also make sure that you read the default issue text.
A github pull request is the act of requesting that a branch is merged with another branch.
This is an advanced feature of GitHub, and will generally be performed by the ESMValTool development team.
The python package manager Conda (anaconda or miniconda) needs to be installed on your system before the tutorial starts. In some cases, your system may have a central version installed already.
More details on this process are available in the [Installation episode]({{ page.root}}[% link _episodes/02-installation.md %}).
{% include links.md %}