This book serves as documentation for the PEcAn Project. It contains descriptions of topics necessary to inform a beginner and advanced user as well as requisite materials for developers. It does not contain low-level descriptions of functions within PEcAn. Our aim for this documentation is to educate you about the PEcAn software, the possibilities of its usage, and the standards,expectations, and core workflows for developers.
-
This book is organized four main topics:
-
Introduction - Brief explanation of PEcAn, how to obtain the PEcAn VM, and explanation of basic web interface functions.
-
Tutorials/Demos/Workflows - All User and Developer tutorials/demos/workflows to explain how to use and add to PEcAn in different ways.
-
Topical Pages - Explanation of main PEcAn components and how they fit together.
-
Appendix - External documentation and sources of information and a FAQ section.
-
-
3.1 General Feedback/Comments/Suggestions
-
We want your ideas, thoughts, comments, and suggestions! As a community we are committed to creating an inclusive and supportive atmosphere so we all to reach out to us in the following ways:
-
Github:https://github.com/PecanProject/pecan
-This is the main hub of communication surrounding PEcAn development. Check out the issues section to see known bugs, upcoming features, and ideas for future development. Feel free to comment on existing issues or open new ones with questions, bug reports, feature requests, and/or ideas.
-
Slack:https://pecanproject.slack.com/
-Slack serves as our day to day mode of communication. To join us in slack you will need to create an account first. This is done in 3 steps:
-
-
Request an inivitation to join Slack, this will be send by email to address you provided.
-
Check your inbox for an email from Slack with subject “Rob Kooper has invited you to join a Slack workspace”. This email should have a link that you can click to join slack.
-
When you click a webpage will open up that asks you to create an account, once that is done you can login into the slack chatrooms.
-
-
Email: pecanproj[at]gmail.com
-If you do not wish your communication with the team to be public, send us an email at the address above and we will get back to you as soon as possible.
-
-
-
3.2 Editing this book
-
The file organization of this documentation can be described simply as follows:
-
-
Each chapter is in its own file (within the corresponding section).
-
Each group of chapters (i.e. “part” in LaTeX) is in its own directory.
-
-
Sections and chapters are rendered (and numbered) in alpha-numerical order of their corresponding file names.
-Therefore, each section directory and chapter file name should be prefixed with a two-digit (zero-padded) number.
-File and directory names should be as similar as possible to the name of the corresponding chapter or section.
-For instance, the file name for this chapter’s source file is 06_reference/10_editing_this_book.Rmd.
-This numbering means that if you need to create an additional chapter before an existing one, you will have to renumber all chapters following it.
-
To ensure correct rendering, you should also make sure that each chapter starts with a level 1 heading (# heading).
-For instance, this chapter’s source starts with:
Furthermore, to keep the organization consistent, each chapter should have exactly one level 1 heading (i.e. do not combine multiple chapters into a single file).
-In other words, do not spread a single chapter across multiple files, and do not put multiple chapters in the same file.
-
Each section directory has a file starting with 00 that contains only the section (or “Part”) title.
-This is used to create the greyed-out section headers in the rendered HTML.
-For instance, this section has a file called 00_introduction.Rmd which contains only the following:
To cross-reference a different section, use that section’s unique tag (starts with #; appears next to the section heading surrounded in curly braces).
-For instance, the following Markdown contains two sections that cross-reference each other:
If no header tag exists for a section you want to cross-reference, you should create one.
-We have no strict rules about this, but it’s useful to have tags that give some sense of their parent hierarchy and reference their parent sections (e.g. #models, #models-ed, and #models-ed-xml to refer to a chapter on models, with a subsection on ED and a sub-subsection on ED XML configuration).
-If section organization changes, it is fine to move header tags, but avoid altering existing tags as this will break all the links pointing to that tag.
-(Note that it is also possible to link to section headings by their exact title. However, this is not recommended because these section titles could change, which would break the links.)
-
When referring to PEcAn packages or specific functions, it is a good idea to link to the rendered package documentation.
-For instance, here are links to the models/ed package, the PEcAn.ED2::modify_ed2in function, and the PEcAnRTM package vignette.
-If necessary, you can also link directly to specific lines or blocks in the source code on GitHub, like this.
-(To get a link to a line, click its line number. To then select a block, shift-click another line number.)
-
To insert figures, use knitr::include_graphics("path/to/figure.png") inside an R code chunk.
-For example:
Note that image file names are relative to the book_source directory, NOT to the markdown file.
-In other words, if myimage.png was in the same directory as this file, I would still have to reference it as 06_reference/myimage.png – I could not just do myimage.png.
-The size, caption, and other properties of the rendered image can be controlled via chunk options.
-
For additional information about how bookdown works (including information about its syntax), see the Bookdown free online book.
-
-
-
3.3 How to create your own version of Documentation
-
To create your own version of documentation you’ll need to follow these steps:
-These procedures assume you have an github account, you forked pecan, you have cloned pecan locally, and have a TRAVIS account.
-1. Create a repository under your github account with the name “pecan-documentation”. Clear it of any files. Set up the repository with Github Pages by going to the settings tab for that repository.
-2. Create a personal access token for github: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line and copy it.
-3. Create a TRAVIS environment variable called GITHUB_PAT and save the access token you made as a secret variable.
-4. Create a branch from your local pecan repository with a name that starts with release/. (ie. Release/vtonydoc)
-5. Make whichever changes you would like to the documentation and push it up to your fork.
-
From here TRAVIS will build your documentation. The web version of your documentation will be rendered with the url following the structure: username.github.io/pecan-documentation/pattern_after_release/
Writing the interface modules between the model and PEcAn
-
-
Note that coupling a model to PEcAn should not require any changes to the model code itself. A key aspect of our design philosophy is that we want it to be easy to add models to the system and we want to using the working version of the code that is used by all other model users, not a special branch (which would rapidly end up out-of-date).
-
-
19.1.1 Using PEcAn Database
-
To run a model within PEcAn requires that the PEcAn database has sufficient information about the model. This includes a MODEL_TYPE designation, the types of inputs the model requires, the location of the model executable, and the plant functional types used by the model.
-
The instructions in this section assume that you will be specifying this information using the BETYdb web-based interface. This can be done either on your local VM (localhost:3280/bety or localhost:6480/bety) or on a server installation of BETYdb. However you interact with BETYdb, we encourage you to set up your PEcAn instance to support database syncs so that these changes can be shared and backed-up across the PEcAn network.
-
-
The figure below summarizes the relevant database tables that need to be updated to add a new model and the primary variables that define each table.
-
-
-
-
19.1.2 Define MODEL_TYPE
-
The first step to adding a model is to create a new MODEL_TYPE, which defines the abstract model class. This MODEL_TYPE is used to specify input requirements, define plant functional types, and keep track of different model versions.
-
The MODEL_TYPE is created by selecting Runs > Model Type and then clicking on New Model Type. The MODEL_TYPE name should be identical to the MODEL package name (see Interface Module below) and is case sensitive.
-
-
-
-
-
19.1.3 MACHINE
-
The PEcAn design acknowledges that the same model executables and input files may exist on multiple computers. Therefore, we need to define the machine that that we are using. If you are running on the VM then the local machine is already defined as pecan. Otherwise, you will need to select Runs > Machines, click New Machine, and enter the URL of your server (e.g. pecan2.bu.edu).
-
-
-
19.1.4 MODEL
-
Next we are going to tell PEcAn where the model executable is. Select Runs > Files, and click ADD. Use the pull down menu to specify the machine you just defined above and fill in the path and name for the executable. For example, if SIPNET is installed at /usr/local/bin/sipnet then the path is /usr/local/bin/ and the file (executable) is sipnet.
-
Now we will create the model record and associate this with the File we just registered. The first time you do this select Runs > Models and click New Model. Specify a descriptive name of the model (which doesn’t have to be the same as MODEL_TYPE), select the MODEL_TYPE from the pull down, and provide a revision identifier for the model (e.g. v3.2.1). Once the record is created select it from the Models table and click EDIT RECORD. Click on “View Related Files” and when the search window appears search for the model executable you just added (if you are unsure which file to choose you can go back to the Files menu and look up the unique ID number). You can then associate this Model record with the File by clicking on the +/- symbol. By contrast, clicking on the name itself will take you to the File record.
-
In the future, if you set up the SAME MODEL VERSION on a different computer you can add that Machine and File to PEcAn and then associate this new File with this same Model record. A single version of a model should only be entered into PEcAn once.
-
If a new version of the model is developed that is derived from the current version you should add this as a new Model record but with the same MODEL_TYPE as the original. Furthermore, you should set the previous version of the model as Parent of this new version.
-
-
-
19.1.5 FORMATS
-
The PEcAn database keep track of all the input files passed to models, as well as any data used in model validation or data assimilation. Before we start to register these files with PEcAn we need to define the format these files will be in. To create a new format see Formats Documentation.
-
-
-
19.1.6 MODEL_TYPE -> Formats
-
For each of the input formats you specify for your model, you will need to edit your MODEL_TYPE record to add an association between the format and the MODEL_TYPE. Go to Runs > Model Type, select your record and click on the Edit button. Next, click on “Edit Associated Formats” and choose the Format you just defined from the pull down menu. If the Input box is checked then all matching Input records will be displayed in the PEcAn site run selection page when you are defining a model run. In other words, the set of model inputs available through the PEcAn web interface is model-specific and dynamically generated from the associations between MODEL_TYPEs and Formats. If you also check the Required box, then the Input will be treated as required and PEcAn will not run the model if that input is not available. Furthermore, on the site selection webpage, PEcAn will filter the available sites and only display pins on the Google Map for sites that have a full set of required inputs (or where those inputs could be generated using PEcAn’s workflows). Similarly, to make a site appear on the Google Map, all you need to do is specify Inputs, as described in the next section, and the point should automatically appear on the map.
-
-
-
19.1.7 INPUTS
-
After a file Format has been created then input files can be registered with the database. Creating Inputs can be found under How to insert new Input data.
-
-
-
19.1.8 Add Plant Functional Types (PFTs)
-
Since many of the PEcAn tools are designed to keep track of parameter uncertainties and assimilate data into models, to use PEcAn with a model it is important to define Plant Functional Types for the sites or regions that you will be running the model.
-
Create a new PFT entry by selecting Data > PFTs and then clicking on New PFT.
-
-
-
Give the PFT a descriptive name (e.g., temperate deciduous). PFTs are MODEL_TYPE specific, so choose your MODEL_TYPE from the pull down menu.
-
-
-
19.1.8.1 Species
-
Within PEcAn there are no predefined PFTs and user can create new PFTs very easily at whatever taxonomic level is most appropriate, from PFTs for individual species up to one PFT for all plants globally. To allow PEcAn to query its trait database for information about a PFT, you will want to associate species with the PFT record by choosing Edit and then “View Related Species”. Species can be searched for by common or scientific name and then added to a PFT using the +/- button.
-
-
-
19.1.8.2 Cultivars
-
You can also define PFTs whose members are cultivars instead of species. This is designed for analyses where you want to want to perform meta-analysis on within-species comparisons (e.g. cultivar evaluation in an agricultural model) but may be useful for other cases when you want to specify different priors for some member of a species. You cannot associate both species and cultivars with the same PFT, but the cultivars in a cultivar PFT may come from different species, potentially including all known cultivars from some of the species, if you wish to and have thought about how to interpret the results.
-
It is not yet possible to add a cultivar PFT through the BETYdb web interface. See this GithHub comment for an example of how to define one manually in PostgreSQL.
-
-
-
-
19.1.9 Adding Priors for Each Variable
-
In addition to adding species, a PFT is defined in PEcAn by the list of variables associated with the PFT. PEcAn takes a fundamentally Bayesian approach to representing model parameters, so variables are not entered as fixed constants but as prior probability distributions.
-
There are a wide variety of priors already defined in the PEcAn database that often range from very diffuse and generic to very informative priors for specific PFTs.
-
These pre-existing prior distributions can be added to a PFT. Navigate to the PFT from Data > PFTs and selecting the edit button in the Actions column for the chosen PFT.
-
-
Click on “View Related Priors” button and search through the list for desired prior distributions. The list can be filtered by adding terms into the search box. Add a prior to the PFT by clicking on the far left button for the desired prior, changing it to an X.
-
-
Save this by scrolling to the bottom of the PFT page and hitting the Update button.
-
-
-
19.1.9.1 Creating new prior distributions
-
A new prior distribution can be created for a pre-existing variable, if a more constrained or specific one is known.
-
-
Select Data > Priors then “New Prior”
-
In the Citation box, type in or select an existing reference that indicates how the prior was defined. There are a number of unpublished citations in current use that simply state the expert opinion of an individual
-
Fill the Variable box by typing in part or all of a pre-existing variable’s name and selecting it
-
The Phylogeny box allows one to specify what taxonomic grouping the prior is defined for, at it is important to note that this is just for reference and doesn’t have to be specified in any standard way nor does it have to be monophyletic (i.e. it can be a functional grouping)
-
The prior distribution is defined by choosing an option from the drop-down Distribution box, and then specifying values for both Parameter a and Parameter b. The exact meaning of the two parameters depends on the distribution chosen. For example, for the Normal distribution a and b are the mean and standard deviation while for the Uniform they are the minimum and maximum. All parameters are defined based on their standard parameterization in the R language
-
Specify the prior sample size in N if the prior is based on observed data (independent of data in the PEcAn database)
-
When this is done, scroll down and hit the Create button
-
-
-
The new prior distribution can then be added a PFT as described in the “Adding Priors for Each Variable” section.
-
-
-
19.1.9.2 Creating new variables
-
It is important to note that the priors are defined for the variable name and units as specified in the Variables table. If the variable name or units is different within the model it is the responsibility of write.configs.MODEL function to handle name and unit conversions (see Interface Modules below). This can also include common but nonlinear transformations, such as converting SLA to LMA or changing the reference temperature for respiration rates.
-
To add a new variable, select Data > Variables and click the New Variable button. Fill in the Name field with the desired name for the variable and the units in the Units field. There are additional fields, such as Standard Units, Notes, and Description, that can be filled out if desired. When done, hit the Create button.
-
-
The new variable can be used to create a prior distribution for it as in the “Creating new prior distributions” section.
-
-
-
-
19.1.10 Interface Modules
-
-
19.1.10.1 Setting up the module directory (required)
-
PEcAn assumes that the interface modules are available as an R package in the models directory named after the model in question. The simplest way to get started on that R package is to make a copy the template directory in the pecan/models folder and re-name it to the name of your model. In the code, filenames, and examples below you will want to substitute the word MODEL for the name of your model (note: R is case-sensitive).
-
If you do not want to write the interface modules in R then it is fairly simple to set up the R functions describe below to just call the script you want to run using R’s system command. Scripts that are not R functions should be placed in the inst folder and R can look up the location of these files using the function system.file which takes as arguments the local path of the file within the package folder and the name of the package (typically PEcAn.MODEL). For example
Within the module folder open the DESCRIPTION file and change the package name to PEcAn.MODEL. Fill out other fields such as Title, Author, Maintainer, and Date.
-
-
-
19.2.0.2 NAMESPACE
-
Open the NAMESPACE file and change all instances of MODEL to the name of your model. If you are not going to implement one of the optional modules (described below) at this time then you will want to comment those out using the pound sign #. For a complete description of R NAMESPACE files see here. If you create additional functions in your R package that you want to be used make sure you include them in the NAMESPACE as well (internal functions don’t need to be declared)
-
-
-
19.2.0.3 Building the package
-
Once the package is defined you will then need to add it to the PEcAn build scripts. From the root of the pecan directory, go into the scripts folder and open the file build.sh. Within the section of code that includes PACKAGES= add model/MODEL to the list of packages to compile. If, in writing your module, you add any other R packages to the system you will want to make sure those are listed in the DESCRIPTION and in the script scripts/install.dependencies.R. Next, from the root pecan directory open all/DESCRIPTION and add your model package to the Suggests: list.
-
At any point, if you want to check if PEcAn can build your MODEL package successfully, just go to the linux command prompt and run scripts/build.sh. You will need to do this before the system can use these packages.
-
-
-
19.2.0.4 write.config.MODEL (required)
-
This module performs two primary tasks. The first is to take the list of parameter values and model input files that it receives as inputs and write those out in whatever format(s) the MODEL reads (e.g. a settings file). The second is to write out a shell script, jobs.sh, which, when run, will start your model run and convert its output to the PEcAn standard (netCDF with metadata currently equivalent to the MsTMIP standard). Within the MODEL directory take a close look at inst/template.job and the example write.config.MODEL to see an example of how this is done. It is important that this script writes or moves outputs to the correct location so that PEcAn can find them. The example function also shows an example of writing a model-specific settings/config file, also by using a template.
-
You are encouraged to read the section above on defining PFTs before writing write.config.MODEL so that you understand what model parameters PEcAn will be passing you, how they will be named, and what units they will be in. Also note that the (optional) PEcAn input/driver processing scripts are called by separate workflows, so the paths to any required inputs (e.g. meteorology) will already be in the model-specific format by the time write.config.MODEL receives that info.
-
-
-
19.2.0.5 Output Conversions
-
The module model2netcdf.MODEL converts model output into the PEcAn standard (netCDF with metadata currently equivalent to the MsTMIP standard). This function was previously required, but now that the conversion is called within jobs.sh it may be easier for you to convert outputs using other approaches (or to just directly write outputs in the standard).
-
Whether you implement this function or convert outputs some other way, please note that PEcAn expects all outputs to be broken up into ANNUAL files with the year number as the file name (i.e. YEAR.nc), though these files may contain any number of scalars, vectors, matrices, or arrays of model outputs, such as time-series of each output variable at the model’s native timestep.
-
Note: PEcAn reads all variable names from the files themselves so it is possible to add additional variables that are not part of the MsTMIP standard. Similarly, there are no REQUIRED output variables, though time is highly encouraged. We are shortly going establish a canonical list of PEcAn variables so that if users add additional output variables they become part of the standard. We don’t want two different models to call the same output with two different names or different units as this would prohibit the multi-model syntheses and comparisons that PEcAn is designed to facilitate.
Converts meteorology input files from the PEcAn standard (netCDF, CF metadata) to the format required by the model. This file is optional if you want to load all of your met files into the Inputs table as described in How to insert new Input data, which is often the easiest way to get up and running quickly. However, this function is required if you want to benefit from PEcAn’s meteorology workflows and model run cloning. You’ll want to take a close look at [Adding-an-Input-Converter] to see the exact variable names and units that PEcAn will be providing. Also note that PEcAn splits all meteorology up into ANNUAL files, with the year number explicitly included in the file name, and thus what PEcAn will actually be providing is in.path, the input path to the folder where multiple met files may stored, and in.prefix, the start of the filename that precedes the year (i.e. an individual file will be named <in.prefix>.YEAR.nc). It is valid for in.prefix to be blank. The additional REQUIRED arguments to met2model.MODEL are outfolder, the output folder where PEcAn wants you to write your meteorology, and start_date and end_date, the time range the user has asked the meteorology to be processed for.
-
-
-
19.2.0.7 Commit changes
-
Once the MODEL modules are written, you should follow the Using-Git instructions on how to commit your changes to your local git repository, verify that PEcAn compiles using scripts/build.sh, push these changes to Github, and submit a pull request so that your model module is added to the PEcAn system. It is important to note that while we encourage users to make their models open, adding the PEcAn interface module to the Github repository in no way requires that the model code itself be made public. It does, however, allow anyone who already has a copy of the model code to use PEcAn so we strongly encourage that any new model modules be committed to Github.
-
-
-
-
19.3 Adding input data
-
-
19.3.1 Input records in BETY
-
All model input data or data used for model calibration/validation must be registered in the BETY database.
-
Before creating a new Input record, you must make sure that the format type of your data is registered in the database. If you need to make a new format record, see Creating a new format record in BETY.
-
-
-
19.3.2 Create a database file record for the input data
-
An input record contains all the metadata required to identify the data, however, this record does not include the location of the data file. Since the same data may be stored in multiple places, every file has its own dbfile record.
-
From your BETY interface:
-
-
Create a DBFILES entry for the path to the file
-
-
From the menu click RUNS then FILES
-
Click “New File”
-
Select the machine your file is located at
-
Fill in the File Path where your file is located (aka folder or directory) NOT including the name of the file itself
-
Fill in the File Name with the name of the file itself. Note that some types of input records will refer to be ALL the files in a directory and thus File Name can be blank
-
Click Update
-
-
-
-
-
19.3.3 Creating a new Input record in BETY
-
From your BETY interface:
-
-
Create an INPUT entry for your data
-
-
From the menu click RUNS then INPUTS
-
Click “New Input”
-
Select the SITE that this data is associated with the input data set
-
Other required fields are a unique name for the input, the start and end dates of the data set, and the format of the data. If the data is not in a currently known format you will need to create a NEW FORMAT and possibly a new input converter. Instructions on how to do add a converter can be found here Input conversion. Instructions on how to add a format record can be found here
-
Parent ID is an optional variable to indicated that one dataset was derived from another.
-
Click “Create”
-
-
Associate the DBFILE with the INPUT
-
-
In the RUNS -> INPUTS table, search and find the input record you just created
-
Click on the EDIT icon
-
Select “View related Files”
-
In the Search window, search for the DBFILE you just created
-
-
Once you have found the DBFILE, click on the “+” icon to add the file
-
Click on “Update” at the bottom when you are done.
-
-
-
-
19.3.4 Adding a new input converter
-
Three Types of data conversions are discussed below: Meteorological data, Vegetation data, and Soil data. Each section provides instructions on how to convert data from their raw formats into a PEcAn standard format, whether it be from a database or if you have raw data in hand.
19.3.4.1.1 Adding a function to PEcAn to convert a met data source
-
In general, you will need to write a function to download the raw met data and one to convert it to the PEcAn standard.
-
Downloading raw data function are named download.<source>.R. These functions are stored within the PEcAn directory: /modules/data.atmosphere/R.
-
Conversion function from raw to standard are named met2CF.<source>.R. These functions are stored within the PEcAn directory: /modules/data.atmosphere/R.
Note: Unless you are also adding a new model, you will not need to write a script to convert from PEcAn standard to PEcAn models. Those conversion scripts are written when a model is added and can be found within each model’s PEcAn directory.
-
Standards dimesion, names, nad units can be found here:Input Standards
-
-
-
19.3.4.1.2 Adding Single-Site Specific Meteorological Data
-
Perhaps you have meteorological data specific to one site, with a unique format that you would like to add to PEcAn. Your steps would be to:
-1. write a script or function to convert your files into the netcdf PEcAn standard
-2. insert that file as an input record for your site following these instructions
-
-
-
19.3.4.1.3 Processing Met data outside of the workflow using PEcAn functions
-
Perhaps you would like to obtain data from one of the sources coupled to PEcAn on its own. To do so you can run PEcAn functions on their own.
-
-
19.3.4.1.3.1 Example 1: Processing data from a database
-
Download Amerifluxlbl from Niwot Ridge for the year 2004:
Using the information returned as the object raw.file you will then convert the raw files into a standard file.
-
Open a connection with BETY. You may need to change the host name depending on what machine you are hosting BETY. You can find the hostname listed in the machines table of BETY.
19.3.4.1.3.2 Example 2: Processing data from data already in hand
-
If you have Met data already in hand and you would like to convert into the PEcAn standard follow these instructions.
-
Update BETY with file record, format record and input record according to this page How to Insert new Input Data
-
If your data is in a csv format you can use the met2CF.csvfunction to convert your data into a PEcAn standard file.
-
Open a connection with BETY. You may need to change the host name depending on what machine you are hosting BETY. You can find the hostname listed in the machines table of BETY.
Prepare the arguments you need to execute the met2CF.csv function
-
in.path <- 'path/where/the/raw/file/lives'
-in.prefix <- 'prefix_of_the_raw_file'
-outfolder <- 'path/to/where/you/want/to/output/thecsv/'
-format.id <- formatid of the format your created
-format <- PEcAn.DB::query.format.vars(format.id=format.id,bety = bety)
-lon <- longitude of your site
-lat <- latitude of your site
-format$time_zone <- time zone of your site
-start_date <- Start date of your data in "y-m-d"
-end_date <- End date of your data in "y-m-d"
Vegetation data will be required to parameterize your model. In these examples we will go over how to produce a standard initial condition file.
-
The main function to process cohort data is the ic_process.R function. As of now however, if you require pool data you will run a separate function, pool_ic_list2netcdf.R.
-
-
19.3.4.2.0.1 Example 1: Processing Veg data from data in hand.
-
In the following example we will process vegetation data that you have in hand using PEcAn.
-
First, you’ll need to create a input record in BETY that will have a file record and format record reflecting the location and format of your file. Instructions can be found in our How to Insert new Input Data page.
-
Once you have created an input record you must take note of the input id of your record. An easy way to take note of this is in the URL of the BETY webpage that shows your input record. In this example we use an input record with the id 1000013064 which can be found at this url: https://psql-pecan.bu.edu/bety/inputs/1000013064# . Note that this is the Boston University BETY database. If you are on a different machine, your url will be different.
-
With the input id in hand you can now edit a pecan XML so that the PEcAn function ic_process will know where to look in order to process your data. The inputs section of your pecan XML will look like this. As of now ic_process is set up to work with the ED2 model so we will use ED2 settings and then grab the intermediary Rds data file that is created as the standard PEcAn file. For your Inputs section you will need to input your input id wherever you see the useic flag.
This IC workflow also supports generating ensembles of initial conditions from posterior estimates of DBH. To do this the tags below can be inserted to the pecan.xml:
Here the id should point to a file that has MCMC samples to generate the ensemble from. The number between the <ensemble> tag defines the number of ensembles requested. The workflow will populate the settings list run$inputs tag with ensemble member information. E.g.:
Note that the argument dir is set to the current directory. You will find the final ED2 file there. More importantly though you will find the .Rds file within the same directory.
-
-
-
19.3.4.2.0.2 Example 3 Pool Initial Condition files
-
If you have pool vegetation data, you’ll need the pool_ic_list2netcdf.R function to convert the pool data into PEcAn
-standard.
-
The function stands alone and requires that you provide a named list of netcdf dimensions and values, and a named list of variables and values. Names and units need to match the standard_vars.csv table found here.
-
#Create a list object with necessary dimensions for your site
-input<-list()
-dims<- list(lat=-115,lon=45, time= 1)
-variables<- list(SoilResp=8,TotLivBiom=295)
-input$dims <- dims
-input$vals <- variables
-
Once this is done, set outdir to where you’d like the file to write out to and a siteid. Siteid in this can be used as an file name identifier. Once part of the automated workflow siteid will reflect the site id within the BET db.
At the moment this file would need to be inserted into Inputs manually. By default, this function also calls soil_params, which will estimate a number of hydraulic and thermal parameters from texture. Be aware that at the moment not all model couplers are yet set up to read this file and/or convert it to model-specific formats.
-
-
-
19.3.4.3.0.2 Example 2: Converting PalEON data
-
In addition to location-specific soil data, PEcAn can extract soil texture information from the PalEON regional soil product, which itself is a subset of the MsTMIP Unified North American Soil Map. If this product is installed on your machine, the appropriate step in the do_conversions workflow is enabled by adding the following tag under <inputs> in your pecan.xml
In the future we aim to extend this extraction to a wider range of soil products.
-
-
-
19.3.4.3.0.3 Example 3: Extracting soil properties from gSSURGO database
-
In addition to location-specific soil data, PEcAn can extract soil texture information from the gSSURGO data product. This product needs no installation and it extract soil proeprties for the lower 48 states in U.S. In order to let the pecan know that you’re planning to use gSSURGO, you can the following XML tag under input in your pecan xml file.
This tutorial explains the process of ingesting data into PEcAn via our Data-Ingest Application. In order to ingest data, the users must first select data that they wish to upload. Then, they enter metadata to help PEcAn parse and load the data into the main PEcAn workflow.
-
-
19.4.1 Loading Data
-
-
19.4.1.1 Selecting Ingest Method
-
The Data-Ingest application is capable of loading data from the DataONE data federation and from the user’s local machine. The first step in the workflow is therefore to select an upload method. The application defaults to uploading from DataONE. To upload data from a local device, simply select the radio button titled Local Files.
-
-
-
19.4.1.2 DataONE Upload Example
-
-
-
-The DataONE download feature allows the user to download data at a given doi or DataONE specific package id. To do so, enter the doi or identifier in the Import From DataONE field and select download. The download process may take a couple of minutes to run depending on the number of files in the dataONE package. This may be a convenient option if the user does not wish to download files directly to their local machine. Once the files have been successfully downloaded from DataONE, they are displayed in a table. Before proceeding to the next step, the user can select a file to ingest by clicking on the corresponding row in the data table.
-
-
-
-
-
19.4.2 Local Upload Example
-
-To upload local files, the user should first select the Local Files button. From there, the user can upload files from their local machines by selecting Browse or by dragging and dropping files into the text box. The files will begin uploading automatically. From there, the user should select a file to ingest and then select the Next Step button.
-
-After this step, the workflow is identical for both methods. However, please note that if it becomes necessary to switch from loading data via DataONE to uploading local files after the first step, please restart the application.
-
-
-
-
-### 2. Creating an Input Record
-Creating an input record requires some basic metadata about the file that is being ingested. Each entry field is briefly explained below.
-
-
-
Site: To link the selected file with a site, the user can scroll or type to search all the sites in PEcAn. See Example:
-
-
-
-
Parent: To link the selected file with another dataset, type to search existing datasets in the Parent field.
-
Name: this field should be autofilled by selecting a file in step 1.
-
Format: If the selected file has an existing format name, the user can search and select in the Format field. If the selected file’s format is not already in pecan, the user can create a new format by selecting Create New Format. Once this new format is created, it will automatically populate the Format box and the Current Mimetype box (See Section 3).
-
Mimetype: If the format already exists, select an existing mimetype.
-
Start and End Date and Time: Inputs can be entered manually or by using the user interface. See example
-
-
-
-
-
Notes: Describe the data that is being uploaded. Please include any citations or references.
-
-
-
-
19.4.3 3. Creating a format record
-
If it is necessary to add a new format to PEcAn, the user should fill out the form attached to the Create New Format button. The inputs to this form are described below:
-
-
Mimetype: type to search existing mimetypes. If the mimetype is not in that list, please click on the link Create New Mimetype and create a new mimetype via the BETY website.
-
New Format Name: Add the name of the new format. Please exclude spaces from the name. Instead please use underscores "_".
-
Header: If there is space before the first line of data in the dataset, please select Yes
-
Skip: The number of lines in the header that should be skipped before the data.
-
Please enter notes that describe the format.
-
-
Example:
-
-
-### 4. Formats_Variables Record
-The final step in the ingest process is to register a formats-variables record. This record links pecan variables with variables from the selected data.
-
-
Variable: PEcAn variable that is equivalent to variable in selected file.
-
Name: The variable name in the imported data need only be specified if it differs from the BETY variable name.
-
Unit: Should be in a format parseable by the udunits library and need only be secified if the units of the data in the file differ from the BETY standard.
-
Storage Type: Storage type need only be specified if the variable is stored in a format other than would be expected (e.g. if numeric values are stored as quoted character strings). Additionally, storage_type stores POSIX codes that are used to store any time variables (e.g. a column with a 4-digit year would be %Y).
-
Column Number: Vector of integers that list the column numbers associated with variables in a dataset. Required for text files that lack headers.
-
-
-
-
Finally, the path to the ingest data is displayed in the Select Files box.
-
-
-
-
19.5 Creating a new format
-
-
19.5.1 Formats in BETY
-
The PEcAn database keeps track of all the input files passed to models, as well as any data used in model validation or data assimilation. Before we start to register these files with PEcAn we need to define the format these files will be in.
-
The main goal is to take all the meta-data we have about a data file and create a record of it that pecan can use as a guide when parsing the data file.
-
This information is stored in a Format record in the bety database. Make sure to read through the current Formats before deciding to make a new one.
-
-
-
19.5.2 Creating a new format in BETY
-
If the Format you are looking for is not available, you will need to create a new record. Before entering information into the database, you need to be able to answer the following questions about your data:
-
-
What is the file MIME type?
-
-
We have a suit of functions for loading in data in open formats such as CSV, txt, netCDF, etc.
-
PEcAn has partnered with the NCSA BrownDog project to create a service that can read and convert as many data formats as possible. If your file type is less common or a proprietary type, you can use the BrownDog DAP to convert it to a format that can be used with PEcAn.
-
If BrownDog cannot convert your data, you will need to contact us about writing a data specific load function.
-
-
What variables does the file contain?
-
-
What are the variables named?
-
What are the variable units?
-
How do the variable names and units in the data map to PEcAn variables in the BETY database? See below for an example. It is most likely that you will NOT need to add variables to BETY. However, identifying the appropriate variables matches in the database may require some work. We are always available to help answer your questions.
-
-
Is there a timestamp on the data?
-
-
What are the units of time?
-
-
-
Here is an example using a fake dataset:
-
-
-
example_data
-
-
This data started out as an excel document, but was saved as a CSV file.
-
To create a Formats record for this data, in the web interface of BETY, select Runs > Formats and click New Format.
-
You will need to fill out the following fields:
-
-
MIME type: File type (you can search for other formats in the text field)
-
Name: The name of your format (this can be whatever you want)
-
Header: Boolean that denotes whether or not your data contains a header as the first line of the data. (1 = TRUE, 0 = FALSE)
-
Skip: The number of lines above the data that should be skipped. For example, metadata that should not be included when reading in the data or blank spaces.
-
Notes: Any additional information about the data such as sources and citations.
-
-
Here is the Formats record for the example data:
-
-
-
format_record_1
-
-
When you have finished this section, hit Create. The final record will be displayed on the screen.
-
-
19.5.2.1 Formats -> Variables
-
After a Format entry has been created, you are encouraged to edit the entry to add relationships between the file’s variables and the Variables table in PEcAn. Not only do these relationships provide meta-data describing the file format, but they also allow PEcAn to search and (for some MIME types) read files.
-
To enter this data, select Edit Record and on the edit screen select View Related Variable.
-
Here is the record for the example data after adding related variables:
-
-
-
format_record_2
-
-
-
19.5.2.1.1 Name and Unit
-
For each variable in the file you will want at a minimum to specify the NAME of the variable within your file and match that to the equivalent Variable in the pulldown.
-
Make sure to search for your variables under Data > Variables before suggesting that we create a new variable record. This may not always be a straightforward process.
-
For example bety contains a record for Net Primary Productivity:
-
-
-
var_record
-
-
This record does not have the same variable name or the same units as NPP in the example data.
-You may have to do some reading to confirm that they are the same variable.
-In this case
-- Both the data and the record are for Net Primary Productivity (the notes section provides additional resources for interpreting the variable.)
-- The units of the data can be converted to those of the vairiable record (this can be checked by running udunits2::ud.are.convertible("g C m-2 yr-1", "Mg C ha-1 yr-1"))
-
Differences between the data and the variable record can be accounted for in the data Formats record.
-
-
Under Variable, select the variable as it is recorded in bety.
-
Under Name, write the name the variable has in your data file.
-
Under Unit, write the units the variable has in your data file.
-
-
NOTE: All units must be written in a udunits compliant format. To check that your units can be read by udunits, in R, load the udunits2 package and run udunits2::is.parseable("g C m-2 yr-1")
-
If the name or the units are the same, you can leave the Name and Unit fields blank. This is can be seen with the variable LAI.
-
-
-
19.5.2.1.2 Storage Type
-
Storage Type only needs to be specified if the variable is stored in a format other than what would be expected (e.g. if numeric values are stored as quoted character strings).
-
One such example is time variables.
-
PEcAn converts all dates into POSIX format using R functions such as strptime. These functions require that the user specify the format in which the date is written.
-
The default is "%Y-%m-%d %H:%M:%S" which would look like "2017-01-01 00:00:00"
Second as integer (00–61), allowing for up to two leap-seconds (but POSIX-compliant implementations will ignore leap seconds).
-
-
-
%T
-
Equivalent to %H:%M:%S.
-
-
-
%y
-
Year without century (00–99). On input, values 00 to 68 are prefixed by 20 and 69 to 99 by 19 – that is the behaviour specified by the 2004 and 2008 POSIX standards, but they do also say ‘it is expected that in a future version the default century inferred from a 2-digit year will change’.
-
-
-
%Y
-
Year with century.
-
-
-
-
-
-
19.5.2.1.3 Column Number
-
If your data is in text format with variables in a standard order then you can specify the Column Number for the variable. This is required for text files that lack headers.
-
-
-
-
19.5.2.2 Retrieving Format Information
-
To acquire Format information from a Format record, use the R function query.format.vars
-
-
19.5.2.2.1 Inputs
-
-
bety: connection to BETY
-
input.id=NA and/or format.id=NA: Input or Format record ID from BETY
-
-
At least one must be specified. Defaults to format.id if both provided.
-
-
var.ids=NA: optional vector of variable IDs. If provided, limits results to these variables.
-
-
-
-
19.5.2.2.2 Output
-
-
R list object containing many things. Fill this in.
-
-
-
-
-
-
-
19.6 Creating a new benchmark reference run
-
The purpose of the reference run record in BETY is to store all the settings from a run that are necessary in exactly recreating it.
-
The pecan.xml file is the home of absolutely all the settings for a particular run in pecan. However, much of the information in the pecan.xml file is server and user specific and more importantly, the pecan.xml files are stored on individual servers and may not be available to the public.
-
When a run that is performed using pecan is registered as a reference run, the settings that were used to make that run are made available to all users through the database.
-
All completed runs are not automatically registered as reference runs. To register a run, navigate to the benchmarking section of the workflow visualizations Shiny app.
Many of PEcAn’s functions require inputs that are provided as data.
-These can be in the data or the inst/extdata folders of a package.
-Data that are not package specific should be placed in the PEcAn.all (base/all) or
-PEcAn.utils (base/utils) packages.
-
Some useful conventions:
-
-
-
31.4 Settings
-
-
A generic settings can be found in the PEcAn.all package
database settings can be specified, and tests run only if a connection is available
-
-
We currently use the following database to run tests against; tests that require access to a database should check db.exists() and be skipped if it returns FALSE to avoid failed tests on systems that do not have the database installed.
Notes: 2007 synthesis. Fluxnet2015 supercedes this for sites that have been updated
-
-
-
14.11 Geostreams
-
Scale: site
-
Resolution: varies
-
Availability: varies by site
-
Notes: This is a protocol, not a single archive. The PEcAn functions currently default to querying [https://terraref.ncsa.illinois.edu/clowder/api/geostreams], which requires login and contains data from only two sites (Urbana IL and Maricopa AZ). However the interface can be used with any server that supports the Geostreams API.
-
-
-
14.12 ERA5
-
Scale: Global
-
Resolution: 3 hrs and 31 km
-
Availability: 1950-present
-
Notes:
-
It’s important to know that the raw ERA5 tiles needs to be downloaded and registered in the database first. Inside the inst folder in the data.atmosphere package there are R files for downloading and registering files in the BETY. However, it assumes that you have registered and setup your API requirements. Check out how to setup your API [here] (https://confluence.ecmwf.int/display/CKB/How+to+download+ERA5#HowtodownloadERA5-3-DownloadERA5datathroughtheCDSAPI).
-In the inst folder you can find two files (ERA5_db_register.R and ERA5_USA_download.R). If you setup your ecmwf account as it’s explained in the link above, ERA5_USA_download.R will help you to download all the tiles with all the variables required for pecan extract.nc.ERA5 function to generate pecan standard met files. Besides installing the required packages for this file, it should work from top to bottom with no problem. After downloading the tiles, there is simple script in ERA5_db_register.R which helps you register your tiles in the bety. met.process later on uses that entery to find the required tiles for extracting met data for your sites. There are important points about this file. 1- Make sure you don’t change the site id in the script (which is the same the ParentSite in ERA5 registeration xml file). 2- Make sure the start and end date in that script matches the downloaded tiles. Set your ERA5.files.path to where you downloaded the tiles and then the rest of the script should be working fine.
-
-
-
-
14.13 Download GFDL
-
The Downlad.GFDL function assimilates 3 hour frequency CMIP5 outputs generated by multiple GFDL models. GFDL developed several distinct modeling streams on the timescale of CMIP5 and AR5. These models include CM3, ESM2M and ESM2G with a spatial resolution of 2 degrees latitude by 2.5 degrees longitude. Each model has future outputs for the AR5 Representative Concentration Pathways ranging from 2006-2100.
-
-
-
14.14 CM3
-
GFDL’s CMIP5 experiments with CM3 included many of the integrations found in the long-term CMIP5 experimental design. The focus of this physical climate model is on the role of aerosols, aerosol-cloud interactions, and atmospheric chemistry in climate variability and climate change.
-
-
-
14.15 ESM2M & ESM2G
-
Two new models representing ocean physics with alternative numerical frameworks to explore the implications of some of the fundamental assumptions embedded in these models. Both ESM2M and ESM2G utilize a more advanced land model, LM3, than was available in ESM2.1 including a variety of enhancements (Milly et al., in prep). GFDL’s CMIP5 experiments with Earth System Models included many of the integrations found in the long-term CMIP5 experimental design. The ESMs, by design, close the carbon cycle and are used to study the impact of climate change on ecosystems, ecosystem changes on climate and human activities on ecosystems.
We recommend that all new users begin with [PEcAn Hands-On Demo 01: Basic Run]. The documentation below assumes you are already familiar with how to navigate to PEcAn’s interactive web interface for running models.
-
-
6.1 Site and model selection
-
This page is used to select the model to run and the site at which you would like to run that model.
-
NOTE: If this page does not load for you, it may be related to a known Google Maps API key issue. See issue #1269 for a possible solution.
-
-
6.1.1 Selecting a model
-
-
On the Select Host webpage use the Host pull-down menu to select the server you want to run on. PEcAn is designed to allow models to be run both locally and on remote high-performance computing (HPC) resources (i.e. clusters). We recommend that users start with local runs. More information about connecting your PEcAn instance to a cluster can be found on the Remote execution with PEcAn page.
-
Next, select the model you want to run under the Model pull-down menu. The list of models currently supported by PEcAn, along with information about these models, is available on the PEcAn Models page.
-
-
If a PEcAn-supported model is not listed, this is most likely because the model has not been installed on the server. The PEcAn team does not have permissions to redistribute all of the models that are coupled to it, so you will have to install some PEcAn-compatible models yourself. Please consult the PEcAn model listing for information about obtaining and installing different models. Once the model is installed and you have added the location of the model executable to Bety (see Adding An Ecosystem Model), your model should appear on the PEcAn Select Host page after your refresh the page.
-
If you would like to add a new model to PEcAn please consult our guide for Adding an Ecosystem Model and contact the PEcAn team for assistance.
-
-
If selecting your model causes your site to disappear from the Google Map, that means the site exists but there are no drivers for that model-site combination registered in the database.
-
-
Click the “Conversion” checkbox. If your site reappears, that means PEcAn should be able to automatically generate the required inputs for this site by converting from existing input files in other formats.
-
If the site still does not reappear, that means there are required input files for that model-site combination that PEcAn cannot autogenerate. This may be because the model has unique input requirements or because it has not yet been fully coupled to the PEcAn input processing workflow. Go to the troubleshooting section under Selecting a site for more information on diagnosing what drivers are missing.
-
-
-
-
-
6.1.2 Selecting a site
-
-
-
6.1.3 Site Groups
-
-
PEcAn provides the option of organizing sites into groups to make them easier to find and easier to run as a group. We have pre-loaded a number of common research networks (e.g., FLUXNET, LTER, NEON), but you are free to create new site groups through Bety.
-
If you are searching for a site that is not part of an existing site group, or you are unsure which site group it belongs to, select “All Sites” to see all sites in Bety. Note that this may take a while to render.
-
-
-
-
6.1.4 Using existing sites
-
-
Find the site on the map The simplest way of determining if a site exists in PEcAn is through the Google Map interface of the web-based workflow. You’ll want to make sure that the “Site Group” is set to “All Sites” and the “Model” is set to “All Models”.
-
Find the site in BETY If the site is not on the map, it may still be in Bety but with insufficient geographic information. To locate the site in Bety, first login to your local version of the BETY database. If using the VM, navigate to localhost:6480/bety and login with username bety and password illinois. Then, navigate to Data > Sites and use the “Search” box to search for your site. If you do find your site, click “Edit” and add geographic information so that the site will show up on the map. Also, note that the site ID number shows up in the URL for the “Show” or “Edit” pages. This ID is often useful to know, for example when editing a PEcAn settings file by hand. If you did not find you site, follow the instructions below to add a site.
-
-
-
-
6.1.5 Adding a new site
-
(TODO: Move most of this out)
-
-
Log into Bety as described above.
-
Pick a citation for your site Each site requires an associated “citation” that must be added before the site itself is added. First, navigate to “Data > Citations” and use the “Search” box to see if the relevant citation already exists. If it does, click the check mark under “Actions” to proceed to site creation.
-
-
-
To create a new citation, click the New Citation button, fill in the fields, and then click “Create”. The “field URL” should contain the web address that takes you to this publication on the publisher’s website. The “PDF” field should be the full web address to a PDF for this citation.
-
Note that our definition of a citation is flexible, and a citation need not be a peer-reviewed publication. Most of the fields in “New Citation” can be left blank, but we recommend at least adding a descriptive title, such as “EBI Farm Field Data” and a relevant contact person as the “Author”.
-
-
-
Once the Citation is created or selected this should automatically take you to the Sites page and list any Sites already associated with this citation. To create a new site click the New Site button.
-
When creating a new site, the most important fields are the Site name and coordinates (latitude and longitude). The coordinates can be entered by hand or by clicking on the site location on the Google Map interface. All other information is optional, but can be useful for searching and indexing purposes.
-
When you are done click Create. At this point, once the PEcAn site-level page is refreshed, the site should automatically appear.
-
-
-
-
6.1.6 Troubleshooting
-
-
6.1.6.1 My site shows up when I don’t have any model selected, but disappears once I select the model I want to run
-
Selecting a model will cause PEcAn to filter the available sites based on whether they possess the required Inputs for a given model (e.g. meteorology). To check what Inputs are missing for a site point your browser to the pecan/checksite.php webpage (e.g. localhost:6480/pecan/checksite.php). This page looks virtually identical to the site selection page, except that it has a Check button instead of Prev and Next. If you select a Machine, Model, and Site and then click Check the page should return a list of what Inputs are missing (listing both the name and the Format ID number). Don’t forget that its possible for PEcAn to have required Inputs in its database, but just not have them for the Machine where you want to run.
-
To see more about what Inputs a given model can accept, and which of those are required, take a look at the MODEL_TYPE table entry in the database (e.g. go to localhost:6480/bety; Select Runs > Model Type; and then click on the model you want to run).
-
For information about loading missing Inputs into the database visit Input records in BETY, and also read the rest of the pages under this section, which will provide important information about the specific classes of Inputs (e.g. meteorology, vegetation, etc).
-
Finally, we are continually developing and refining workflows and standards for processing Input data in a model-agnostic way. The details about what Inputs can be processed automatically are discussed input-by-input in the sections below. For those looking to dive into the code or troubleshoot further, these conversions are ultimately handled under the PEcAn.workflow::do_conversions workflow module.
-
-
-
-
-
6.2 Model configuration
-
This page is used for basic model configuration, including when your model will run and what input data it will use.
-
-
6.2.1 Choosing meteorology
-
Once a Machine, Model, and Site have been selected, PEcAn will take you to the Input selection page. From this page you will select what Plant Functional Type (PFT) you want to run at a site, the start and end dates of the run, and various Input selections. The most common of these across all models is the need to specify meteorological forcing data. The exact name of the menu item for meteorology will vary by model because all of the Input requirements are generated individually for each model based on the MODEL_TYPE table. In general there are 3 possible cases for meteorology
-
-
PEcAn already has driver files in its database
-
PEcAn does not have drivers, but can generate them from publicly available data
-
You need (or want) to upload your own drivers
-
-
The first two cases will appear automatically in the the pull down menu. For meteorological files that already exist you will see the date range that’s available. By contrast, met that can be generated will appear as “Use ”, where is the origin of the data (e.g. “Use Ameriflux” will use the micromet from an Ameriflux eddy covariance tower, if one is present at the site).
-
If you want to upload your own met data this can be done in three ways.
-
-
The default way to add met data is to incorporate it into the overall meteorological processing workflow. This is preferred if you are working with a common meteorological data product that is not yet in PEcAn’s workflow. This case can be divided into two special cases:
-
-
Data is in a common MIME-type that PEcAn already has a converter for (e.g. CSV). In this case you’ll want to create a new Format record for the meta-data so that the existing converter can process this data. See documentation for [Creating a new Format record in BETY] for more details.
-
Data is in a more complicated format or interactive database, but large/useful enough to warrent a custom conversion function. Details on creating custom met conversions is in the [Input Conversion], though at this stage you would also be strongly encouraged to contact the PEcAn development team.
-
-
The second-best way is to upload data in PEcAn’s standard meteorological format (netCDF files, CF metadata). See [Input Conversion] for details about variables and units. From this standard, PEcAn can then convert the file to the model-specific format required by the model you have chosen. This approach is preferred for a rare or one-off meterological file format, because PEcAn will also be able to convert the file into the format required by any other model as well.
-
The last option for adding met data is to add it in a model-specific format, which is often easiest if you’ve already been running your model at a site and are just switching to using PEcAn.
-
-
-
-
6.2.2 Met workflow
-
In a nutshell, the PEcAn met workflow is designed to reduce the problem of converting n possible met inputs into m possible model formats, which requires n x m conversion functions as well as numerous custom functions for downscaling, gap filling, etc. Instead, PEcAn works with a single met standard, and thus requires n conversion functions, one for converting each data source into the PEcAn standard, and then m conversion functions for converting from that standard to what an individual model requires. For a new model joining the PEcAn system the burden in particularly low – writing one conversion function provides access to n inputs. Similarly, PEcAn performs all other operations/manipulations (extracting a site, downscaling, gap filling, etc) within the PEcAn standard, which means these operations only need be implemented once.
-
Consider a generic met data product named MET for simplicity. PEcAn will use a function, download.MET, to pull data for the selected year from a public data source (e.g. Ameriflux, North American Regional Reanalysis, etc). Next, PEcAn will use a function, met2CF.MET, to convert the data into the PEcAn standard. If the data is already at the site scale it will then gapfill the data. If the data is a regional or global data product, PEcAn will then permute the data to allow easier site-level extraction, then it will extract data for the requested site and data range. Modules to address the temporal and spatial downscaling of meteorological data products, as well as their uncertainties, are in development but not yet part of the operational workflow. All of these functions are located within the data.atmosphere module.
-
Once data is in the standard format and processed, it will be converted to the model-specific format using a met2model.MODEL function (located in that MODEL’s module).
-
More detailed information on how PEcAn processes inputs can be found on our [Input Conversion] page.
-
-
-
6.2.3 Troubleshooting meteorological conversions
-
At the current moment, most of the issues below address possible errors that the Ameriflux meteorology workflow might report
-
-
6.2.3.1 Could not do gapfill … The following variables have NA’s
-
This error message means that there were gaps in the downloaded data, for whatever variables that were listed, which were larger than the current algorithm could fill. Particularly common is missing radiation or PAR data, as Ameriflux frequently converts nighttime data to NULL, and work is in progress to detect this based on solar geometry. Also common are incomplete years (first or last year of tower operations).
-
-
-
6.2.3.2 Could not get information about . Is this an Ameriflux site?
-
This message occurs when PEcAn believes that a site is part of Ameriflux (because it was listed on the Ameriflux or FLUXNET webpage and has a US-* site code), but no data is present on the Ameriflux server. The most common reasons for this is that you have selected a site that has not submitted data to Ameriflux yet (or that data hasn’t been processed yet), or you have selected a year that’s outside the tower’s operational period. Visit Ameriflux and FLUXNET for lists of available site years.
-
-
-
6.2.3.3 Could not download data for for the year
-
This is similar to the previous error, but in this case PEcAn did find data for the site listed, but just not for the year requested. This can usually be fixed by just altering the years of the run to match those with available data.
-
-
-
6.2.3.4 I could not find the requested var (or dimvar) in the file!
-
PEcAn could not find a required variable within the downloaded file. Most likely this is due to that variable not being measured at this site. The most common cause of failure is the absence of atmospheric pressure data (PRESS), but since most models have a low sensitivity to this variable we are working on methods to estimate this from other sources.
-
-
-
-
-
6.3 Selecting Plant Functional Types (PFTs) and other parameter groupings.
-
-
6.3.1 Using existing PFTs
-
PEcAn does not automatically know what vegetation types are present at your study site so you need to select the PFT.
-
Some models, such as ED2 and LINKAGES, support competition among multiple PFTs and thus you are encouraged to highlight multiple choices. Other models, such as SIPNET and DALEC, only support one PFT at a site.
-
Many models also have parameters that control non-vegetation processes (e.g. soil biogeochemistry and hydrology). PEcAn allows users to assign these parameters to functional groups as well (e.g. a soils PFT)
-
-
-
6.3.2 Creating new PFTs
-
To modify or add a new Plant Functional Type (PFT), or to change a PFT’s priors, navigate
-on the grey menu bar to Data > PFTs
-
-
To add a new pft, click “new PFT” at the top and enter a name and description. (hint:
-we’re trying to name PFTs based on model.biome.pft, ED2 is the default model if one
-isn’t specified)
-
To add new species to a PFT click on [+] View Related Species and type the species,
-genus, or family you are looking for into the Search box. Click on the + to add.
-
To remove a species from a PFT, click on [+] View Related Species and click on the X
-of the species you want to remove from the PFT.
-
To remove a prior, click [-] View Related Prior and click on the X of the variable who’s
-prior you want to remove. This will cause the parameter to be excluded from all
-analyses (meta-analysis, sensitivity analysis, etc) and revert to its default value.
-
To add a prior, choose one from the white box of priors on the right to choose.
-
To view the specification of a prior, or to add a new prior, click BETY-DB > Priors and
-enter the information on the variable, distribution name, distribution parameters, etc. N
-is the sample size underlying the prior specification (0 is ok for uninformative priors).
-
You can also got to Data > Variables in order to use the search function to find an
-existing variable (or create a new one). Please try not to create new variables
-unnecessarily (e.g. changes of variable name or units to what your model uses is handled
-internally, so you want to find the trait with the correct MEANING).
-
-
Additional information on adding PFTs, Species, and Priors can be found in Adding An [Ecosystem Model].
-
-
-
6.3.3 Choosing initial vegetation
-
On the Input Selection webpage, in addition to selecting PFTs, start & end dates, and meteorology, many models also require some way of specifying the initial conditions for the vegetation, which may range from setting the aboveground biomass and LAI up to detailed inventory-like data on species composition and stand structure.
-
At the moment, PEcAn has three cases for initial conditions:
-
-
If files already exist in the database, they can simply be selected from the menu. For ED2, there are 3 different veg files (site, pss, css) and it is important that you select a complete set, not mix and match.
As with meteorology, PEcAn is working to develop a model-agnostic workflow for converting various sources of vegetation data to common standards, developing common processing tools, and then writing out to model-specific formats. This process is in a much early stage than the meteorology workflow, as we are still researching what the options are for standard formats, but ultimately aims to be much more broad in scope, considering not just plot inventory data but also historical documentation, paleoecological proxies, satellite remote sensing (e.g. LANDSAT), airborne hyperspectral imagery, and active remote sensing (Lidar, Radar).
-
At the moment, what is functional is a prototype workflow that works for inventory-based vegetation data. This data can come from either files that have been registered with the BETY Inputs and Formats tables or can be queried from the USFS Forest Inventory and Analysis (FIA). For more information visit Section 13.1.2.2 Vegetation Data
-
-
-
6.3.4 US FIA
-
This tool works with an internal copy of the FIA that is uploaded to a postGRES database along side BETY, however for space reasons this database does not ship with the PEcAn VM. To turn this feature on:
For web-base runs, specify the database settings in the config.php
-
For R-based runs, specify the database settings in the THE PEcAn XML
-
-
More detailed information on how PEcAn processes inputs can be found on our [Input Conversion]page.
-
-
-
6.3.5 Spin up
-
A number of ecosystem models are typically initialized by spinning up to steady state. At the moment PEcAn doesn’t handle spin up automatically (e.g. looping met, checking for stability), but there are various ways to achieve a spin-up within the system.
-
Option 1: If there are model-specific settings in a model’s settings/config file, then that file can be accessed by clicking on the Edit model config check box. If this box is selected then PEcAn will pause the site run workflow after it has generated your model config file, but before it runs the model, and give you an opportunity to edit the file by hand, allowing you to change any model-specific spin up settings (e.g met recycling, spin up length)
-
Option 2: Set start_year very early and set the met drivers to be a long time series (e.g. PalEON, something custom uploaded to Inputs)
-
Option 3: In the MODEL_TYPE table, add your model’s restart format as an optional input, modify the model specific write.config function to use that restart, and then load a previous spin-up to the Inputs table
-
Beyond these options, we hope to eventually develop more general, model-agnostic tools for spin up. In particular, we have started to explore the accelerated spin-up and semi-analytical techniques being developed by Yiqi Luo’s lab
-
-
-
6.3.6 Selecting a soils product
-
Many models have requirements for soils information, which may include: site-specific soil texture and depth information; soil biogeochemical initial conditions (e.g. soil carbon and nitrogen pools); soil moisture initial conditions; and soil thermal initial conditions.
-
As with Choosing initial vegetation, we eventually hope to develop data standards, soils workflows, and spin-up tools, but at the moment this workflow is in the early stages of development. Model requirements need to be met byCreating a new Input record in BETY into the database or using files that have already been uploaded. Similar to met, we recommend that this file be in the PEcAn-standard netCDF described below, but model-specific files can also be registered.
-
-
-
6.3.7 Soil texture, depth, and physical parameters
-
A PEcAn-standard netCDF file format exists for soil texture, depth, and physical parameters, using PEcAn standard names that are largely a direct extention of the CF standard.
-
The easiest way to create this file is with the PEcAn R function soil2netcdf as described in the Soil Data section of the Advanced Users Guide.
-
A table of standard names and units can be listed using PEcAn.data.land::soil.units() with no arguments.
-
More detailed information on how PEcAn processes inputs can be found on our [Input Conversion] page.
-
-
-
6.3.8 Other model inputs
-
Finally, any other model-specific inputs (e.g. N deposition, land use history, etc), should be met by Creating a new Input record in BETY or using files that have already been uploaded.
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
Our Standards
-
Examples of behavior that contributes to creating a positive environment include:
-
-
Using welcoming and inclusive language
-
Being respectful of differing viewpoints and experiences
-
Gracefully accepting constructive criticism
-
Focusing on what is best for the community
-
Showing empathy towards other community members
-
-
Examples of unacceptable behavior by participants include:
-
-
The use of sexualized language or imagery and unwelcome sexual attention or advances
-
Trolling, insulting/derogatory comments, and personal or political attacks
-
Public or private harassment
-
Publishing others’ private information, such as a physical or electronic address, without explicit permission
-
Other conduct which could reasonably be considered inappropriate in a professional setting
-
-
Our Responsibilities
-
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
Scope
-
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
-
Enforcement
-
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at pecanproj[at]gmail.com. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
-
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.
In addition to the state assimilation routines found in the assim.sequential module, another approach for state data assimilation in PEcAn is through the DART workflow created by the DARES group in NCAR.
-
This section gives a straight-forward explanation how to implement DART, focused on the technical aspects of the implementation. If there are any questions, feel free to send @Viskari an email (tt.viskari@gmail.com) or contacting DART support as they are quite awesome in helping people with problems. Also, if there are any suggestions on how to improve the wiki, please let me know.
-
Running with current folders in PEcAn
-
Currently the DART folders in PEcAn are that you can simply copy the structure there over a downloaded DART workflow and it should replace/add relevant files and folders. The most important step after that is to check and change the run paths in the following files:
-Path_name files in the work folders
-T_ED2IN file, as it indicates where the run results be written.
-advance_model.csh, as it indicates where to copy files from/to.
-
Second thing is setting the state vector size. This is explained in more detail below, but essentially this is governed by the variable model_size in model_mod.f90. In addition to this, it should be changed in utils/F2R.f90 and R2F.f90 programs, which are responsible for reading and writing state variable information for the different ensembles. This also will be expanded below. Finally, the new state vector size should be updated for any other executable that runs it.
-
Third thing needed are the initial condition and observation sequence files. They will always follow the same format and are explained in more detail below.
-
Finally the ensemble size, which is the easiest to change. In the work subfolder, there is a file named input.nml. Simply changing the ensemble size there will set it for the run itself. Also remember that initial conditions file should have the equal amount of state vectors as there are ensemble members.
-
Adjusting the workflow
-
The central file for the actual workflow is advance_model.csh. It is a script DART calls to determine how the state vector changes between the two observation times and is essentially the only file one needs to change when changing state models or observations operators. The file itself should be commented to give a good idea of the flow, but beneath is a crude order of events.
-1. Create a temporary folder to run the model in and copy/link required files in to it.
-2. Read in the state vector values and times from DART. Here it is important to note that the values will be in binary format, which need to be read in by a Fortran program. In my system, there is a program called F2R which reads in the binary values and writes out in ascii form the state vector values as well as which ED2 history files it needs to copy based on the time stamps.
-3. Run the observation operator, which writes the state vector state in to the history files and adjusts them if necessary.
-4. Run the program.
-5. Read the new state vector values from output files.
-6. Convert the state vector values to the binary. In my system, this is done by the program R2F.
-
Initial conditions file
-
The initial conditions file, commonly named filter_ics although you can set it to something else in input.nml, is relatively simple in structure. It has one sequence repeating over the number of ensemble members.
-First line contains two times: Seconds and days. Just use one of them in this situation, but it has to match the starting time given in input.nml.
-After that each line should contain a value from the state vector in the order you want to treat them.
-R functions filter_ics.R and B_filter_ics.R in the R folder give good examples of how to create these.
-
Observations files
-
The file which contains the observations is commonly known as obs_seq.out, although again the name of the file can be changed in input.nml. The structure of the file is relatively straight-forward and the R function ObsSeq.R in the R subfolder has the write structure for this. Instead of writing it out here, I want to focus on a few really important details in this file.
-Each observations will have a time, a value, an uncertainty, a location and a kind. The first four are self-explanatory, but the kind is really important, but also unfortunately really easy to misunderstand. In this file, the kind does not refer to a unit or a type of observation, but which member of the state vector is this observation of. So if the kind was, for example, 5, it would mean that it was of the fifth member of the state vector. However, if the kind value is positive, the system assumes that there is some sort of an operator change in comparing the observation and state vector value which is specified in a subprogram in model_mod.f90.
-
So for an direct identity comparison between the observation and the state vector value, the kind needs to be negative number of the state vector component. Thus, again if the observation is of the fifth state vector value, the kind should be set as -5. Thus it is recommendable that the state vector values have already been altered to be comparable with the observations.
-
As for location, there are many ways to set in DART and the method needs to be chosen when compiling the code by giving the program which of the location mods it is to use. In our examples we used a 1-dimensional location vector with scaled values between 0 and 1. For future it makes sense to switch to a 2 dimensional long- and lat-scale, but for the time being the location does not impact the system a lot. The main impact will be if the covariances will be localized, as that will be decided on their locations.
-
State variable vector in DART
-
Creating/adjusting a state variable vector in DART is relatively straight-forward. Below are listed the steps to specify a state variable vector in DART.
-
I. For each specific model, there should be an own folder within the DART root models folder. In this folder there is a model_mod.f90, which contains the model specific subroutines necessary for a DART run.
-
At the beginning of this file there should be the following line:
-
integer, parameter :: model_size = [number]
-
The number here should be the number of variables in the vector. So for example if there were three state variables, then the line should look like this:
-
integer, parameter :: model_size = 3
-
This number should also be changed to match with any of the other executables called during the run as indicated by the list above.
-
-
In the DART root, there should be a folder named obs_kind, which contains a file called DEFAULT_obs_kind_mod.F90. It is important to note that all changes should be done to this file instead of obs_kind_mod.f90, as during compilation DART creates obs_kind_mod.f90 from DEFAULT_obs_kind_mod.F90.
-This program file contains all the defined observation types used by DART and numbers them for easier reference later. Different types are classified according to observation instrument or relevant observation phenomenon. Adding a new type only requires finding an unused number and starting a new identifying line with the following:
-
-
integer, parameter, public :: &
-KIND_…
-
Note that the observation kind should always be easy to understand, so avoid using unnecessary acronyms. For example, when adding an observation type for Leaf Area Index, it would look like below:
-
integer, parameter, public :: &
-KIND_LEAF_AREA_INDEX = [number]
-
-
In the DART root, there should be a folder named obs_def, which contains several files starting with obs_def_. There files contain the different available observation kinds classified either according to observation instrument or observable system. Each file starts with the line
-
-
! BEGIN DART PREPROCESS KIND LIST
-
And end with line
-
! END DART PREPROCESS KIND LIST
-
The lines between these two should contain
-
! The desired observation reference, the observation type, COMMON_CODE.
-
For example, for observations relating to phenology, I have created a file called obs_def_phen_mod.f90. In this file I define the Leaf Area Index observations in the following way.
-
! BEGIN DART PREPROCESS KIND LIST
-! LAI, TYPE_LEAF_AREA_INDEX, COMMON_CODE
-! END DART PREPROCESS KIND LIST
-
Note that the exclamation marks are necessary for the file.
-
-
In the model specific folder, in the work subfolder there is a namelist file input.nml. This contains all the run specific information for DART. In it, there is a subtitle &preprocess, under which there is a line
-
-
input_files = ‘….’
-
This input_files sections must be set to refer to the obs_def file created in step III. The input files can contain references to multiple obs_def files if necessary.
-
As an example, the reference to the obs_def_phen_mod.f90 would look like
-input_files = ‘../../../obs_def/obs_def_phen_mod.f90’
-
V. Finally, as an optional step, the different values in state vector can be typed. In model_mod, referred to in step I, there is a subroutine get_state_meta_data. In it, there is an input variable index_in, which refers to the vector component. So for instance for the second component of the vector index_in would be 2. If this is done, the variable kind has to be also included at the beginning of the model_mod.f90 file, at the section which begins
-
use obs_kind_mod, only ::
-
The location of the variable can be set, but for a 0-dimensional model we are discussing here, this is not necessary.
-
Here, though, it is possible to set the variable types by including the following line
-
if(index_in .eq. [number]) var_type = [One of the variable kinds set in step II]
-
-
If the length of the state vector is changed, it is important that the script ran with DART produces a vector of that length. Change appropriately if necessary.
-
-
After these steps, DART should be able to run with the state vector of interest.
The database synchronization consists of 2 parts:
-- Getting the data from the remote servers to your server
-- Sharing your data with everybody else
-
-
16.1 How does it work?
-
Each server that runs the BETY database will have a unique machine_id and a sequence of ID’s associated. Whenever the user creates a new row in BETY it will receive an ID in the sequence. This allows us to uniquely identify where a row came from. This is information is crucial for the code that works with the synchronization since we can now copy those rows that have an ID in the sequence specified. If you have not asked for a unique ID your ID will be 99.
-
The synchronization code itself is split into two parts, loading data with the load.bety.sh script and exporting data using dump.bety.sh. If you do not plan to share data, you only need to use load.bety.sh to update your database.
-
-
-
16.2 Set up
-
Requests for new machine ID’s is currently handled manually. To request a machine ID contact Rob Kooper kooper@illinois.edu. In the examples below this ID is referred to as ‘my siteid’.
-
To setup the database to use this ID you need to call load.bety in ‘CREATE’ mode (replacing with the ID if your site)
WARNING: At the moment running CREATE deletes all current records in the database. If you are running from the VM this includes both all runs you have done and all information that the database is prepopulated with (e.g. input and model records). Remote records can be fetched (see below), but local records will be lost (we’re working on improving this!)
-
-
-
16.3 Fetch latest data
-
When logged into the machine you can fetch the latest data using the load.bety.sh script. The script will check what site you want to get the data for and will remove all data in the database associated with that id. It will then reinsert all the data from the remote database.
-
The script is configured using environment variables. The following variables are recognized:
-- DATABASE: the database where the script should write the results. The default is bety.
-- OWNER: the owner of the database (if it is to be created). The default is bety.
-- PG_OPT: additional options to be added to psql (default is nothing).
-- MYSITE: the (numerical) ID of your site. If you have not requested an ID, use 99; this is used for all sites that do not want to share their data (i.e. VM). 99 is in fact the default.
-- REMOTESITE: the ID of the site you want to fetch the data from. The default is 0 (EBI).
-- CREATE: If ‘YES’, this indicates that the existing database (bety, or the one specified by DATABASE) should be removed. Set to YES (in caps) to remove the database. THIS WILL REMOVE ALL DATA in DATABASE. The default is NO.
-- KEEPTMP: indicates whether the downloaded file should be preserved. Set to YES (in caps) to keep downloaded files; the default is NO.
-- USERS: determines if default users should be created. Set to YES (in caps) to create default users with default passwords. The default is NO.
-
All of these variables can be specified as command line arguments as well, to see the options use -h.
-
load.bety.sh -h
-./scripts/load.bety.sh [-c YES|NO] [-d database] [-h] [-m my siteid] [-o owner] [-p psql options] [-r remote siteid] [-t YES|NO] [-u YES|NO]
- -c create database, THIS WILL ERASE THE CURRENT DATABASE, default is NO
- -d database, default is bety
- -h this help page
- -m site id, default is 99 (VM)
- -o owner of the database, default is bety
- -p additional psql command line options, default is empty
- -r remote site id, default is 0 (EBI)
- -t keep temp folder, default is NO
- -u create carya users, this will create some default users
-
-dump.bety.sh -h
-./scripts/dump.bety.sh [-a YES|NO] [-d database] [-h] [-l 0,1,2,3,4] [-m my siteid] [-o folder] [-p psql options] [-u YES|NO]
- -a use anonymous user, default is YES
- -d database, default is bety
- -h this help page
- -l level of data that can be dumped, default is 3
- -m site id, default is 99 (VM)
- -o output folder where dumped data is written, default is dump
- -p additional psql command line options, default is -U bety
- -u should unchecked data be dumped, default is NO
-
-
-
16.4 Sharing data
-
Sharing your data requires a few steps. First, before entering any data, you will need to request an ID from the PEcAn developers. Simply open an issue at github and we will generate an ID for you. If possible, add the URL of your data host.
-
You will now need to synchronize the database again and use your ID. For example if you are given ID=42 you can use the following command: MYID=42 REMOTEID=0 ./scripts/load.bety.sh. This will load the EBI database and set the ID’s such that any data you insert will have the right ID.
-
To share your data you can now run the dump.bey.sh. The script is configured using environment variables, the following variables are recognized:
-- DATABASE: the database where the script should write the results. The default is bety.
-- PG_OPT: additional options to be added to psql (default is nothing).
-- MYSITE: the ID of your site. If you have not requested an ID, use 99, which is used for all sites that do not want to share their data (i.e. VM). 99 is the default.
-- LEVEL: the minimum access-protection level of the data to be dumped (0=private, 1=restricted, 2=internal collaborators, 3=external collaborators, 4=public). The default level for exported data is level 3.
-- note that currently only the traits and yields tables have restrictions on sharing. If you share data, records from other (meta-data) tables will be shared. If you wish to extend the access_level to other tables please submit a feature request.
-- UNCHECKED: specifies whether unchecked traits and yields be dumped. Set to YES (all caps) to dump unchecked data. The default is NO.
-- ANONYMOUS: specifies whether all users be anonymized. Set to YES (all caps) to keep the original users (INCLUDING PASSWORD) in the dump file. The default is NO.
-- OUTPUT: the location of where on disk to write the result file. The default is ${PWD}/dump.
-
NOTE: If you want your dumps to be accessible to other PEcAn servers you need to perform the following additional steps
-
-
Open pecan/scripts/load.bety.sh
-
In the DUMPURL section of the code add a new record indicating where you are dumping your data. Below is the example for SITE number 1 (Boston University)
-
-
elif [ "${REMOTESITE}" == "1" ]; then
- DUMPURL="http://psql-pecan.bu.edu/sync/dump/bety.tar.gz"
-
-
Check your Apache settings to make sure this location is public
-
Commit this code and submit a Pull Request
-
From the URL in the Pull Request, PEcAn administrators will update the machines table, the status map, and notify other users to update their cron jobs (see Automation below)
-
-
Plans to simplify this process are in the works
-
-
-
16.5 Automation
-
Below is an example of a script to synchronize PEcAn database instances across the network.
-
db.sync.sh
-
#!/bin/bash
-## make sure psql is in PATH
-export PATH=/usr/pgsql-9.3/bin/:$PATH
-## move to export directory
-cd /fs/data3/sync
-## Dump Data
-MYSITE=1 /home/dietze/pecan/scripts/dump.bety.sh
-## Load Data from other sites
-MYSITE=1 REMOTESITE=2 /home/dietze/pecan/scripts/load.bety.sh
-MYSITE=1 REMOTESITE=5 /home/dietze/pecan/scripts/load.bety.sh
-MYSITE=1 REMOTESITE=0 /home/dietze/pecan/scripts/load.bety.sh
-## Timestamp sync log
-echo $(date +%c) >> /home/dietze/db.sync.log
-
Typically such a script is set up to run as a cron job. Make sure to schedule this job (crontab -e) as the user that has database privileges (typically postgres). The example below is a cron table that runs the sync every hour at 12 min after the hour.
All databases need maintenance performed on them. Depending upon the database type this can happen automatically, or it needs to be run through a scheduler or manually. The BETYdb database is Postgresql and it needs to be reindexed and vacuumed on a regular basis. Reindexing introduces efficiencies back into the database by reorganizing the indexes. Vacuuming the database frees up resources to the database by rearranging and compacting the database. Both of these operations are necessary and safe. As always if there’s a concern, a backup of the database should be made ahead of time. While running the reindexing and vacuuming commands, users will notice a slowdown at times. Therefore it’s better to run these maintenance tasks during off hours.
-
-
16.6.1 Reindexing the database
-
As mentioned above, reindexing allows the database to become more efficient. Over time as data gets updated and deleted, the indexes become less efficient. This has a negative inpact on executed statements. Reindexing makes the indexes efficient again (at least for a while) allowing faster statement execution and reducing the overall load on the database.
-
The reindex.bety.sh script is provided to simplify reindexing the database.
-
reindex.bety.sh -h
-./reindex.bety.sh [-c datalog] [-d database] [-h] [-i table names] [-p psql options] [-q] [-s] [-t tablename]
- -c catalog, database catalog name used to search for tables, default is bety
- -d database, default is bety
- -h this help page
- -i table names, list of space-separated table names to skip over when reindexing
- -p additional psql command line options, default is -U bety
- -q the reindexing should be quiet
- -s reindex the database after reindexing the tables (this should be done sparingly)
- -t tablename, the name of the one table to reindex
-
If the database is small enough it’s reasonable to reindex the entire database at one time. To do this manually run or schedule the REINDEX statement. For example:
-
reindex.bety.sh -s
-
For larger databases it may be desireable to reindex entire tables at a time. An efficient way to do this is to reindex the larger tables and then the entire database. For example:
For very large databases it may be desirable to reindex one or more individual indexes before reindexing tables and the databases. In this case running specific psql commands to reindex those specific indexes, followed by reindexing the table is a possible approach. For example:
-
psql -U bety -c "REINDEX INDEX index_yields_on_citation_id; REINDEX INDEX index_yields_on_cultivar_id;"
-reindex.bety.sh -t yields;
-
Splitting up the indexing commands over time allows the database to operate efficiently with minimal impact on users. One approach is to schedule the reindexing of large, complex tables at a spcific off-time during the week, followed by a general reindexing and excluding those large tables on a weekend night.
-
Please refere to the Automation section above for information on using cron to schedule reindexing commands.
-
-
-
16.6.2 Vacuuming the database
-
Vacuuming the BETYdb Postgresql database reduces the amount of resources it uses and introduces its own efficiencies.
-
Over time, modified and deleted records leave ‘holes’ in the storage of the database. This is a common feature for most databases. Each database has its own way of handing this, in Postgresql it’s the VACUUM command. The VACUUM command performs two main operations: cleaning up tables to make memory use more efficient, and analyze tables for optimum statement execution. The use of the keyword ANALYZE indicates the second operation should take place.
-
The vacuum.bety.sh script is provided to simplify vacuuming the database.
-
vacuum.bety.db -h
-./vacuum.bety.sh [-c datalog] [-d database] [-f] [-h] [-i table names] [-n] [-p psql options] [-q] [-s] [-t tablename] [-z]
- -c catalog, database catalog name used to search for tables, default is bety
- -d database, default is bety
- -f perform a full vacuum to return resources to the system. Specify rarely, if ever
- -h this help page
- -i table names, list of space-separated table names to skip over when vacuuming
- -n only vacuum the tables and do not analyze, default is to first vacuum and then analyze
- -p additional psql command line options, default is -U bety
- -q the export should be quiet
- -s skip vacuuming the database after vacuuming the tables
- -t tablename, the name of the one table to vacuum
- -z only perform analyze, do not perform a regular vacuum, overrides -n and -f, sets -s
-
For small databases with light loads it may be possible to set aside a time for a complete vacuum. During this time, commands executed against the database might fail (a temporary condition as the database gets cleaned up). The following commands can be used to perform all the vaccum operations in one go.
-
vacuum.bety.sh -f
-
Generally it’s not desireable to have down time. If the system running the database doesn’t need resources that the database is using returned to it, a FULL vacuum can be avoided. This is the default behavior of the script
-
vacuum.bety.sh
-
In larger databases, vacuuming the entire database can take a long time causing a negative impact on users. This means that individual tables need to be vacuumed. How often a vacuum needs to be performed is dependent upon a table’s activity. The more frequently updates and deletes occur on a table, the more frequent the vaccum should be. For large tables it may be desireable to separate the table cleanup from the analysis. An example for completely vacuuming and analyzing a table is:
Similar to indexes, vacuuming the most active tables followed by general database vacuuming and vacuum analyze may be a desireable approach.
-
Also note that it isn’t necessary to run VACUUM ANALYZE for each vacuum performed. Separating the commands and performing a VACUUM ANALYZE after several regular vacuums may be sufficient, with less load on the database.
-
If the BETYdb database is running on a system with limited resources, or with resources that have become limited, the VACCUM command can return resources to the system from the database. The normal vacuuming process releases resources back to the database for reuse, but not to the system; generally this isn’t a problem. Postgresql has a VACUUM keyword FULL that returns resources back to the system. Requesting a FULL vacuum will lock the table being vacuumed while it is being re-written preventing any statements from being executed against it. If performing VECUUM FULL against the entire database, only the table being actively worked on is locked.
-
To minimize the impact a VACUUM FULL has on users, it’s best to perform a normal vacuum before a FULL vacuum. If this approach is taken, there sould be a minimal time gap between the normal VACUUM and the VACUUM FULL commands. A normal vacuum allows changes to be made thus requiring the full vacuum to handle those changes, extending it’s run time. Reducing the time between the two commands lessens the work VACUUM FULL needs to do.
Give its impact, it’s typically not desireable to perform a VACUUM FULL after every normal vacuum; it should be done on an “as needed” basis or infrequently.
-
-
-
-
16.7 Troubleshooting
-
There are several possibilities if a scheduled cron job apepars to be running but isn’t producing the expected results. The following are suggestions on what to try to resolve the issue.
-
-
16.7.1 Username and password
-
The user that scheduled a cron job may not have access permissions to the database. This can be easily confirmed by running the command line from the cron job while logged in as the user that scheduled the job. An error message will be shown if the user doesn’t have permissions.
-
To resolve this, be sure to include a valid database user (not a BETYdb user) with their credentials on the command in crontab.
-
-
-
16.7.2 db_hba.conf file
-
Iit’s possible that the machine hosting the docker image of the database doesn’t have permissions to access the database. This may due to the cron job running on a machine that is not the docker instance of the database.
-
It may be necessary to look at the loga on the hosting machine to determine if database access permissions are causing a problem. Logs are stored in different locations depending upon the Operating System of the host and upon other environmental factors. This document doesn’t provide information on where to find the logs.
-
To begin, it’s best to look at the contents of the relevent database configuration file. The following command will display the contents of the db_hba.conf file.
This command should return a series of text lines. For each row except those begining with ‘local’, the fourth item describes the machines that can access the database. In some cases an IP mask is specified in the fifth that further restricts the machines that have access. The special work ‘all’ in the fourth column grants permissions to all machines. The last column on each line contains the authentication option for the machine(s) specified in the fourth column (with a possible fifth column IP mask modifier).
-
Ensure that the host machine is listed under the fourth column (machine addresse range, or ‘all’), is also included in the IP mask if one was specified, and finally that any authentication option are not set to ‘reject’. If the host machine is not included the db_hba.conf file will need to be updated to allow access.
Nodes: red = down, yellow = out-of-date schema, green = good
-
Edges: red = fail, yellow = out-of-date sync, green = good
-
-
-
16.9 Tasks
-
Following is a list of tasks we plan on working on to improve these scripts:
-- pecanproject/bety#368 allow site-specific customization of information and UI elements including title, contacts, logo, color scheme.
This section provides additional details about the BETY database used by PEcAn. It will discuss best practices for setting up the BETY database, how to backup the database and how to restore the database.
-
-
21.1 Best practices
-
When using the BETY database in non testing mode, it is best not to use the default users. This is accomplished when running the initialize of the database. When the database is initally created the database will be created with some default users (best known is the carya user) as well as the guestuser that can be used in the BETY web application. To disable these users you will either need to disable the users from the web interface, or you can reinitialize the database and remove the -u flag from the command line (the -u flag will create the default users). To disable the guestuser as well you can remove the -g flag from the command line, or disable the account from BETY.
-
The default installation of BETY and PEcAn will assume there is a database called bety with a default username and password. The default installation will setup the database account to not have any superuser abilities. It is also best to limit access to the postgres database from trusted hosts, either by using firewalls, or configuring postgresql to only accept connections from a limited set of hosts.
-
-
-
21.2 Backup of BETY database
-
It is good practice to make sure you backup the BETY database. Just creating a copy of the files on disk is not enough to ensure you have a valid backup. Most likely if you do this you will end up with a corrupted backup of the database.
-
To backup the database you can use the pg_dump command, which will make sure the database id backed up in a consistent state. You can run sudo -u postgres pg_dump -d bety -Z 9 -f bety.sql.gz, this will create a compressed file that can be used to resotore the database.
-
In the PEcAn distribution in scripts folder there is a script called backup.bety.sh. This script will create the backup of the database. It will create multiple backups allowing you to restore the database from one of these copies. The database will be backed up to one of the following files:
-- bety-d-X, daily backup, where X is the day of the month.
-- bety-w-X, weekly backup, where X is the week number in the year
-- bety-m-X, montly backup, where X is the month of the year
-- bety-y-X, yearly backup, where X is the actual year.
-Using this scheme, we can restore the database using any of the files generated.
-
It is recommeneded to run this script using a cronjob at midnight such that you have a daily backup of the database and do not have to remember to create these backups. When running this script (either cron or by hand) make sure to place the backups on a different machine than the machine that holds the database in case of a larger system failure.
-
-
-
21.3 Restore of BETY database
-
Hopefully this section will never need to be used. Following are 5 steps that have been used to restore the database. Before you start it is worth it to read up online a bit on restoring the database as well as join the slack channel and ask any of the people there for help.
-
-
stop apache (BETY/PEcAn web apps) service httpd stop or service apache2 stop
load database (assuming dump is called bety.sql.gz) zcat bety.sql.gz | grep -v search_path | sudo -u postgres psql -d bety
-
start apache again service httpd start or service apache2 start
-
-
If during step 5 there is a lot of errors, it is helpful to add -v ON_ERROR_STOP=1 to the end of the command. This will stop the restore at the first error and will help with debugging the issue.
This page will only list any steps you have to do to upgrade an existing system. When updating PEcAn it is highly encouraged to update BETY. You can find instructions on how to do this, as well on how to update the database in the Updating BETYdb gitbook page.
-
-
8.1.1 Updating PEcAn
-
The latest version of PEcAn code can be obtained from the PEcAn repository on GitHub:
The PEcAn build system is based on GNU Make.
-The simplest way to install is to run make from inside the PEcAn directory.
-This will update the documentation for all packages and install them, as well as all required dependencies.
-
For more control, the following make commands are available:
-
-
make document – Use devtools::document to update the documentation for all package.
-Under the hood, this uses the roxygen2 documentation system.
-
make install – Install all packages and their dependnencies using devtools::install.
-By default, this only installs packages that have had their code changed and any dependent packages.
-
make check – Perform a rigorous check of packages using devtools::check
-
make test – Run all unit tests (based on testthat package) for all packages, using devtools::test
-
make clean – Remove the make build cache, which is used to track which packages have changed.
-Cache files are stored in the .doc, .install, .check, and .test subdirectories in the PEcAn main directory.
-Running make clean will force the next invocation of make commands to operate on all PEcAn packages, regardless of changes.
-
-
The following are some additional make tricks that may be useful:
-
-
Install, check, document, or test a specific package – make .<cmd>/<pkg-dir>; e.g. make .install/utils or make .check/modules/rtm
-
Force make to run, even if package has not changed – make -B <command>
-
Run make commands in parallel – make -j<ncores>; e.g. make -j4 install to install packages using four parallel processes. Note that packages containing compiled code (e.g. PEcAn.RTM, PEcAn.BASGRA) might fail when j is greater than 1, because of limitations in the way R calls make internally while compiling them. See GitHub issue 1976 for more details.
-
-
All instructions for the make build system are contained in the Makefile in the PEcAn root directory.
-For full documentation on make, see the man pages by running man make from a terminal.
-
-
-
-
-
8.2 Git and GitHub Workflow
-
Using Git
-
-
-
8.2.1 Using Git
-
This document describes the steps required to download PEcAn, make changes to code, and submit your changes.
-
-
If you are new to GitHub or to PEcAn, start with the one-time set-up instructions under Before any work is done. Also see the excellent tutorials and references in the Git) section right below this list and at the bootom in References.
-
To make trivial changes, see [Quick and Easy].
-
To make a few changes to the code, start with the [Basic Workflow].
-
To make substantial changes and/or if plan to contribute over time see [Recommended Workflow: A new branch for each change].
-
-
-
8.2.1.1 Git
-
Git is a free & open source, distributed version control system designed
-to handle everything from small to very large projects with speed and
-efficiency. Every Git clone is a full-fledged repository with complete
-history and full revision tracking capabilities, not dependent on
-network access or a central server. Branching and merging are fast and
-easy to do.
URLs In the rest of the document will use specific URL’s to clone the code.
-There a few URL’s you can use to clone a project, using https, ssh and
-git. You can use either https or git to clone a repository and write to
-it. The git protocol is read-only.
-This document describes the steps required to download PEcAn, make changes to code, and submit your changes.
-
If during above process you want to work on something else, commit all
-your code, create a new branch, and work on new branch.
These instructions apply to other repositories too.
-
-
-
8.2.1.3 PEcAn Project Branches
-
We follow branch organization laid out on this page.
-
In short, there are three main branches you must be aware of:
-
-
develop - Main Branch containing the latest code. This is the main branch you will make changes to.
-
master - Branch containing the latest stable code. DO NOT MAKE CHANGES TO THIS BRANCH.
-
release/vX.X.X - Named branches containing code specific to a release. Only make changes to this branch if you are fixing a bug on a release branch.
-
-
-
-
8.2.1.4 Milestones, Issues, Tasks
-
The Milestones, issues, and tasks can be used to organize specific features or research projects. In general, there is a heirarchy:
-
-
milestones (Big picture, “Epic”): contains many issues, organized by release.
-
issues (Specific features / bugs, “Story”): may contain a list of tasks; represent
-
task list (to do list, “Tasks”): list of steps required to close an issue, e.g.:
-
-
-
-
-
* first do this
-
-
-
* then this
-
-
-
* completed when x and y
-
-
-
-
-
-
8.2.1.5 Editing files on GitHub
-
The easiest approach is to use GitHub’s browser based workflow. This is useful when your change is a few lines, if you are editing a wiki, or if the edit is trivial (and won’t break the code). The GitHub documentation is here but it is simple: finding the page or file you want to edit, click “edit” and then the GitHub web application will automatically forking and branch, then allow you to submit a pull request. However, it should be noted that unless you are a member of the PEcAn project that the “edit” button will not be active and you’ll want to follow the workflow described below for forking and then submitting a pull request.
-
-
-
-
8.2.2 Recommended Git Workflow
-
Summary: development should occur on a fork of the main repository.
-
-
Fork
-
Create Branch
-
Develop
-
Push changes to your fork
-
Create pull request from branch on your fork to develop branch on pecanproject/pecan
-
-
Each feature should be in its own branch (for example each issue is a branch, names of branches are often the issue in a bug tracking system).
-
Commit and Push Frequency On your branch, commit any time that you have done work that you do not want to re-do. Remember, pushing changes to your branch is like saving a draft. Submit a pull request when you are done.
-
-
8.2.2.1 Before any work is done
-
The first step below only needs to be done once when you first start working on the PEcAn code. The steps below that need to be done to set up PEcAn on your computer, and would need to be repeated if you move to a new computer.
-
All contributors should create a fork of the PEcAn source code in their own folder see github help: “fork a repo”). This forked repository will allow you to create branches and submit these changes back to GitHub using pull requests to the develop branch of PEcAn.
-
The pull request will start a review process that will eventually result in the code being merged into the main copy of the codebase. See https://help.github.com/articles/fork-a-repo for more information, especially on how to keep your fork up to date with respect to the original. (Rstudio users should also see Git + Rstudio, below).
-
You can setup SSH keys to make it easier to commit cod back to GitHub. This might especially be true if you are working from a cluster, see set up ssh keys
-
There is a script in the scripts folder called scripts/syncgit.sh that will keep your fork in sync with the main pecanproject repository.
Fork PEcAn on GitHub. Go to the PEcAn source code and click on the Fork button in the upper right. This will create a copy of PEcAn in your personal space.
If you have used the instructions above, you can use the helper script called scripts/syncgit.sh to keep the master and develop branches of your own fork in sync with the PEcAnProject/pecan repository.
-
After following the above, your .git/config file will include the following:
You can reference and close issues from comments, pull requests, and commit messages. This should be done when you commit code that is related to or will close/fix an existing issue.
-
There are two ways to do this. One easy way is to include the following text in your commit message:
to reference: just the issue number (e.g. “gh-xxx”)
-
-
-
-
-
8.2.3 Useful Git tools
-
-
8.2.3.1 GitHub Desktop
-
The easiest way to get working with GitHub is by installing the GitHub
-client. For instructions for your specific OS and download of the
-GitHub client, see https://help.github.com/articles/set-up-git.
-This will help you set up an SSH key to push code back to GitHub. To
-check out a project you do not need to have an ssh key and you can use
-the https or git url to check out the code.
-
-
-
8.2.3.2 Git + Rstudio
-
Rstudio is nicely integrated with many development tools, including git and GitHub.
-It is quite easy to check out source code from within the Rstudio program or browser.
-The Rstudio documentation includes useful overviews of version control
-and R package development.
-
Once you have git installed on your computer (see the Rstudio version control documentation for instructions), you can use the following steps to install the PEcAn source code in Rstudio.
-
-
-
-
8.2.4 Advanced
-
-
8.2.4.1 Fixing a release Branch
-
If you would like to make changes to a release branch, you must follow a different workflow, as the release branch will not contain the latest code on develop and must remain seperate.
Lightweight tags are useful, but here we discuss the annotated tags that are used for marking stable versions, major releases, and versions associated with published results.
-
The basic command is git tag. The -a flag means ‘annotated’ and -m is used before a message. Here is an example:
-
git tag -a v0.6 -m "stable version with foo and bar features, used in the foobar publication by Bob"
-
Adding a tag to the a remote repository must be done explicitly with a push, e.g.
-
git push v0.6
-
To use a tagged version, just checkout:
-
git checkout v0.6
-
To tag an earlier commit, just append the commit SHA to the command, e.g.
-
git tag -a v0.99 -m "last version before 1.0" 9fceb02
In this section, development topics are introduced and discussed. PEcAn code lives within the If you are looking for an issue to work on, take a look through issues labled “good first issue”. To get started you will want to review
-
We use GitHub to track development.
-
To learn about GitHub, it is worth taking some time to read through the FAQ. When in doubt, the first step is to click the “Help” button at the top of the page.
-
-
To address specific people, use a github feature called @mentions e.g. write @dlebauer, @robkooper, @mdietze, or @serbinsh … in the issue to alert the user as described in the GitHub documentation on notifications
-
-
-
8.2.6.1 Bugs, Issues, Features, etc.
-
-
-
8.2.6.2 Reporting a bug
-
-
(For developers) work through debugging.
-
Once you have identified a problem, that you can not resolve, you can write a bug report
-
Write a bug report
-
submit the bug report
-
If you do find the answer, explain the resolution (in the issue) and close the issue
-
-
-
-
8.2.6.3 Required content
-
Note:
-
-
a bug is only a bug if it is reproducible
-
clear bug reports save time
-
-
-
Clear, specific title
-
Description -
-
-
-
What you did
-
What you expected to happen
-
What actually happened
-
What does work, under what conditions does it fail?
-
Reproduction steps - minimum steps required to reproduce the bug
-
-
-
additional materials that could help identify the cause:
-
-
-
screen shots
-
stack traces, logs, scripts, output
-
specific code and data / settings / configuration files required to reproduce the bug
-
environment (operating system, browser, hardware)
-
-
-
-
8.2.6.4 Requesting a feature
-
(from The Pragmatic Programmer, available as
-ebook
-through UI libraries, hardcopy on David’s bookshelf)
-
-
-
focus on “user stories”, e.g. specific use cases
-
Be as specific as possible,
-
Here is an example:
-
-
-
Bob is at www.mysite.edu/maps
-
map of the the region (based on user location, e.g. US, Asia, etc)
-
option to “use current location” is provided, if clicked, map zooms in to, e.g. state or county level
-
for site run:
-
-
option to select existing site or specify point by lat/lon
-
option to specify a bounding box and grid resolution in
-either lat/lon or polar stereographic.
-
-
asked to specify start and end times in terms of year, month, day, hour, minute. Time is recorded in UTC not local time, this should be indicated.
-
-
-
-
8.2.6.5 Closing an issue
-
-
Definition of “Done”
-
-
-
test
-
documentation
-
-
-
when issue is resolved:
-
-
-
status is changed to “resolved”
-
assignee is changed to original author
-
-
-
if original author agrees that issue has been resolved
-
-
-
original author changes status to “closed”
-
-
-
except for trivial issues, issues are only closed by the author
-
-
-
-
8.2.6.6 When to submit an issue?
-
Ideally, non-trivial code changes will be linked to an issue and a commit.
-
This requires creating issues for each task, making small commits, and referencing the issue within your commit message. Issues can be created on GitHub. These issues can be linked to commits by adding text such as fixes gh-5).
-
Rationale: This workflow is a small upfront investment that reduces error and time spent re-creating and debugging errors. Associating issues and commits, makes it easier to identify why a change was made, and potential bugs that could arise when the code is changed. In addition, knowing which issue you are working on clarifies the scope and objectives of your current task.
-
-
-
-
-
-
8.3 Coding Practices
-
-
-
8.3.1 Coding Style
-
Consistent coding style improves readability and reduces errors in
-shared code.
-
Unless otherwise noted, PEcAn follows the Tidyverse style guide, so please familiarize yourself with it before contributing.
-In addition, note the following:
-
-
Document all functions using roxygen2.
-See Roxygen2 for more details.
-
Put your name on things.
-Any function that you create or make a meaningful contribution to should have your name listed after the author tag in the function documentation.
-It is also often a good idea to add your name to extended comments describing particularly complex or strange code.
-
Write unit tests with testthat.
-Tests are a complement to documentation - they define what a function is (and is not) expected to do.
-Not all functions necessarily need unit tests, but the more tests we have, the more confident we can be that changes don’t break existing code.
-Whenever you discover and fix a bug, it is a good idea to write a unit test that makes sure the same bug won’t happen again.
-See Unit_Testing for instructions, and Advanced R: Tests.
-
Do not use abbreviations.
-Always write out TRUE and FALSE (i.e. do not use T or F).
-Do not rely on partial argument matching – write out all arguments in full.
-
Avoid dots in function names.
-R’s S3 methods system uses dots to denote object methods (e.g. print.matrix is the print method for objects of class matrix), which can cause confusion.
-Use underscores instead (e.g. do_analysis instead of do.analysis).
-(NOTE that many old PEcAn functions violate this convention. The plan is to deprecate those in PEcAn 2.0. See GitHub issue #392).
-
Use informative file names with consistent extensions.
-Standard file extensions are .R for R scripts, .rds for individual objects (via saveRDS function), and .RData (note: capital D!) for multiple objects (via the save function).
-For function source code, prefer multiple files with fewer functions in each to large files with lots of files (though it may be a good idea to group closely related functions in a single file).
-File names should match, or at least closely reflect, their files (e.g. function do_analysis should be defined in a file called do_analysis.R).
-Do not use spaces in file names – use dashes (-) or underscores (_).
-
For using external packages, add the package to Imports: and call the corresponding function with package::function.
-Do not use @importFrom package function or, worse yet, @import package.
-(The exception is infix operators like magrittr::%>% or ggplot2::%+%, which can be imported via roxygen2 documentation like @importFrom magrittr %>%).
-Do not add packages to Depends.
-In general, try to avoid adding new dependencies (especially ones that depend on system libraries) unless they are necessary or already widely used in PEcAn (e.g. GDAL, NetCDF, XML, JAGS, dplyr).
-For a more thorough and nuanced discussion, see the package dependencies appendix.
-
-
-
-
-
8.3.2 Logging
-
During development we often add many print statements to check to see how the code is doing, what is happening, what intermediate results there are etc. When done with the development it would be nice to turn this additional code off, but have the ability to quickly turn it back on if we discover a problem. This is where logging comes into play. Logging allows us to use “rules” to say what information should be shown. For example when I am working on the code to create graphs, I do not have to see any debugging information about the SQL command being sent, however trying to figure out what goes wrong during a SQL statement it would be nice to show the SQL statements without adding any additional code.
-
PEcAn provides a set of logger.* functions that should be used in place of base R’s stop, warn, print, and similar functions. The logger functions make it easier to print to a system log file, and to control the level of output produced by PEcAn.
-
-
The file test.logger.R provides descriptive examples
Logger functions and their corresponding levels (in order of increasing level):
-
logger.debug ("DEBUG") – Low-level diagnostic messages that are hidden by default. Good examples of this are expanded file paths and raw results from database queries or other analyses.
-
logger.info ("INFO") – Informational messages that regular users may want to see, but which do not indicate anything unexpected. Good examples of this are progress updates updates for long-running processes, or brief summaries of queries or analyses.
-
logger.warn ("WARN") – Warning messages about issues that may lead to unexpected but valid results. Good examples of this are interactions between arguments that lead to some arguments being ignored or removal of missing or extreme values.
-
logger.error ("ERROR") – Error messages from which PEcAn has some capacity to recover. Unless you have a very good reason, we recommend avoiding this in favor of either logger.severe to actually stop execution or logger.warn to more explicitly indicate that the problem is not fatal.
-
logger.severe – Catastrophic errors that warrant immediate termination of the workflow. This is the only function that actually stops R’s execution (via stop).
-
The logger.setLevel function sets the level at which a message will be printed. For instance, logger.setLevel("WARN") will suppress logger.info and logger.debug messages, but will print logger.warn and logger.error messages. logger.setLevel("OFF") suppresses all logger messages.
-
To print all messages to console, use logger.setUseConsole(TRUE)
-
-
-
-
-
8.3.3 Package Data
-
-
8.3.3.1 Summary:
-
Files with the following extensions will be read by R as data:
-
-
plain R code in .R and .r files are sourced using source()
-
text tables in .tab, .txt, .csv files are read using read()
-** objects in R image files: .RData, .rda are loaded using load()
-
capitalization matters
-
all objects in foo.RData are loaded into environment
easy way: by adding the following line to the package DESCRIPTION:
-note: this should be used with caution or it can cause difficulty as discussed in redmine issue #1118
Currently, a limited number of data formats can be accessed using the data function by placing one of the following filetypes in a packages’ data directory:
-* files ending .R or .r are source()d in, with the R working
-directory changed temporarily to the directory containing the respective
-file. (data ensures that the utils package is attached, in case it
-had been run viautils::data.)
-* files ending .RData or .rda are load()ed.
-* files ending .tab, .txt or .TXT are read using read.table(..., header = TRUE), and hence result in a data frame.
-* files ending .csv or .CSV are read using read.table(..., header = TRUE, sep = ';'), and also result in a data frame.
-
If your data does not fall in those 4 categories, or you can use the
-system.file function to get access to the data:
The arguments are folder, filename(s) and then package. It will return
-the fully qualified path name to a file in a package, in this case it
-points to the trait data. This is almost the same as the data function,
-however we can now use any function to read the file, such as read.csv
-instead of read.csv2 which seems to be the default of data. This also
-allows us to store arbitrary files in the data folder, such as the the
-bug file and load it when we need it.
To update documentation throughout PEcAn, run make document in the PEcAn root directory.
-Make sure you do this before opening a pull request –
-PEcAn’s automated testing (Travis) will check if any documentation is out of date and will throw an error like the following if it is:
-
These files were changed by the build process:
-{...}
Unit tests are short (<1 minute runtime) tests of functionality of specific functions.
-Ideally, every function should have at least one unit test associated with it.
-
A unit test should be written for each of the following situations:
-
-
Each bug should get a regression test.
-
-
-
The first step in handling a bug is to write code that reproduces the error
-
This code becomes the test
-
most important when error could re-appear
-
essential when error silently produces invalid results
-
-
-
Every time a (non-trivial) function is created or edited
-
-
-
Write tests that indicate how the function should perform
-
-
example: expect_equal(sum(1,1), 2) indicates that the sum
-function should take the sum of its arguments
-
-
Write tests for cases under which the function should throw an
-error
Any functionality that you would like to protect over the long term. Functionality that is not tested is more likely to be lost.
-PEcAn uses the testthat package for unit testing.
-A general overview of is provided in the “Testing” chapter of Hadley Wickham’s book “R packages”.
-Another useful resource is the testthatpackage documentation website.
-See also our testthat appendix.
-Below is a lightning introduction to unit testing with testthat.
-
-
Each package’s unit tests live in .R scripts in the folder <package>/tests/testthat.
-In addition, a testthat-enabled package has a file called <packagename>/tests/testthat.R with the following contents:
Integration tests consist of running the PEcAn workflow in full.
-One way to do integration tests is to manually run workflows for a given version of PEcAn, either through the web interface or by manually creating a pecan.xml file.
-Such manual tests are an important part of checking PEcAn functionality.
-
Alternatively, the base/workflow/inst/batch_run.R script can be used to quickly run a series of user-specified integration tests without having to create a bunch of XML files.
-This script is powered by the PEcAn.workflow::create_execute_test_xml() function,
-which takes as input information about the model, meteorology driver, site ID, run dates, and others,
-uses these to construct a PEcAn XML file,
-and then uses the system() command to run a workflow with that XML.
-
If run without arguments, batch_run.R will try to run the model configurations specified in the base/workflow/inst/default_tests.csv file.
-This file contains a CSV table with the following columns:
-
-
model – The name of the model (models.model_name column in BETY)
-
revision – The version of the model (models.revision column in BETY)
-
met – The name of the meteorology driver source
-
site_id – The numeric site ID for the model run (sites.site_id)
-
pft – The name of the plant functional type to run. If NA, the script will use the first PFT associated with the model.
-
start_date, end_date – The start and end dates for the model run, respectively. These should be formatted according to ISO standard (YYYY-MM-DD, e.g. 2010-03-16)
-
sensitivity – Whether or not to run the sensitivity analysis. TRUE means run it, FALSE means do not.
-
ensemble_size – The number of ensemble members to run. Set this to 1 to do a single run at the trait median.
-
comment – An string providing some user-friendly information about the run.
-
-
The batch_run.R script will run a workflow for every row in the input table, sequentially (for now; eventually, it will try to run them in parallel),
-and at the end of each workflow, will perform some basic checks, including whether or not the workflow finished and if the model produced any output.
-These results are summarized in a CSV table (by default, a file called test_result_table.csv), with all of the columns as the input test CSV plus the following:
-
-
outdir – Absolute path to the workflow directory.
-
workflow_complete – Whether or not the PEcAn workflow completed. Note that this is a relatively low bar – PEcAn workflows can complete without having run the model or finished some other steps.
-
has_jobsh – Whether or not PEcAn was able to write the model’s job.sh script. This is a good indication of whether or not the model’s write.configs step was successful, and may be useful for separating model configuration errors from model execution errors.
-
model_output_raw – Whether or not the model produced any output files at all. This is just a check to see of the <workflow>/out directory is empty or not. Note that some models may produce logfiles or similar artifacts as soon as they are executed, whether or not they ran even a single timestep, so this is not an indication of model success.
-
model_output_processed – Whether or not PEcAn was able to post-process any model output. This test just sees if there are any files of the form YYYY.nc (e.g. 1992.nc) in the <workflow>/out directory.
-
-
Right now, these checks are not particularly robust or comprehensive, but they should be sufficient for catching common errors.
-Development of more, better tests is ongoing.
-
The batch_run.R script can take the following command-line arguments:
-
-
--help – Prints a help message about the script’s arguments
-
--dbfiles=<path> – The path to the PEcAn dbfiles folder. The default value is ~/output/dbfiles, based on the file structure of the PEcAn VM. Note that for this and all other paths, if a relative path is given, it is assumed to be relative to the current working directory, i.e. the directory from which the script was called.
-
--table=<path> – Path to an alternate test table. The default is the base/workflow/inst/default_tests.csv file. See preceding paragraph for a description of the format.
-
--userid=<id> – The numeric user ID for registering the workflow. The default value is 99000000002, corresponding to the guest user on the PEcAn VM.
-
--outdir=<path> – Path to a directory (which will be created if it doesn’t exist) for storing the PEcAn workflow outputs. Default is batch_test_output (in the current working directory).
-
--pecandir=<path> – Path to the PEcAn source code root directory. Default is the current working directory.
-
--outfile=<path> – Full path (including file name) of the CSV file summarizing the results of the runs. Default is test_result_table.csv. The format of the output
-
-
-
-
8.4.3 Continuous Integration
-
Every time anyone commits a change to the PEcAn code, the act of pushing to GitHub triggers an automated build and test of the full PEcAn codebase, and all pull requests must report a successful CI build before they will be merged. This will sometimes feel like a burden when the build breaks on an issue that looks trivial, but anything that breaks the build is important enough to fix. It’s much better to find errors early and fix them before they get incorporated into the released PEcAn code.
-
At this writing PEcAn’s CI builds primarily use Travis CI and the rest of this section assumes a Travis build, but as of May 2020 we also have an experimental GitHub Actions build, and if we switch completely to GitHub Actions then this guide will need to be rewritten.
-
All our Travis builds run on the same version of Linux (currently Ubuntu 16.04, xenial) using four different versions of R in parallel: the two most recent previous releases, current release, and nightly builds of the R development branch. In most cases the build should pass on all four versions, but only the current release (R:release) is truly mandatory. We aim to fix errors found on R:release before merging, errors found on R:devel before the next R release, and errors found on older releases as developer time and forward compatibility allow.
-
Each build starts by launching a separate clean virtual machine for each R version and performs roughly the following actions on all of them:
-
-
Installs binary system dependencies needed by PEcAn (NetCDF and HDF5 libraries, JAGS, udunits, etc).
-
Installs all the R packages that are declared as dependencies in any PEcAn package, as computed by scripts/generate_dependencies.R.
-
Clones the PEcAn repository from GitHub, and checks out the branch to be tested.
-
Retrieves any cached files available from previous Travis builds.
-
-
The main thing in the cache is previously-installed dependency R packages, to avoid recompiling them every time.
-
If the cache becomes stale or is preventing a package update needed by the build (e.g. to get a new version that contains a needed bug fix), delete the cache through the Travis web interface and it will be reconstructed on the next build.
-
Because PEcAn has so many dependencies, builds with no cache will spend most of their time recompiling packages and will probably run out of time before the tests complete. You can fix this by using scripts/travis/cache_buildup.sh and scripts/travis/prime_travis_cache.sh to build up the cache incrementally through one or more custom builds, each of which installs some dependencies and then uploads a freshened cache without running any tests. Once all dependencies have been cached, restart the standard full build.
-
-
Initializes a skeleton version of the PEcAn database (BeTY) containing a few public records to be used by the test runs.
-
Installs all the PEcAn packages, recompiling documentation and installing dependencies as needed.
-
Runs package unit tests (the same ones you run locally with make test or devtools::test(pkgname)).
-
-
As discussed in Unit testing, these tests should run quickly and test individual components in relative isolation.
-
Any test that calls the skip_on_ci function will be skipped. This is useful for tests that need to run for a very long time (e.g. large data product downloads) or require resources that aren’t available on Travis (e.g. specific models), but be sure to run these tests locally before pushing your code!
-
-
Runs R package checks (the same ones you run locally with make check or devtools::check(pkgname)), skipping tests and documentation rebuild because we just did those in the previous steps.
-
-
Any ERROR in the check output will stop the build immediately.
-
If there are no ERRORs, any WARNINGs or NOTEs are compared against a stored historic check result in <package>/tests/Rcheck_reference.log. If the package has no stored reference result, all WARNINGs and NOTEs are considered newly added and reported as build failures.
-
If all messages from the current built were also present in the reference result, the check passes. If any messages are newly added, a build failure is reported.
-
Each line of the check log is considered a separate message, and the test requires exact matching, so a change from Undocumented arguments in documentation object 'foo': 'x' to Undocumented arguments in documentation object 'foo': 'x', 'y' will be counted as a new warning… and you should fix both of them while you’re at it!
-
The idea here is to enforce good coding practice and catch likely errors in all new code while recognizing that we have a lot of legacy code whose warnings need to be fixed as we have time rather than all at once.
-
As we fix historic warnings, we will revoke their grandfathered status by removing them from the stored check results, so that they will break the build if they reappear.
-
If your PR reports a failure in pre-existing code that you think ought to be grandfathered, please fix it as part of your PR anyway. It’s frustrating to see tests complain about code you didn’t touch, but the failures all need to be cleaned up eventually and it’s likely easier to fix the error than to figure out how to re-ignore it.
-
-
Runs a simple PEcAn workflow from beginning to end (three years of SIPNET at Niwot Ridge using stored weather data, meta-analysis on fixed effects only, sensitivity analysis on NPP), and verifies that the models complete successfully.
-
Checks whether any version-controlled files have been changed by the build and testing process, and marks a build failure if they have.
-
-
If your build fails at this step, the most common cause is that you forgot to Roxygenize before committing.
-
This step will also detect newly added files, e.g. tests improperly writing to the current working directory rather than tempdir() and then failing to clean up after themselves.
-
-
-
If any of these steps reports an error, the build is marked as “broken” and stops before the next step. If they all pass, the Travis CI bot marks the build as successful and tells the GitHub bot that it’s OK to allow your changes to be merged… but the final decision belongs to the human reviewing your code and they might still ask you for other changes!
-
After a successful build, Travis performs two post-build steps:
-
-
Compiles the PEcAn documentation book (book_source) and the tutorials (documentation/tutorials) and uploads them to the PEcAn website.
-
-
This is only done for commits to the master or develop branch, so changes to in-progress pull requests never change the live documentation until after they are merged.
-
-
Packs up selected build artifacts into a cache file and uploads it to the Travis servers for use on the next build.
-
-
The post-build steps are allowed to fail without breaking the build. If you made documentation changes but don’t see them deployed, or if your build seems to be reinstalling packages that ought to be cached, inspect the Travis logs of the previous supposedly-successful build to see if their uploads succeeded.
-
All of the above descriptions apply to the build Travis generates when you push to the main PecanProject/pecan repository, either by directly pushing to a branch or by opening a pull request. If you like, you can also enable Travis builds from your own PEcAn fork. This can be useful for several reasons:
-
-
It lets your test whether your changes worked before you open a pull request.
-
It often lets you get faster test results: The PEcAn project uses Travis CI’s free tier, which only allows a few simultaneous build jobs per repository. If many people are pushing code at the same time, your build might wait in line for a long time at PecanProject/pecan but start immediately at yourname/pecan.
-
If you will be editing the documentation a lot and want to see rendered previews of your in-progress work (instead of waiting until it is merged into develop), you can clone the pecan-documentation repository to your own GitHub account and let Travis update it for you.
NB: pecan.xml is configured for the virtual machine, you will need to change the field from ‘/home/carya/’ to wherever you installed your ‘sites’, usually $HOME
-
-
-
-
-
8.6 Directory structure
-
-
8.6.1 Overview of PEcAn repository as of PEcAn 1.5.3
-
-
pecan/
- +- base/ # Core functions
- +- all # Dummy package to load all PEcAn R packages
- +- db # Modules for querying the database
- +- logger # Report warnings without killing workflows
- +- qaqc # Model skill testing and integration testing
- +- remote # Communicate with and execute models on local and remote hosts
- +- settings # Functions to read and manipulate PEcAn settings files
- +- utils # Misc. utility functions
- +- visualization # Advanced PEcAn visualization module
- +- workflow # functions to coordinate analysis steps
- +- book_source/ # Main documentation and developer's guide
- +- CHANGELOG.md # Running log of changes in each version of PEcAn
- +- docker/ # Experimental replacement for PEcAn virtual machine
- +- documentation # index_vm.html, references, other misc.
- +- models/ # Wrappers to run models within PEcAn
- +- ed/ # Wrapper scripts for running ED within PEcAn
- +- sipnet/ # Wrapper scripts for running SIPNET within PEcAn
- +- ... # Wrapper scripts for running [...] within PEcAn
- +- template/ # Sample wrappers to copy and modify when adding a new model
- +- modules # Core modules
- +- allometry
- +- data.atmosphere
- +- data.hydrology
- +- data.land
- +- meta.analysis
- +- priors
- +- rtm
- +- uncertainty
- +- ...
- +- scripts # R and Shell scripts for use with PEcAn
- +- shiny/ # Interactive visualization of model results
- +- tests/ # Settings files for host-specific integration tests
- +- web # Main PEcAn website files
-
-
-
8.6.2 Generic R package structure:
-
see the R development wiki for more information on writing code and adding data.
-
+- DESCRIPTION # short description of the PEcAn library
- +- R/ # location of R source code
- +- man/ # Documentation (automatically compiled by Roxygen)
- +- inst/ # files to be installed with package that aren't R functions
- +- extdata/ # misc. data files (in misc. formats)
- +- data/ # data used in testing and examples (saved as *.RData or *.rda files)
- +- NAMESPACE # declaration of package imports and exports (automatically compiled by Roxygen)
- +- tests/ # PEcAn testing scripts
- +- testthat/ # nearly all tests should use the testthat framework and live here
For a useful analogy for Docker containerization, we refer you to the webcomic xkcd.
-
Docker is a technology for encapsulating software in “containers”, somewhat similarly to virtual machines.
-Like virtual machines, Docker containers facilitate software distribution by bundling the software with all of its dependencies in a single location.
-Unlike virtual machines, Docker containers are meant to only run a single service or process and are build on top of existing services provided by the host OS (such as disk access, networking, memory management etc.).
-
In Docker, an image refers to a binary snapshot of a piece of software and all of its dependencies.
-A container refers to a running instance of a particular image.
-A good rule of thumb is that each container should be responsible for no more than one running process.
-A software stack refers to a collection of containers, each responsible for its own process, working together to power a particular application.
-Docker makes it easy to run multiple software stacks at the same time in parallel on the same machine.
-Stacks can be given a unique name, which is passed along as a prefix to all their containers.
-Inside these stacks, containers can communicate using generic names not prefixed with the stack name, making it easy to deploy multiple stacks with the same internal configuration.
-Containers within the same stack communicate with each other via a common network.
-Like virtual machines or system processes, Docker stacks can also be instructed to open specific ports to facilitate communication with the host and other machines.
-
The PEcAn database BETY provides an instructive case-study.
-BETY is comprised of two core processes – a PostgreSQL database, and a web-based front-end to that database (Apache web server with Ruby on Rails).
-Running BETY as a “Dockerized” application therefore involves two containers – one for the PostgreSQL database, and one for the web server.
-We could build these containers ourselves by starting from a container with nothing but the essentials of a particular operating system, but we can save some time and effort by starting with an existing image for PostgreSQL from Docker Hub.
-When starting a Dockerized BETY, we start the PostgreSQL container first, then start the BETY container telling it how to communicate with the PostgreSQL container.
-To upgrade an existing BETY instance, we stop the BETY container, download the latest version, tell it to upgrade the database, and re-start the BETY container.
-There is no need to install new dependencies for BETY since they are all shipped as part of the container.
-
The PEcAn Docker architecture is designed to facilitate installation and maintenance on a variety of systems by eliminating the need to install and maintain complex system dependencies (such as PostgreSQL, Apache web server, and Shiny server).
-Furthermore, using separate Docker containers for each ecosystem model helps avoid clashes between different software version requirements of different models (e.g. some models require GCC <5.0, while others may require GCC >=5.0).
-
The full PEcAn Docker stack is described in more detail in the next section.
-
-
-
24.1.2 Working with Docker
-
To run an image, you can use the Docker command line interface.
-For example, the following runs a PostgreSQL image based on the pre-existing PostGIS image by mdillon:
This will start the PostgreSQL+PostGIS container.
-The following options were used:
-
-
--detach makes the container run in the background.
-
--rm removes the container when it is finished (make sure to use the volume below).
-
--name the name of the container, also the hostname of the container which can be used by other docker containers in the same network inside docker.
-
--network pecan the network that the container should be running in, this leverages of network isolation in docker and allows this container to be connected to by others using the postgresql hostname.
-
--publish exposes the port to the outside world, this is like ssh, and maps port 9876 to port 5432 in the docker container
-
--volume maps a folder on your local machine to the machine in the container. This allows you to save data on your local machine.
-
mdillon/postgis:9.6-alpine is the actual image that will be run, in this case it comes from the group/person mdillon, the container is postgis and the version 9.6-alpine (version 9.6 build on alpine linux).
-
-
Other options that might be used:
-
-
--tty allocate a pseudo-TTY to send stdout and stderr back to the console.
-
--interactive keeps stdin open so the user can interact with the application running.
-
--env sets environment variables, these are often used to change the behavior of the docker container.
-
-
To see a list of all running containers you can use the following command:
To see the log files of this container you use the following command (you can either use their name or id as returned by docker ps). The -f flag will follow the stdout/stderr from the container, use Ctrl-C to stop following the stdout/stderr.
Containers that are running in the foreground (without the --detach) can be stopped by pressing Ctrl-C. Any containers running in the background (with --detach) will continue running until the machine is restarted or the container is stopped using docker stop.
-
-
-
24.1.3docker-compose
-
For a quick introduction to docker-compose, we recommend the following YouTube video: Docker Compose in 12 Minutes.
docker-compose provides a convenient way to configure and run a multi-container Docker stack.
-Basically, a docker-compose setup consists of a list of containers and their configuration parameters, which are then internally converted into a bunch of docker commands.
-To configure BETY as described above, we can use a docker-compose.yml file like the following:
This simple file allows us to bring up a full BETY application with both database and BETY application. The BETY app will not be brought up until the database container has started.
-
You can now start this application by changing into the same directory as the docker-compose.yml file (cd /path/to/file) and then running:
-
docker-compose up
-
This will start the application, and you will see the log files for the 2 different containers.
This section will let you download some configuration files. The documentation provides links to the latest released version (master branch in GitHub) or the develop version that we are working on (develop branch in GitHub) which will become the next release. If you cloned the PEcAn GitHub repository you can use git checkout <branch> to switch branches.
-
The PEcAn Docker stack is configured using a docker-compose.yml file. You can download just this file directly from GitHub latest or develop. You can also find this file in the root of cloned PEcAn GitHub repository. There is no need to edit the docker-compose.yml file. You can use either the .env file to change some of the settings, or the docker-compose.override.yml file to modify the docker-compose.yml file. This makes it easier for you to get an updated version of the docker-compose.yml file and not lose any chances you have made to it.
-
Some of the settings in the docker-compose.yml can be set using a .env file. You can download either the latest or the develop version. If you have cloned the GitHub repository it is also located in the docker folder. This file should be called .env and be placed in the same folder as your docker-compose.yml file. This file will allow you to set which version of PEcAn or BETY to use. See the comments in this file to control the settings. Option you might want to set are:
-
-
PECAN_VERSION : The docker images to use for PEcAn. The default is latest which is the latest released version of PEcAn. Setting this to develop will result in using the version of PEcAn which will become the next release.
-
PECAN_FQDN : Is the name of the server where PEcAn is running. This is what is used to register all files generated by this version of PEcAn (see also TRAEFIK_HOST).
-
PECAN_NAME : A short name of this PEcAn server that is shown in the pull down menu and might be easier to recognize.
-
BETY_VERSION : This controls the version of BETY. The default is latest which is the latest released version of BETY. Setting this to develop will result in using the version of BETY which will become the next release.
-
TRAEFIK_HOST : Should be the FQDN of the server, this is needed when generating a SSL certificate. For SSL certificates you will need to set TRAEFIK_ACME_ENABLE as well as TRAEFIK_ACME_EMAIL.
-
TRAEFIK_IPFILTER : is used to limit access to certain resources, such as RabbitMQ and the Traefik dashboard.
-
-
A final file, which is optional, is a docker-compose.override.yml. You can download a version for the latest and develop versions. If you have cloned the GitHub repository it is located in the docker folder. Use this file as an example of what you can do, only copy the pieces over that you really need. This will allow you to make changes to the docker-compose file for your local installation. You can use this to add additional containers to your stack, change the path where docker stores the data for the containers, or you can use this to open up the postgresql port.
Once you have the docker-compose.yml file as well as the optional .env and docker-compose.override.yml in a folder you can start the PEcAn stack. The following instructions assume you are in the same directory as the file (if not, cd into it).
-
In the rest of this section we will use a few arguments for the docker-compose application. The location of these arguments are important. The general syntax of docker-compose is docker-compose <ARGUMENTS FOR DOCKER COMPOSE> <COMMAND> <ARGUMENTS FOR COMMAND> [SERVICES]. More generally, docker-compose options are very sensitive to their location relative to other commands in the same line – that is, docker-compose -f /my/docker-compose.yml -p pecan up -d postgres is not the same as docker-compose -d postgres -p pecan up -f /my/docker-compose.yml. If expected ever don’t seem to be working, check that the arguments are in the right order.)
-
-
-f <filename> : ARGUMENTS FOR DOCKER COMPOSE : Allows you to specify a docker-compose.yml file explicitly. You can use this argument multiple times. Default is to use the docker-compose.yml and docker-compose.override.yml in your current folder.
-
-p <projectname> : ARGUMENTS FOR DOCKER COMPOSE : Project name, all volumes, networks, and containers will be prefixed with this argument. The default value is to use the current folder name.
-
-d : ARGUMENTS FOR up COMMAND : Will start all the containers in the background and return back to the command shell.
-
-
If no services as added to the docker-compose command all services possible will be started.
-
-
-
24.3.2 Initialize PEcAn (first time only)
-
Before you can start to use PEcAn for the first time you will need to initialize the database (and optionally add some data). The following two sections will first initialize the database and secondly add some data to the system.
-
-
24.3.2.1 Initialize the PEcAn database
-
The commands described in this section will set up the PEcAn database (BETY) and pre-load it with some common “default” data.
-p pecan – This tells docker-compose to do all of this as part of a “project” -p we’ll call pecan. By default, the project name is set to the name of the current working directory. The project name will be used as a prefix to all containers started by this docker-compose instance (so, if we have a service called postgres, this will create a container called pecan_postgres).
-
up -d – up is a command that initializes the containers. Initialization involves downloading and building the target containers and any containers they depend on, and then running them. Normally, this happens in the foreground, printing logs directly to stderr/stdout (meaning you would have to interrupt it with Ctrl-C), but the -d flag forces this to happen more quietly and in the background.
-
postgres – This indicates that we only want to initialize the service called postgres (and its dependencies). If we omitted this, docker-compose would initialize all containers in the stack.
-
-
The end result of this command is to initialize a “blank” PostGIS container that will run in the background.
-This container is not connected to any data (yet), and is basically analogous to just installing and starting PostgreSQL to your system.
-As a side effect, the above command will also create blank data “volumes” and a “network” that containers will use to communicate with each other.
-Because our project is called pecan and docker-compose.yml describes a network called pecan, the resulting network is called pecan_pecan.
-This is relevant to the following commands, which will actually initialize and populate the BETY database.
-
Assuming the above has run successfully, next run the following:
The breakdown of this command is as follows: {#docker-run-init}
-
-
docker run – This says we will be running a container.
-
--rm – This automatically removes the resulting container once the specified command exits, as well as any volumes associated with the container. This is useful as a general “clean-up” flag for one-off commands (like this one) to make sure you don’t leave any “zombie” containers or volumes around at the end.
-
--network pecan_pecan – Thsi will start the container in the same network space as the posgres container, allowing it to push data into the database.
-
pecan/db – This is the name of the container, this holds a copy of the database used to initialize the postgresql database.
-
-
Note that this command may throw a bunch of errors related to functions and/or operators already existing.
-This is normal – it just means that the PostGIS extension to PostgreSQL is already installed.
-The important thing is that you see output near the end like:
If you do not see this output, you can look at the troubleshooting section at the end of this section for some troubleshooting tips, as well as some solutions to common problems.
-
Once the command has finished successfully, proceed with the next step which will load some initial data into the database and place the data in the docker volumes.
-
-
-
24.3.2.2 Add first user to PEcAn database
-
You can add an initial user to the BETY database, for example the following commands will add the guestuser account as well as the demo carya account:
-
# guest user
-docker-compose run --rm bety user guestuser guestuser "Guest User" guestuser@example.com 4 4
-
-# example user
-docker-compose run --rm bety user carya illinois "Carya Demo User" carya@example.com 1 1
-
-
-
24.3.2.3 Add example data (first time only)
-
The following command will add some initial data to the PEcAn stack and register the data with the database.
docker run – This says we will be running a specific command inside the target Docker container. See docker run --help and the Docker run reference for more information.
-
-ti – This is actually two flags, -t to allocate a pseudo-tty and -i to keep STDIN open even if detached. -t is necessary to ensure lower-level script commands run correctly. -i makes sure that the command output (stdin) is displayed.
-
--rm – This automatically removes the resulting container once the specified command exits, as well as any volumes associated with the container. This is useful as a general “clean-up” flag for one-off commands (like this one) to make sure you don’t leave any “zombie” containers or volumes around at the end.
-
--network pecan_pecan – This indicates that the container will use the existing pecan_pecan network. This network is what ensures communication between the postgres container (which, recall, is just a PostGIS installation with some data) and the “volumes” where the actual data are persistently stored.
-
pecan/data:develop – This is the name of the image in which to run the specified command, in the form repository/image:version. This is interpreted as follows:
-
-
First, it sees if there are any images called pecan/data:develop available on your local machine. If there are, it uses that one.
-
If that image version is not available locally, it will next try to find the image online. By default, it searches Docker Hub, such that pecan/data gets expanded to the container at https://hub.docker.com/r/pecan/data. For custom repositories, a full name can be given, such as hub.ncsa.illinois.edu/pecan/data:latest.
-
If :version is omitted, Docker assumes :latest. NOTE that while online containers should have a :latest version, not all of them do, and if a :latest version does not exist, Docker will be unable to find the image and will throw an error.
-
-
Everything after the image name (here, pecan/data:develop) is interpreted as an argument to the image’s specified entrypoint.
-
--volume pecan_pecan:/data – This mounts the data from the subsequent container (pecan/data:develop) onto the current project volume, called pecan_pecan (as with the network, the project name pecan is the prefix, and the volume name also happens to be pecan as specified in the docker-compose.yml file).
-
--env FQDN=docker – the Fully Qualified Domain Name, this is the same value as specified in the .env file (for the web, monitor and executor containers). This will link the data files to the name in the machines table in BETY.
-
pecan/data:develop – As above, this is the target image to run. Since there is no argument after the image name, this command will run the default command (CMD) specified for this docker container. In this case, it is the docker/add_data.sh script from the PEcAn repository.
-
-
Under the hood, this container runs the docker/add-data.sh script, which copies a bunch of input files and registers them with the PEcAn database.
-
Successful execution of this command should take some time because it involves copying reasonably large amounts of data and performing a number of database operations.
-
-
-
24.3.2.4 Start PEcAn
-
If you already completed the above steps, you can start the full stack by just running the following:
This will build and start all containers required to run PEcAn.
-With the -d flag, this will run all of these containers quietly in the background, and show a nice architecture diagram with the name and status of each container while they are starting.
-Once this is done you have a working instance of PEcAn.
-
If all of the containers started successfully, you should be able to access the various components from a browser via the following URLs (if you run these commands on a remote machine replace localhost with the actual hostname).
-
-
PEcAn web interface (running models) – http://localhost:8000/pecan/ (NOTE: The trailing backslash is necessary.)
Monitor, service that monitors models and shows all models that are online as well as how many instances are online and the number of jobs waiting. The output is in JSON – http://localhost:8000/monitor/
-
-
-
-
24.3.2.5 Start model runs using curl
-
To test PEcAn you can use the following curl statement, or use the webpage to submit a request (if you run these commands on a remote machine replace localhost with the actual hostname):
To see what is happening behind the scenes you can use look at the log file of the specific docker containers, once of interest are pecan_executor_1 this is the container that will execute a single workflow and pecan_sipnet_1 which executes the sipnet mode. To see the logs you use docker logs pecan_executor_1 Following is an example output:
-
2018-06-13 15:50:37,903 [MainThread ] INFO : pika.adapters.base_connection - Connecting to 172.18.0.2:5672
-2018-06-13 15:50:37,924 [MainThread ] INFO : pika.adapters.blocking_connection - Created channel=1
-2018-06-13 15:50:37,941 [MainThread ] INFO : root - [*] Waiting for messages. To exit press CTRL+C
-2018-06-13 19:44:49,523 [MainThread ] INFO : root - b'{"folder": "/data/workflows/PEcAn_99000000001", "workflowid": "99000000001"}'
-2018-06-13 19:44:49,524 [MainThread ] INFO : root - Starting job in /data/workflows/PEcAn_99000000001.
-2018-06-13 19:45:15,555 [MainThread ] INFO : root - Finished running job.
-
This shows that the executor connects to RabbitMQ, waits for messages. Once it picks up a message it will print the message, and execute the workflow in the folder passed in with the message. Once the workflow (including any model executions) is finished it will print Finished. The log file for pecan_sipnet_1 is very similar, in this case it runs the job.sh in the run folder.
-
To run multiple executors in parallel you can duplicate the executor section in the docker-compose file and just rename it from executor to executor1 and executor2 for example. The same can be done for the models. To make this easier it helps to deploy the containers using Kubernetes allowing to easily scale up and down the containers.
-
-
-
-
24.3.3 Troubleshooting
-
When initializing the database, you will know you have encountered more serious errors if the command exits or hangs with output resembling the following:
-
LINE 1: SELECT count(*) FROM formats WHERE ...
- ^
-Error: Relation `formats` does not exist
-
If the above command fails, you can try to fix things interactively by first opening a shell inside the container…
-
docker run -ti --rm --network pecan_pecan pecan/bety:latest /bin/bash
-
…and then running the following commands, which emulate the functionality of the entrypoint.sh with the initialize argument.
The PEcAn docker architecture consists of many containers (see figure below) that will communicate with each other. The goal of this architecture is to easily expand the PEcAn system by deploying new model containers and registering them with PEcAn. Once this is done the user can now use these new models in their work. The PEcAn framework will setup the configurations for the models, and send a message to the model containers to start execution. Once the execution is finished the PEcAn framework will continue. This is exactly as if the model is running on a HPC machine. Models can be executed in parallel by launching multiple model containers.
-
-As can be seen in the figure the architecture leverages of two standard containers (in orange). The first container is postgresql with postgis (mdillon/postgis) which is used to store the database used by both BETY and PEcAn. The second containers is a messagebus, more specifically RabbitMQ (rabbitmq).
-
The BETY app container (pecan/bety) is the front end to the BETY database and is connected to the postgresql container. A http server can be put in front of this container for SSL termination as well to allow for load balancing (by using multiple BETY app containers).
-
The PEcAn framework containers consist of multiple unique ways to interact with the PEcAn system (none of these containers will have any models installed):
-
-
PEcAn shiny hosts the shiny applications developed and will interact with the database to get all information necessary to display
-
PEcAn rstudio is a rstudio environment with the PEcAn libraries preloaded. This allows for prototyping of new algorithms that can be used as part of the PEcAn framework later.
-
PEcAn web allows the user to create a new PEcAn workflow. The workflow is stored in the database, and the models are executed by the model containers.
-
PEcAn cli will allow the user to give a pecan.xml file that will be executed by the PEcAn framework. The workflow created from the XML file is stored in the database, and the models are executed by the model containers.
-
-
The model containers contain the actual models that are executed as well as small wrappers to make them work in the PEcAn framework. The containers will run the model based on the parameters received from the message bus and convert the outputs back to the standard PEcAn output format. Once the container is finished processing a message it will immediatly get the next message and start processing it.
-
-
-
24.4.2 PEcAn’s docker-compose
-
The PEcAn Docker architecture is described in full by the PEcAn docker-compose.yml file.
-For full docker-compose syntax, see the official documentation.
-
This section describes the top-level structure and each of the services, which are as follows:
There are two ways you can override different values in the docker-compose.yml file. The first method is to create a file called .env that is placed in the same folder as the docker-compose.yml file. This file can override some of configuration variables used by docker-compose. For example the following is an example of the env file
-
# This file will override the configation options in the docker-compose
-# file. Copy this file to the same folder as docker-compose as .env
-
-# ----------------------------------------------------------------------
-# GENERAL CONFIGURATION
-# ----------------------------------------------------------------------
-
-# project name (-p flag for docker-compose)
-#COMPOSE_PROJECT_NAME=pecan
-
-# ----------------------------------------------------------------------
-# TRAEFIK CONFIGURATION
-# ----------------------------------------------------------------------
-
-# hostname of server
-#TRAEFIK_HOST=Host:pecan-docker.ncsa.illinois.edu;
-
-# only allow access from localhost and NCSA
-#TRAEFIK_IPFILTER=172.16.0.0/12, 141.142.0.0/16
-
-# Run traffik on port 80 (http) and port 443 (https)
-#TRAEFIK_HTTP_PORT=80
-#TRAEFIK_HTTPS_PORT=443
-#TRAEFIK_HTTPS_OPTIONS=TLS
-
-# enable SSL cerificate generation
-#TRAEFIK_ACME_ENABLE=true
-
-# Use you real email address here to be notified if cert expires
-#TRAEFIK_ACME_EMAIL=pecanproj@gmail.com
-
-# Always use https, trafic to http is redirected to https
-#TRAEFIK_HTTP_REDIRECT=Redirect.EntryPoint:https
-
-# ----------------------------------------------------------------------
-# PEcAn CONFIGURATION
-# ----------------------------------------------------------------------
-
-# what version of pecan to use
-#PECAN_VERSION=develop
-
-# the fully qualified hostname used for this server
-#PECAN_FQDN=pecan-docker.ncsa.illinois.edu
-
-# short name shown in the menu
-#PECAN_NAME=pecan-docker
-
-# ----------------------------------------------------------------------
-# BETY CONFIGURATION
-# ----------------------------------------------------------------------
-
-# what version of BETY to use
-#BETY_VERSION=develop
-
-# what is our server number, 99=vm, 98=docker
-#BETY_LOCAL_SERVER=98
-
-# secret used to encrypt cookies in BETY
-#BETY_SECRET_KEY=1208q7493e8wfhdsohfo9ewhrfiouaho908ruq30oiewfdjspadosuf08q345uwrasdy98t7q243
-
-# ----------------------------------------------------------------------
-# MINIO CONFIGURATION
-# ----------------------------------------------------------------------
-
-# minio username and password
-#MINIO_ACCESS_KEY=carya
-#MINIO_SECRET_KEY=illinois
-
-# ----------------------------------------------------------------------
-# PORTAINER CONFIGURATION
-# ----------------------------------------------------------------------
-
-# password for portainer admin account
-# use docker run --rm httpd:2.4-alpine htpasswd -nbB admin <password> | cut -d ":" -f 2
-#PORTAINER_PASSWORD=$2y$05$5meDPBtS3NNxyGhBpYceVOxmFhiiC3uY5KEy2m0YRbWghhBr2EVn2
-
-# ----------------------------------------------------------------------
-# RABBITMQ CONFIGURATION
-# ----------------------------------------------------------------------
-
-# RabbitMQ username and password
-#RABBITMQ_DEFAULT_USER=carya
-#RABBITMQ_DEFAULT_PASS=illinois
-
-# create the correct URI with above username and password
-#RABBITMQ_URI=amqp://carya:illinois@rabbitmq/%2F
-
-# ----------------------------------------------------------------------
-# RSTUDIO CONFIGURATION
-# ----------------------------------------------------------------------
-
-# Default RStudio username and password for startup of container
-#PECAN_RSTUDIO_USER=carya
-#PECAN_RSTUDIO_PASS=illinois
-
You can also extend the docker-compose.yml file with a docker-compose.override.yml file (in the same directory), allowing you to add more services, or for example to change where the volumes are stored (see official documentation). For example the following will change the volume for postgres to be stored in your home directory:
The root of the docker-compose.yml file contains three sections:
-
-
services – This is a list of services provided by the application, with each service corresponding to a container.
-When communicating with each other internally, the hostnames of containers correspond to their names in this section.
-For instance, regardless of the “project” name passed to docker-compose up, the hostname for connecting to the PostgreSQL database of any given container is always going to be postgres (e.g. you should be able to access the PostgreSQL database by calling the following from inside the container: psql -d bety -U bety -h postgres).
-The services comprising the PEcAn application are described below.
-
networks – This is a list of networks used by the application.
-Containers can only communicate with each other (via ports and hostnames) if they are on the same Docker network, and containers on different networks can only communicate through ports exposed by the host machine.
-We just provide the network name (pecan) and resort to Docker’s default network configuration.
-Note that the services we want connected to this network include a networks: ... - pecan tag.
-For more details on Docker networks, see the official documentation.
-
volumes – Similarly to networks, this just contains a list of volume names we want.
-Briefly, in Docker, volumes are directories containing files that are meant to be shared across containers.
-Each volume corresponds to a directory, which can be mounted at a specific location by different containers.
-For example, syntax like volumes: ... - pecan:/data in a service definition means to mount the pecan “volume” (including its contents) in the /data directory of that container.
-Volumes also allow data to persist on containers between restarts, as normally, any data created by a container during its execution is lost when the container is re-launched.
-For example, using a volume for the database allows data to be saved between different runs of the database container.
-Without volumes, we would start with a blank database every time we restart the containers.
-For more details on Docker volumes, see the official documentation.
-Here, we define three volumes:
-
-
postgres – This contains the data files underlying the PEcAn PostgreSQL database (BETY).
-Notice that it is mounted by the postgres container to /var/lib/postgresql/data.
-This is the data that we pre-populate when we run the Docker commands to initialize the PEcAn database.
-Note that these are the values stored directly in the PostgreSQL database.
-The default files to which the database points (i.e. dbfiles) are stored in the pecan volume, described below.
-
rabbitmq – This volume contains persistent data for RabbitMQ.
-It is only used by the rabbitmq service.
-
pecan – This volume contains PEcAn’s dbfiles, which include downloaded and converted model inputs, processed configuration files, and outputs.
-It is used by almost all of the services in the PEcAn stack, and is typically mounted to /data.
-
-
-
-
-
24.4.4traefik
-
Traefik manages communication among the different PEcAn services and between PEcAn and the web.
-Among other things, traefik facilitates the setup of web access to each PEcAn service via common and easy-to-remember URLs.
-For instance, the following lines in the web service configure access to the PEcAn web interface via the URL http://localhost:8000/pecan/ :
portainer is lightweight management UI that allows you to manage the docker host (or swarm). You can use this service to monitor the different containers, see the logfiles, and start and stop containers.
-
The portainer service configuration looks like this:
-
NA: ~
-
Portainer is accessible by browsing to localhost:8000/portainer/. You can either set the password in the .env file (for an example see env.example) or you can use the web browser and go to the portainer url. If this is the first time it will ask for your password.
-
-
-
24.4.6minio
-
Minio is a service that provides access to the a folder on disk through a variety of protocols, including S3 buckets and web-based access.
-We mainly use Minio to facilitate access to PEcAn data using a web browser without the need for CLI tools.
The Minio interface is accessible by browsing to localhost:8000/minio/.
-From there, you can browse directories and download files.
-You can also upload files by clicking the red “+” in the bottom-right corner.
-
Note that it is currently impossible to create or upload directories using the Minio interface (except in the /data root directory – those folders are called “buckets” in Minio).
-Therefore, the recommended way to perform any file management tasks other than individual file uploads is through the command line, e.g.
This service allows PEcAn model outputs to be accessible via the THREDDS data server (TDS).
-When the PEcAn stack is running, the catalog can be explored in a web browser at http://localhost:8000/thredds/catalog.html.
-Specific output files can also be accessed from the command line via commands like the following:
-
Note that everything after outputs/ exactly matches the directory structure of the workflows directory.
-
Which files are served, which subsetting services are available, and other aspects of the data server’s behavior are configured in the docker/thredds_catalog.xml file.
-Specifically, this XML tells the data server to use the datasetScan tool to serve all files within the /data/workflows directory, with the additional filter that only files ending in .nc are served.
-For additional information about the syntax of this file, see the extensive THREDDS documentation.
image – This pulls a container with PostgreSQL + PostGIS pre-installed.
-Note that by default, we use PostgreSQL version 9.5.
-To experiment with other versions, you can change 9.5 accordingly.
-
networks – This allows PostgreSQL to communicate with other containers on the pecan network.
-As mentioned above, the hostname of this service is just its name, i.e. postgres, so to connect to the database from inside a running container, use a command like the following: psql -d bety -U bety -h postgres
-
volumes – Note that the PostgreSQL data files (which store the values in the SQL database) are stored on a volume called postgres (which is not the same as the postgresservice, even though they share the same name).
-
-
-
-
24.4.9rabbitmq
-
RabbitMQ is a message broker service.
-In PEcAn, RabbitMQ functions as a task manager and scheduler, coordinating the execution of different tasks (such as running models and analyzing results) associated with the PEcAn workflow.
Note that the traefik.frontend.rule indicates that browsing to http://localhost:8000/rabbitmq/ leads to the RabbitMQ management console.
-
By default, the RabbitMQ management console has username/password guest/guest, which is highly insecure.
-For production instances of PEcAn, we highly recommend changing these credentials to something more secure, and removing access to the RabbitMQ management console via Traefik.
-
-
-
24.4.10bety
-
This service operates the BETY web interface, which is effectively a web-based front-end to the PostgreSQL database.
-Unlike the postgres service, which contains all the data needed to run PEcAn models, this service is not essential to the PEcAn workflow.
-However, note that certain features of the PEcAn web interface do link to the BETY web interface and will not work if this container is not running.
This service will show the documentation for the version of PEcAn running as well as a homepage with links to all relevant endpoints. You can access this at http://localhost:8000/. You can find the documentation for PEcAn at http://localhost:8000/docs/pecan/.
This service runs the PEcAn web interface.
-It is effectively a thin wrapper around a standard Apache web server container from Docker Hub that installs some additional dependencies and copies over the necessary files from the PEcAn source code.
In terms of actively developing PEcAn using Docker, this is the service to modify when making changes to the web interface (i.e. PHP, HTML, and JavaScript code located in the PEcAn web directory).
-
-
-
24.4.13executor
-
This service is in charge of running the R code underlying the core PEcAn workflow.
-However, it is not in charge of executing the models themselves – model binaries are located on their own dedicated Docker containers, and model execution is coordinated by RabbitMQ.
Its Dockerfile is ships with the PEcAn source code, in docker/executor/Dockerfile.
-Its image is built on top of the pecan/base image (docker/base/Dockerfile), which contains the actual PEcAn source.
-To facilitate caching, the pecan/base image is itself built on top of the pecan/depends image (docker/depends/Dockerfile), a large image that contains an R installation and PEcAn’s many system and R package dependencies (which usually take ~30 minutes or longer to install from scratch).
-
In terms of actively developing PEcAn using Docker, this is the service to modify when making changes to the PEcAn R source code.
-Note that, unlike changes to the web image’s PHP code, changes to the R source code do not immediately propagate to the PEcAn container; instead, you have to re-compile the code by running make inside the container.
-
-
-
24.4.14monitor
-
This service will show all models that are currently running http://localhost:8000/monitor/. This list returned is JSON and shows all models (grouped by type and version) that are currently running, or where seen in the past. This list will also contain a list of all current active containers, as well as how many jobs are waiting to be processed.
-
This service is also responsible for registering any new models with PEcAn so users can select it and execute the model from the web interface.
Additional models are added as additional services.
-In general, their configuration should be similar to the following configuration for SIPNET, which ships with PEcAn:
This section will discuss how to add new models to PEcAn docker. To be able to add a new model to PEcAn when using docker is as simple as starting a new container. The model will come online and let the PEcAn framework know there is a new model available, there is no need to go through the process of registering this model with PEcAn. Users will be able to select this new model from web interface and run with this model selected.
-
For this process to work a docker image of the model will need to be created as well as small json file that is used to announce this new model. A separate service in PEcAn (monitor) will use this json file to keep track of all models available as well as register these models with PEcAn.
Each model will have a small json file called model_info.json that is used to describe the model and used by the monitor service to register the model with PEcAn. This file will contain information about the model that is send as part of the heartbeat of the container to the monitor service. Below is an example of this file for the ED model. The required fields are name, type, version and binary. The fields type and version are used by PEcAn to send the messages to RabbitMQ. There is no need to specify the queue name explicitly. The queue name will be created by combining these two fields with an underscore. The field binary is used to point to the actual binary in the docker container.
-
There are 2 special values that can be used, @VERSION@ which will be replaced by the version that is passed in when building the container, and @BINARY@ which will be replaced by the binary when building the docker image.
Other fields that are recommended, but currently not used yet, are:
-
-
description : a longer description of the model.
-
creator : contact person about this docker image.
-
contribuor : other people that have contributed to this docker image.
-
links : addtional links to help people when using this docker image, for example values that can be used are source to link to the source code, issues to link to issue tracking system, and documentation to link to model specific documentation.
-
citation : how the model should be cited in publications.
-
-
-
-
24.5.2 Model build
-
In general we try to minimize the size of the images. To be able to do this we split the process of creating the building of the model images into two pieces (or leverage of an image that exists from the original model developers). If you look at the example Dockerfile you will see that there are 2 sections, the first section will build the model binary, the second section will build the actual PEcAn model, which copies the binary from the first section.
-
This is an example of how the ED2 model is build. This will install all the packages needed to build ED2 model, gets the latest version from GitHub and builds the model.
-
The second section will create the actual model runner. This will leverage the PEcAn model image that has PEcAn already installed as well as the python code to listen for messages and run the actual model code. This will install some additional packages needed by the model binary (more about that below), copy the model_info.json file and change the @VERSION@ and @BINARY@ placeholders.
-
It is important values for type and version are set correct. The PEcAn code will use these to register the model with the BETY database, which is then used by PEcAn to send out a message to a specfic worker queue, if you do not set these variables correctly your model executor will pick up messages for the wrong model.
-
To build the docker image, we use a Dockerfile (see example below) and run the following command. This command will expect the Dockerfile to live in the model specific folder and the command is executed in the root pecan folder. It will copy the content of the pecan folder and make it available to the build process (in this example we do not need any additional files).
-
Since we can have multiple different versions of a model be available for PEcAn we ar using the following naming schema pecan/model-<modeltype>-<version>:<pecan version. For example the image below will be named pecan/model-ed2-git, since we do not specify the exact version it will be atomically be named pecan/model-ed2-git:latest.
WARNING: Dockerfile environment variables set via ENV are assigned all at once; they do not evaluate successively, left to right.
-Consider the following block:
In this block, the expansion for setting MODEL_TYPE_VERSIONis not aware of the current values of MODEL_TYPE or MODEL_VERSION, and will therefore be set incorrectly to just _ (unless they have been set previously, in which case it will be aware only of their earlier values).
-As such, variables depending on other variables must be set in a separate, subsequent ENV statement than the variables they depend on.
-
Once the model has build and is working we can add it to the PEcAn stack and be able to use this model in the web interface. There are two methods to start this new model. First, we can add it to the docker-compose.yml file and start the container using docker-compose -p pecan -d up.
Following are some solutions for common problems that you might encounter when building the docker images for a model.
-
-
-
24.5.4 Debugging missing libraries
-
When building the model binary it might require specific libraries to be installed. In the second stage the model binary is copied into a new image, which could result in the binary missing specific libraries. In the case of the ED2 model the following was used to find the libraries that are needed to be installed (libgfortran5 and libopenmpi3).
-
The first step is to build the model using the Dockerfile (in this case the ap-get install was missing in the second stage).
-
Step 5/9 : RUN git clone https://github.com/EDmodel/ED2.git && cd ED2/ED/build && curl -o make/include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` && if [ "${MODEL_VERSION}" != "git" ]; then git checkout ${MODEL_VERSION}; fi && ./install.sh -g -p VM
-... LOTS OF OUTPUT ...
-make[1]: Leaving directory '/src/ED2/ED/build/bin-opt-E'
-Installation Complete.
-Removing intermediate container a53eba9a8fc1
- ---> 7f23c6302130
-Step 6/9 : FROM pecan/executor:latest
- ---> f19d81b739f5
-... MORE OUTPUT ...
-Step 9/9 : COPY --from=model-binary /src/ED2/ED/build/ed_2.1-opt /usr/local/bin/ed2.${MODEL_VERSION}
- ---> 07ac841be457
-Successfully built 07ac841be457
-Successfully tagged pecan/pecan-ed2:latest
-
At this point we have created a docker image with the binary and all PEcAn code that is needed to run the model. Some models (especially those build as native code) might be missing additional packages that need to be installed in the docker image. To see if all libraries are installed for the binary.
Start the build container again (this is the number before the line FROM pecan/executor:latest, 7f23c6302130 in the example), and find the missing libraries listed above (for example libmpi_usempif08.so.40):
This shows the pages is libopenmpi3 that needs to be installed, do this for all missing packages, modify the Dockerfile and rebuild. Next time you run the ldd command there should be no more packages being listed.
For general use, it is sufficient to use the pre-built PEcAn images hosted on Docker Hub (see Docker quickstart).
-However, there are cases where it makes sense to re-build the Docker images locally.
-The following is a list of PEcAn-specific images and reasons why you would want to rebuild them locally:
-
-
pecan/depends – Rebuild if:
-
-
You modify the docker/depends/Dockerfile
-
You introduce new system dependencies (i.e. things that need to be installed with apt-get)
-
You introduce new R package dependencies, and you want those R package installations to be cached during future builds. For packages with fast build times, it may be fine to let them be installed as part of PEcAn’s standard build process (i.e. make).
-
-
pecan/base – Rebuild if:
-
-
You built a new version of pecan/depends (on which pecan/base depends)
-
You modify the docker/base/Dockerfile
-
You made changes to the PEcAn R package source code, the Makefile, or web/workflow.R.
-
-
NOTE that changes to the web interface code affect pecan/web, notpecan/base
-
-
-
pecan/executor – Rebuild if:
-
-
You built a new version of pecan/base (on which pecan/executor depends) and/or, pecan/depends (on which pecan/base depends)
-
You modified the docker/executor/Dockerfile
-
You modified the RabbitMQ Python scripts (e.g. docker/receiver.py, docker/sender.py)
-
-
pecan/web – Rebuild if you modified any of the following:
-
-
docker/web/Dockerfile
-
The PHP/HTML/JavaScript code for the PEcAn web interface in web/ (exceptweb/workflow.R – that goes in pecan/base)
-
docker/config.docker.php (the config.php file for Docker web instances)
-
documentation/index_vm.html (the documentation HTML website)
-
NOTE: Because changes to this code are applied instantly (i.e. do not require compilation or installation), a more effective way to do local development may be to mount the web/ or other relevant folders as a volume onto the pecan/web container.
-
-
-
The easiest way to quickly re-build all of the images is using the docker.sh script in the PEcAn source code root directory.
-This script will build all of the docker images locally on your machine, and tag them as latest.
-This will not build the pecan/depends image by default because that takes considerably longer.
-However, you can force the script to build pecan/depends as well by setting the DEPEND environment variable to 1 (i.e. DEPEND=1 ./docker.sh).
-The following instructions provide details on how to build each image individually.
-
To build an image locally, use the docker build command as described below.
-For more details, see docker build --help or the online Docker build documentation.
-
First, in a terminal window, navigate (cd) into the PEcAn source code root directory.
-From there, the general syntax for building an image looks like the following:
docker build – This is the core command.
-The standard syntax is docker build [OPTIONS] <PATH>, where <PATH> refers to the directory to be used as the “build context”.
-The “build context” is the working directory assumed by the Dockerfiles.
-In PEcAn, this is always the PEcAn source code root directory, which allows Dockerfiles to use instructions such as COPY web/workflow.R /work/.
-In this example, the <PATH> is set to the current working directory, i.e. . because we are already in the PEcAn root directory.
-If you were located in a different directory, you would have to provide a path to the PEcAn source code root directory.
-Also, by default, docker build will look for a Dockerfile located at <PATH>/Dockerfile, but this is modified by the -f option described below.
-
-t pecan/depends:latest – The -t/--tag option specifies how the image will be labeled.
-By default, Docker only defines unique image IDs, which are hexidecimal strings that are unintuitive and hard to remember.
-Tags are useful for referring to specific images in a human-readable way.
-Note that the same unique image can have multiple tags associated with it, so it is possible for, e.g. pecan/depends:latest, pecan/depends:custom, and even mypecan/somethingelse:20.0 to refer to the same exact image.
-To see a table of all local images, including their tags and IDs, run docker image ls.
-
-
NOTE: PEcAn’s docker-compose.yml can be configured via the PECAN environment variable to point at different versions of PEcAn images.
-By default, it points to the :latest versions of all images.
-However, if you wanted to, for instance, build :local images corresponding to your local source code and then run that version of PEcAn, you would run:
-
-
PECAN=local docker-compose -p pecan up -d
-
This is an effective way to do local development and testing of different PEcAn versions, as described below.
-
-f docker/depends/Dockerfile – The -f/--file tag is used to provide an alternative location and file name for the Dockerfile.
-
-
-
24.6.1 Local development and testing with Docker
-
The following is an example of one possible workflow for developing and testing PEcAn using local Docker images.
-The basic idea is to mount a local version of the PEcAn source code onto a running pecan/executor image, and then send a special “rebuild” RabbitMQ message to the container to trigger the rebuild whenever you make changes.
-NOTE: All commands assume you are working from the PEcAn source code root directory.
-
-
In the PEcAn source code directory, create a docker-compose.override.yml file with the following contents.:
This will mount the current directory . to the /pecan directory in the executor container.
-The special docker-compose.override.yml file is read automatically by docker-compose and overrides or extends any instructions set in the original docker-compose.yml file.
-It provides a convenient way to host server-specific configurations without having to modify the project-wide (and version-controlled) default configuration.
-For more details, see the Docker Compose documentation.
-
Update your PEcAn Docker stack with docker-compose up -d.
-If the stack is already running, this should only restart your executor instance while leaving the remaining containers running.
-
To update to the latest local code, run ./scripts/docker_rebuild.sh.
-Under the hood, this uses curl to post a RabbitMQ message to a running Docker instance.
-By default, the scripts assumes that username and password are both guest and that the RabbitMQ URL is http://localhost:8000/rabbitmq.
-All of these can be customized by setting the environment variables RABBITMQ_USER, RABBITMQ_PASSWORD, and RABBITMQ_URL, respectively (or running the script prefixed with those variables, e.g. RABBITMQ_USER=carya RABBITMQ_PASSWORD=illinois ./scripts/docker_rebuild.sh).
-This step can be repeated whenever you want to trigger a rebuild of the local code.
-
-
NOTE: The updates with this workflow are specific to the running container session; restarting the executor container will revert to the previous versions of the installed packages.
-To make persistent changes, you should re-build the pecan/base and pecan/executor containers against the current version of the source code.
-
NOTE: The mounted PEcAn source code directory includes everything in your local source directory, including installation artifacts used by make.
-This can lead to two common issues:
-- Any previous make cache files (stuff in the .install, .docs, etc. directories) persist across container instances, even though the installed packages may not. To ensure a complete build, it’s a good idea to run make clean on the host machine to remove these artifacts.
-- Similarly, any installation artifacts from local builds will be carried over to the build. In particular, be wary of packages with compiled code, such as modules/rtm (PEcAnRTM) – the compiled .o, .so, .mod, etc. files from compilation of such packages will carry over into the build, which can cause conflicts if the package was also built locally.
-
The docker-compose.override.yml is useful for some other local modifications.
-For instance, the following adds a custom ED2 “develop” model container.
Similarly, this snippet modifies the pecan network to use a custom IP subnet mask.
-This is required on the PNNL cluster because its servers’ IP addresses often clash with Docker’s default IP mask.
24.7.1 “Package not available” while building images
-
PROBLEM: Packages fail to install while building pecan/depends and/or pecan/base with an error like the following:
-
Installing package into ‘/usr/local/lib/R/site-library’
-(as ‘lib’ is unspecified)
-Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2018-09-01/src/contrib:
- cannot open URL 'https://mran.microsoft.com/snapshot/2018-09-01/src/contrib/PACKAGES'
-Warning message:
-package ‘<PACKAGE>’ is not available (for R version 3.5.1)
-
CAUSE: This can sometimes happen if there are problems with Microsoft’s CRAN snapshots, which are the default repository for the rocker/tidyverse containers.
-See GitHub issues rocker-org/rocker-versioned#102 and #58.
-
SOLUTION: Add the following line to the depends and/or base Dockerfiles before (i.e. above) any commands that install R packages (e.g. Rscript -e "install.packages(...)"):
-
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.org'))" >> /usr/local/lib/R/etc/Rprofile.site
-
This will set the default repository to the more reliable (albeit, more up-to-date; beware of breaking package changes!) RStudio CRAN mirror.
-Then, build the image as usual.
This document will slowly replace each of the components with the appropriate docker images. At then end of this document you should be able to use the docker-compose command to bring up the full docker stack as if you had started with this origianally.
-
-
24.8.1 Running BETY as a docker container
-
This will replace the BETY application running on the machine with a docker image. This will assume you still have the database running on the local machine and the only thing we replace is the BETY application.
-
If you are running systemd (Ubuntu 16.04 or Centos 7) you can copy the following file to /etc/systemd/system/bety.service (replace LOCAL_SERVER=99 with your actual server). If you have postgres running on another server replace 127.0.0.1 with the actual ip address of the postgres server.
At this point we can enable the bety service (this only needs to be done once). First we need to tell systemd a new service is available using systemctl daemon-reload. Next we enable the BETY service so it will restart automatically when the machine reboots, using systemctl enable bety. Finally we can start the BETY service using systemctl start bety. At this point BETY is running as a docker container on port 8000. You can see the log messages using journalctl -u bety.
-
Next we need to modify apache configuration files. The file /etc/apache2/conf-enabled/bety.conf will be replaced with the following content:
Once this modified we can restart apache using systemctl restart apache2. At this point BETY is running in a container and is accessable trough the webserver at http://server/bety/.
-
To upgrade to a new version of BETY you can now use the docker commands. You can use the following commands to stop BETY, pull the latest image down, migrate the database (you made a backup correct?) and start BETY again.
Once you are satisfied with the migration of BETY you can remove the bety folder as well as any ruby binaries you have installed.
-
-
-
-
-
24.9 The PEcAn Docker API
-
If you have a running instance of Dockerized PEcAn (or other setup where PEcAn workflows are submitted via RabbitMQ),
-you have the option of running and managing PEcAn workflows using the pecanapi package.
-
For more details, see the pecanapipackage vignette and function-level documentation.
-What follows is a lightning introduction.
-
-
24.9.0.1 Installation
-
The package can be installed directly from GitHub via devtools::install_github:
With pecanapi, creating a workflow, submitting it to RabbitMQ, monitoring its progress, and processing its output can all be accomplished via an R script.
-
Start by loading the package (and the magrittr package, for the %>% pipe operator).
This section provides additional details about how PEcAn uses RabbitMQ to manage communication between its Docker containers.
-
In PEcAn, we use the Python pika client to post and retrieve messages from RabbitMQ.
-As such, every Docker container that communicates with RabbitMQ contains two Python scripts: sender.py and reciever.py.
-Both are located in the docker directory in the PEcAn source code root.
-
-
24.10.1 Producer – sender.py
-
The sender.py script is in charge of posting messages to RabbitMQ.
-In the RabbitMQ documentation, it is known as a “producer”.
-It runs once for every message posted to RabbitMQ, and then immediately exits (unlike the receiver.py, which runs continuously – see below).
<URI> – The unique identifier of the RabbitMQ instance, similar to a URL.
-The format is amqp://username:password@host/vhost.
-By default, this is amqp://guest:guest@rabbitmq/%2F (the %2F here is the hexadecimal encoding for the / character).
-
<queue> – The name of the queue on which to post the message.
-
<message> – The contents of the message to post, in JSON format.
-A typical message posted by PEcAn looks like the following:
The PEcAn.remote::start_rabbitmq function is a wrapper for this script that provides an easy way to post a folder message to RabbitMQ from R.
-
-
-
24.10.2 Consumer – receiver.py
-
Unlike sender.py, receiver.py runs like a daemon, constantly listening for messages.
-In the RabbitMQ documentation, it is known as a “consumer”.
-In PEcAn, you can tell that it is ready to receive messages if the corresponding logs (e.g. docker-compose logs executor) show the following message:
-
[*] Waiting for messages. To exit press CTRL+C.
-
Our reciever is configured by three environment variables:
-
-
RABBITMQ_URI – This defines the URI where RabbitMQ is running.
-See corresponding argument in the producer
-
RABBITMQ_QUEUE – This is the name of the queue on which the consumer will listen for messages, just as in the producer.
-
APPLICATION – This specifies the name (including the path) of the default executable to run when receiving a message.
-At the moment, it should be an executable that runs in the directory specified by the message’s folder variable.
-In the case of PEcAn models, this is usually ./job.sh, such that the folder corresponds to the run directory associated with a particular runID (i.e. where the job.sh is located).
-For the PEcAn workflow itself, this is set to R CMD BATCH workflow.R, such that the folder is the root directory of the workflow (in the executor Docker container, something like /data/workflows/PEcAn_<workflowID>).
-This default executable is overridden if the message contains a custom_application key.
-If included, the string specified by the custom_application key will be run as a command exactly as is on the container, from the directory specified by folder.
-For instance, in the example below, the container will print “Hello there!” instead of running its default application.
This will set the hostname to the name of the current machine (defined by the $fqdn variable earlier in the config.php file) to an array with one entry, whose key is rabbitmq and whose value is the RabbitMQ URI (amqp://...).
-
These values are converted into the appropriate entries in the pecan.xml in web/04-runpecan.php.
-
-
-
24.10.4 RabbitMQ in the PEcAn XML
-
RabbitMQ is a special case of remote execution, so it is configured by the host node.
-An example RabbitMQ configuration is as follows:
Here, uri and queue have the same general meanings as described in “producer”.
-Note that queue here refers to the target model.
-In PEcAn, RabbitMQ model queues are named as MODELTYPE_REVISION,
-so the example above refers to the SIPNET model version 136.
-Another example is ED2_git, referring to the latest git version of the ED2 model.
-
-
-
24.10.5 RabbitMQ configuration in Dockerfiles
-
As described in the “consumer” section, our standard RabbitMQ receiver script is configured using three environment variables: RABBITMQ_URI, RABBITMQ_QUEUE, and APPLICATION.
-Therefore, configuring a container to work with PEcAn’s RabbitMQ instance requires setting these three variables in the Dockerfile using an ENV statement.
-
For example, this excerpt from docker/base/Dockerfile.executor (for the pecan/executor image responsible for the PEcAn workflow) sets these variables as follows:
Similarly, this excerpt from docker/models/Dockerfile.sipnet (which builds the SIPNET model image) is a typical example for a model image.
-Note the use of ARG here to specify a default version model version of 136 while allowing this to be configurable (via --build-arg MODEL_VERSION=X) at build time:
WARNING: Dockerfile environment variables set via ENV are assigned all at once; they do not evaluate successively, left to right.
-Consider the following block:
In this block, the expansion for setting RABBITMQ_QUEUEis not aware of the current values of MODEL_TYPE or MODEL_VERSION, and will therefore be set incorrectly to just _ (unless they have been set previously, in which case it will be aware only of their earlier values).
-As such, variables depending on other variables must be set in a separate, subsequent ENV statement than the variables they depend on.
-
-
-
24.10.6 Case study: PEcAn web interface
-
The following describes in general terms what happens during a typical run of the PEcAn web interface with RabbitMQ.
-
-
The user initializes all containers with docker-compose up.
-All the services that interact with RabbitMQ (executor and all models) run receiver.py in the foreground, waiting for messages to tell them what to do.
-
The user browses to http://localhost:8000/pecan/ and steps through the web interface.
-All the pages up to the 04-runpecan.php run on the web container, and are primarily for setting up the pecan.xml file.
-
Once the user starts the PEcAn workflow at 04-runpecan.php, the underlying PHP code connects to RabbitMQ (based on the URI provided in config.php) and posts the following message to the pecan queue:
The executor service, which is listening on the pecan queue, hears this message and executes its APPLICATION (R CMD BATCH workflow.R) in the working directory specified in the message’s folder.
-The executor service then performs the pre-execution steps (e.g. trait meta-analysis, conversions) itself.
-Then, to actually execute the model, executor posts the following message to the target model’s queue:
The target model service, which is listening on its dedicated queue, hears this message and runs its APPLICATION, which is job.sh, in the directory indicated by the message.
-Upon exiting (normally), the model service writes its status into a file called rabbitmq.out in the same directory.
-
The executor container continuously looks for the rabbitmq.out file as an indication of the model run’s status.
-Once it sees this file, it reads the status and proceeds with the post-execution parts of the workflow.
-(NOTE that this isn’t perfect. If the model running process exits abnormally, the rabbitmq.out file may not be created, which can cause the executor container to hang. If this happens, the solution is to restart the executor container with docker-compose restart executor).
This chapter contains details about installing and maintaining the uncontainerized version of PEcAn on a virtual machine or a server. If you are running PEcAn inside of Docker, many of the particulars will be different and you should refer to the docker chapter instead of this one.
The PEcAn virtual machine consists of all of PEcAn pre-compiled within a Linux operating system and saved in a “virtual machine” (VM). Virtual machines allow for running consistent set-ups without worrying about differences between operating systems, library dependencies, compiling the code, etc.
-
-
Install VirtualBox This is the software that runs the virtual machine. You can find the download link and instructions at http://www.virtualbox.org. NOTE: On Windows you may see a warning about Logo testing, it is okay to ignore the warning.
-
Download the PEcAn VM You can find the download link at http://opensource.ncsa.illinois.edu/projects/artifacts.php?key=PECAN, under the “Files” header. Click the “.ova” file to begin the download. Note that the file is ~7 GB, so this download can take several minutes to hours depending on your connection speed. Also, the VM requires >4 GB of RAM to operate correctly. Please check current usage of RAM and shutdown processes as needed.
-
Import the VM Once the download is complete, open VirtualBox. In the VirtualBox menus, go to “File” → “Import Appliance” and locate the downloaded “.ova” file.
-
-
For Virtualbox version 5.x: In the Appliance Import Settings, make sure you select “Reinitialize the MAC address of all network cards” (picture below). This is not selected by default and can result in networking issues since multiple machines might claim to have the same network MAC Address.
-
-
For Virtualbox versions starting with 6.0, there is a slightly different interface (see figure). Select “Generate new MAC addresses for all network adapters” from the MAC Address Policy:
-
-
NOTE: If you experience network connection difficulties in the VM with this enabled, try re-importing the VM without this setting selected).
-
Finally, click “Import” to build the Virtual Machine from its image.
-
-
Launch PEcAn Double click the icon for the PEcAn VM. A terminal window will pop up showing the machine booting up which may take a minute. It is done booting when you get to the pecan login: prompt. You do not need to login as the VM behaves like a server that we will be accessing through you web browser. Feel free to minimize the VM window.
-
-
-
If you do want to login to the VM, the credentials are as follows: username: carya, password: illinois (after the pecan tree, [Carya illinoinensis][pecan-wikipedia]).
-
-
-
Open the PEcAn web interface With the VM running in the background, open any web browser on the same machine and navigate to localhost:6480/pecan/ to start the PEcAn workflow. (NOTE: The trailing backslash may be necessary depending on your browser)
-
-
-
To ssh into the VM, open up a terminal on your machine and execute ssh -l carya -p 6422 localhost. Username and password are the same as when you log into the machine.
-
-
-
-
-
23.2 AWS Setup
-
***********Mirror of earlier section in installation section?*********************
-
The following are Mike’s rough notes from a first attempt to port the PEcAn VM to the AWS. This was done on a Mac
-
These notes are based on following the instructions here
-
-
23.2.1 Convert PEcAn VM
-
AWS allows upload of files as VMDK but the default PEcAn VM is in OVA format
-
-
If you haven’t done so already, download the PEcAn VM
In the code below, make sure to change the PEcAn version, the name of the bucket, and the name of the region. Make sure that the PEcAn version matches the one you downloaded.
-
Also, you may want to choose a considerably larger instance type. The one chosen below is that corresponding to the AWS Free Tier
Make sure to note the ID of the image since you’ll need it to check the VM status. Once the image is uploaded it will take a while (typically about an hour) for Amazon to convert the image to one it can run. You can check on this progress by running
-
ec2-describe-conversion-tasks <image.ID>
-
-
-
23.2.6 Configuring the VM
-
On the EC2 management webpage, https://console.aws.amazon.com/ec2, if you select Instances on the left hand side (LHS) you should be able to see your new PEcAn image as an option under Launch Instance.
-
Before launching, you will want to update the firewall to open up additional ports that PEcAn needs – specifically port 80 for the webpage. Port 22 (ssh/sftp) should be open by default. Under “Security Groups” select “Inbound” then “Edit” and then add “HTTP”.
-
Select “Elastic IPs” on the LHS, and “Allocate New Address” in order to create a public IP for your VM.
-
Next, select “Network Interfaces” on the LHS and then under Actions select “Associate Addresses” then choose the Elastic IP you just created.
Select “Load Balancers” on the LHS, click on “Create Load Balancer”, follow Wizard keeping defaults.
-
To be able to launch multiple VMs: Under “Instances” convert VM to an Image. When done, select Launch, enable multiple instances, and associate with the previous security group. Once running, go back to “Load Balancers” and add the instances to the load balancer. Each instance can be accessed individually by it’s own public IP, but external users should access the system more generally via the Load Balancers DNS.
-
-
-
23.2.8 Booting the VM
-
Return to “Instances” using the menu on the LHS.
-
To boot the VM select “Actions” then “Instance State” then “Start”. In the future, once you have the VM loaded and configured this last step is the only one you will need to repeat to turn your VM on and off.
-
The menu provided should specify the Public IP where the VM has launched
-
-
-
-
-
23.3 Shiny Setup
-
Installing and configuring Shiny for PEcAn
-authors - Alexey Shiklomanov
-- Rob Kooper
-
NOTE: Instructions are only tested for CentOS 6.5 and Ubuntu 16.04
-NOTE: Pretty much every step here requires root access.
-
-
23.3.1 Install the Shiny R package and Shiny server
-
Follow the instructions on the Shiny download page for the operating system you are using.
-
-
-
23.3.2 Modify the shiny configuration file
-
The Shiny configuration file is located in /etc/shiny-server/shiny-server.conf. Comment out the entire file and add the following, replacing <username> with your user name and <location> with the URL location you want for your app. This will allow you to run Shiny apps from your web browser at https://your.server.edu/shiny/your-location
-
run as shiny;
-server {
- listen 3838;
- location /<location>/ {
- run as <username>;
- site_dir /path/to/your/shiny/app;
- log_dir /var/log/shiny-server;
- directory_index on;
- }
-}
-
For example, my configuration on the old test-pecan looks like this.
-
run as shiny;
-server {
- listen 3838;
- location /ashiklom/ {
- run as ashiklom;
- site_dir /home/ashiklom/fs-data/pecan/shiny/;
- log_dir /var/log/shiny-server;
- directory_index on;
- }
-}
If you change the configuration, for example to add a new location, you will need to restart Shiny server.
-If you are setting up a new instance of Shiny, skip this step and continue with the guide, since there are a few more steps to get Shiny working.
-If there is an instance of Shiny already running, you can restart it with:
-
## On CentOS
-sudo service shiny-server stop
-sudo service shiny-server start
-
-## On Ubuntu
-sudo systemctl stop shiny-server.service
-sudo systemctl start shiny-server.service
-
-
-
23.3.3 Set the Apache proxy
-
Create a file with the following name, based on the version of the operating system you are using:
23.3.3.1Ubuntu only: Enable the new shiny configuration
-
sudo a2enconf shiny
-
This will create a symbolic link to the newly created shiny.conf file inside the /etc/apache2/conf-enabled directory.
-You can do ls -l /etc/apache2/conf-enabled to confirm that this worked.
-
-
-
-
23.3.4 Enable and start the shiny server, and restart apache
-
-
23.3.4.1 On CentOS
-
sudo ln -s /opt/shiny-server/config/init.d/redhat/shiny-server /etc/init.d
-sudo service shiny-server stop
-sudo service shiny-server start
-sudo service httpd restart
-
You can check that Shiny is running with service shiny-server status.
-
-
-
23.3.4.2 On Ubuntu
-
Enable the Shiny server service.
-This will make sure Shiny runs automatically on startup.
-
sudo systemctl enable shiny-server.service
-
Restart Apache.
-
sudo apachectl restart
-
Start the Shiny server.
-
sudo systemctl start shiny-server.service
-
If there are problems, you can stop the shiny-server.service with…
-
sudo systemctl stop shiny-server.service
-
…and then use start again to restart it.
-
-
-
-
23.3.5 Troubleshooting
-
Refer to the log files for shiny (/var/log/shiny-server.log) and httpd (on CentOS, /var/log/httpd/error-log; on Ubuntu, /var/log/apache2/error-log).
Installing and configuring Thredds for PEcAn
-authors - Rob Kooper
-
NOTE: Instructions are only tested for Ubuntu 16.04 on the VM, if you have instructions for CENTOS/RedHat please update this documentation
-NOTE: Pretty much every step here requires root access.
-
-
23.4.1 Install the Tomcat 8 and Thredds webapp
-
The Tomcat 8 server can be installed from the default Ubuntu repositories. The thredds webapp will be downloaded and installed from unidata.
-
First step is to install Tomcat 8 and configure it. The flag -Dtds.content.root.path should point to the location of where the thredds folder is located. This needs to be writeable by the user for tomcat. -Djava.security.egd is a special flag to use a different random number generator for tomcat. The default would take to long to generate a random number.
To customize the thredds server for your installation edit the file in /home/carya/thredds/threddsConfig.xml. For example the following file is included in the VM.
-
<?xml version="1.0" encoding="UTF-8"?>
-<threddsConfig>
-
- <!-- all options are commented out in standard install - meaning use default values -->
- <!-- see http://www.unidata.ucar.edu/software/thredds/current/tds/reference/ThreddsConfigXMLFile.html -->
- <serverInformation>
- <name>PEcAn</name>
- <logoUrl>/pecan/images/pecan_small.jpg</logoUrl>
- <logoAltText>PEcAn</logoAltText>
-
- <abstract>Scientific Data</abstract>
- <keywords>meteorology, atmosphere, climate, ocean, earth science</keywords>
-
- <contact>
- <name>Rob Kooper</name>
- <organization>NCSA</organization>
- <email>kooper@illinois.edu</email>
- <!--phone></phone-->
- </contact>
- <hostInstitution>
- <name>PEcAn</name>
- <webSite>http://www.pecanproject.org/</webSite>
- <logoUrl>/pecan/images/pecan_small.jpg</logoUrl>
- <logoAltText>PEcAn Project</logoAltText>
- </hostInstitution>
- </serverInformation>
-
- <!--
- The <catalogRoot> element:
- For catalogs you don't want visible from the /thredds/catalog.xml chain
- of catalogs, you can use catalogRoot elements. Each catalog root config
- catalog is crawled and used in configuring the TDS.
-
- <catalogRoot>myExtraCatalog.xml</catalogRoot>
- <catalogRoot>myOtherExtraCatalog.xml</catalogRoot>
- -->
-
- <!--
- * Setup for generated HTML pages.
- *
- * NOTE: URLs may be absolute or relative, relative URLs must be relative
- * to the webapp URL, i.e., http://server:port/thredds/.
- -->
- <htmlSetup>
- <!--
- * CSS documents used in generated HTML pages.
- * The CSS document given in the "catalogCssUrl" element is used for all pages
- * that are HTML catalog views. The CSS document given in the "standardCssUrl"
- * element is used in all other generated HTML pages.
- * -->
- <standardCssUrl>tds.css</standardCssUrl>
- <catalogCssUrl>tdsCat.css</catalogCssUrl>
- <openDapCssUrl>tdsDap.css</openDapCssUrl>
-
- <!--
- * The Google Analytics Tracking code you would like to use for the
- * webpages associated with THREDDS. This will not track WMS or DAP
- * requests for data, only browsing the catalog.
- -->
- <googleTrackingCode></googleTrackingCode>
-
- </htmlSetup>
-
- <!--
- The <TdsUpdateConfig> element controls if and how the TDS checks
- for updates. The default is for the TDS to check for the current
- stable and development release versions, and to log that information
- in the TDS serverStartup.log file as INFO entries.
-
- <TdsUpdateConfig>
- <logVersionInfo>true</logVersionInfo>
- </TdsUpdateConfig>
- -->
-
- <!--
- The <CORS> element controls Cross-Origin Resource Sharing (CORS).
- CORS is a way to allow a website (such as THREDDS) to open up access
- to resources to web pages and applications running on a different domain.
- One example would be allowing a web-application to use fonts from
- a separate host. For TDS, this can allow a javascript app running on a
- different site to access data on a THREDDS server.
- For more information see: https://en.wikipedia.org/wiki/Cross-origin_resource_sharing
- The elements below represent defaults. Only the <enabled> tag is required
- to enable CORS. The default allowed origin is '*', which allows sharing
- to any domain.
- <CORS>
- <enabled>false</enabled>
- <maxAge>1728000</maxAge>
- <allowedMethods>GET</allowedMethods>
- <allowedHeaders>Authorization</allowedHeaders>
- <allowedOrigin>*</allowedOrigin>
- </CORS>
- -->
-
- <!--
- The <CatalogServices> element:
- - Services on local TDS served catalogs are always on.
- - Services on remote catalogs are set with the allowRemote element
- below. They are off by default (recommended).
- -->
- <CatalogServices>
- <allowRemote>false</allowRemote>
- </CatalogServices>
-
- <!--
- Configuring the CDM (netcdf-java library)
- see http://www.unidata.ucar.edu/software/netcdf-java/reference/RuntimeLoading.html
-
- <nj22Config>
- <ioServiceProvider class="edu.univ.ny.stuff.FooFiles"/>
- <coordSysBuilder convention="foo" class="test.Foo"/>
- <coordTransBuilder name="atmos_ln_sigma_coordinates" type="vertical" class="my.stuff.atmosSigmaLog"/>
- <typedDatasetFactory datatype="Point" class="gov.noaa.obscure.file.Flabulate"/>
- </nj22Config>
- -->
-
- <!--
- CDM uses the DiskCache directory to store temporary files, like uncompressed files.
- <DiskCache>
- <alwaysUse>false</alwaysUse>
- <scour>1 hour</scour>
- <maxSize>1 Gb</maxSize>
- </DiskCache>
- -->
-
- <!--
- Caching open NetcdfFile objects.
- default is to allow 50 - 100 open files, cleanup every 11 minutes
- <NetcdfFileCache>
- <minFiles>50</minFiles>
- <maxFiles>100</maxFiles>
- <scour>11 min</scour>
- </NetcdfFileCache>
- -->
-
- <!--
- The <HTTPFileCache> element:
- allow 10 - 20 open datasets, cleanup every 17 minutes
- used by HTTP Range requests.
- <HTTPFileCache>
- <minFiles>10</minFiles>
- <maxFiles>20</maxFiles>
- <scour>17 min</scour>
- </HTTPFileCache>
- -->
-
- <!--
- Writing GRIB indexes.
- <GribIndexing>
- <setExtendIndex>false</setExtendIndex>
- <alwaysUseCache>false</alwaysUseCache>
- </GribIndexing>
- -->
-
- <!--
- Persist joinNew aggregations to named directory. scour every 24 hours, delete stuff older than 90 days
- <AggregationCache>
- <scour>24 hours</scour>
- <maxAge>90 days</maxAge>
- <cachePathPolicy>NestedDirectory</cachePathPolicy>
- </AggregationCache>
- -->
-
- <!--
- How to choose the template dataset for an aggregation. latest, random, or penultimate
- <Aggregation>
- <typicalDataset>penultimate</typicalDataset>
- </Aggregation>
- -->
-
- <!--
- The Netcdf Subset Service is off by default.
- <NetcdfSubsetService>
- <allow>false</allow>
- <scour>10 min</scour>
- <maxAge>-1 min</maxAge>
- </NetcdfSubsetService>
- -->
-
- <!--
- <Opendap>
- <ascLimit>50</ascLimit>
- <binLimit>500</binLimit>
- <serverVersion>opendap/3.7</serverVersion>
- </Opendap>
- -->
-
- <!--
- The WCS Service is off by default.
- Also, off by default (and encouraged) is operating on a remote dataset.
- <WCS>
- <allow>false</allow>
- <allowRemote>false</allowRemote>
- <scour>15 min</scour>
- <maxAge>30 min</maxAge>
- </WCS>
- -->
-
- <!--
- <WMS>
- <allow>false</allow>
- <allowRemote>false</allowRemote>
- <maxImageWidth>2048</maxImageWidth>
- <maxImageHeight>2048</maxImageHeight>
- </WMS>
- -->
-
- <!--
- <NCISO>
- <ncmlAllow>false</ncmlAllow>
- <uddcAllow>false</uddcAllow>
- <isoAllow>false</isoAllow>
- </NCISO>
- -->
-
- <!-- CatalogGen service is off by default.
- <CatalogGen>
- <allow>false</allow>
- </CatalogGen>
- -->
-
- <!-- DLwriter service is off by default.
- As is support for operating on remote catalogs.
- <DLwriter>
- <allow>false</allow>
- <allowRemote>false</allowRemote>
- </DLwriter>
- -->
-
- <!-- DqcService is off by default.
- <DqcService>
- <allow>false</allow>
- </DqcService>
- -->
-
- <!--
- Link to a Viewer application on the HTML page:
- <Viewer>my.package.MyViewer</Viewer>
- -->
-
- <!--
- Add a DataSource - essentially an IOSP with access to Servlet request parameters
- <datasetSource>my.package.DatsetSourceImpl</datasetSource>
- -->
-
- <!--
- set FeatureCollection logging
- <FeatureCollection>
- <RollingFileAppender>
- <MaxFileSize>1 MB</MaxFileSize>
- <MaxBackups>5</MaxBackups>
- <Level>INFO</Level>
- </RollingFileAppender>
- </FeatureCollection>
- -->
-
- <!--
- Configure how the NetCDF-4 C library is discovered and used.
- libraryPath: The directory in which the native library is installed.
- libraryName: The name of the native library. This will be used to locate the proper .DLL, .SO, or .DYLIB file
- within the libraryPath directory.
- useForReading: By default, the native library is only used for writing NetCDF-4 files; a pure-Java layer is
- responsible for reading them. However, if this property is set to true, then it will be used for reading
- NetCDF-4 (and HDF5) files as well.
- -->
- <!--
- <Netcdf4Clibrary>
- <libraryPath>/usr/local/lib</libraryPath>
- <libraryName>netcdf</libraryName>
- <useForReading>false</useForReading>
- </Netcdf4Clibrary>
- -->
-</threddsConfig>
-
-
-
-
23.4.2 Update the catalog
-
For example to update the catalog with the latest data, run the following command from the root crontab. This cronjob will also synchronize the database with remote servers and dump your database (by default in /home/carya/dump)
These are specific notes for installing PEcAn on Ubuntu (14.04) and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed.
-
This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful.
NOTE This will allow anybody to login to the machine through the rstudio interface and run any arbitrary code. The login used however is the same as the system login/password.
These are specific notes for installing PEcAn on CentOS (7) and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed.
-
This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful.
NOTE This will allow anybody to login to the machine through the rstudio interface and run any arbitrary code. The login used however is the same as the system login/password.
These are specific notes for installing PEcAn on Mac OSX and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed.
-
This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful.
There are two flavors of BETY, PHP and RUBY. The PHP version allows for a minimal interaction with the database while the RUBY version allows for full interaction with the database.
-
-
23.5.4.2.1 PHP version
-
The php version comes with PEcAn and is already configured.
-
-
-
23.5.4.2.2 RUBY version
-
The RUBY version requires a few extra packages to be installed first.
This page contains instructions on how to download and install ecosystem models that have been or are being coupled to PEcAn.
-These instructions have been tested on the PEcAn unbuntu VM. Commands may vary on other operating systems.
-Also, some model downloads require permissions before downloading, making them unavailable to the general public. Please contact the PEcAn team if you would like access to a model that is not already installed on the default PEcAn VM.
You will see a new directory in scripts: US-UMB_I1850CLM45CN
-Enter this directory and run (you shouldn’t have to do this normally, but there is a bug with the python script and doing this ensures all files get to the right place):
not sure this was the right location, but wherever the executable is
-
You should begin to see output files that look like this:
-US-UMB_I1850CLM45CN.clm2.h0.yyyy-mm.nc (yyyy is year, mm is month)
-These are netcdf files containing monthly averages of lots of variables.
-
The lnd_in file in the run directory can be modified to change the output file frequency and variables.
Within CLM-FATES, to be able to build an executable we need to create a reference run. We’ll also use this reference run to grab defaults from, so we’ll be registering the location of both the reference case (location of executable, scripts, etc) and the reference inputs with the PEcAn database. To begin, copy reference run script from pecan
Edit reference case script to set NETCDF_HOME, CROOT (reference run case), DIN_LOC_ROOT (reference run inputs). Also, make sure DIN_LOC_ROOT exists as FATES will not create it itself. Then run the script
-
./create_1x1_ref_case.sh
-
Be aware that this script WILL ask you for your password on the NCAR server to download the reference case input data (the guest password may work, haven’t tried this). If it gives an error at the pio stage check the log, but the most likely error is it being unable to find a version of netcdf it likes.
-
Once FATES is installed, set the whole reference case directory as the Model path (leave filename blank) and set the whole inputs directory as an Input with format clm_defaults.
-
-
-
-
23.5.5.6 GDAY
-
Navigate to a directory you would like to store GDAY and run the following:
maespa.out is your executable. Example input files can be found in the inpufiles directory. Executing measpa.out from within one of the example directories will produce output.
-
MAESPA developers have also developed a wrapper package called Maeswrap. The usual R package installation method install.packages may present issues with downloading an unpacking a dependency package called rgl. Here are a couple of solutions:
Remote execution with PEcAn - Running analyses and generally working with external machines (HPC) in the context of PEcAn.
-
-
-
-
7.1 Additional web configuration
-
Additional settings for web configuration:
-
-
Web interface setup
-
Brown Dog
-
Advanced setup
-
-
[Sensitivity analysis] (TODO)
-
[Uncertainty analysis] (TODO)
-
-
Editing model configuration files
-
-
-
7.1.1 Web interface setup
-
There are few options which you can change via web interface.
-
To visit the configuration page either you can just click on the setups link on the introduction page alternatively can type <host>/setups/.
-
The list of configuration available
-
-
Database configuration : BETYdb(Biofuel Ecophysiological Traits and Yields database) configuration details, can be edited according to need.
-
Browndog configuration : Browndog configuration details, Used to connect browndog. Its included by default in VM.
-
FIA Database : FIA(Forest Inventory and Analysis) Database configuration details, Can be used to add additional data to models.
-
Google MapKey : Google Map key, used to access the google map by PEcAn.
-
Change Password : A small infomation to change the VM user password. (if using Docker image it won’t work)
-
Automatic Sync : If ON then it will sync the database between local machine and the remote servers. Still unders testing part might be buggy.
-
-
Still work on the adding other editing feature going on, this page will be updated as new configuration will be available.
-
-
-
7.1.2 Brown Dog
-
The Browndog service provides PEcAn with access to large and diverse sets of data at the click of a button in the format that PEcAn needs. By clicking the checkbox you will be using the Browndog Service to process data.
These analyses can be run through the web interface, but lack graphical interfaces and currently can only be configured throughthe XML settings. To run these analyses use the Edit pecan.xml checkbox on the Input configuration page. Eventually, these modules will be integrated into the web user interface.
All functions pertaining to Parameter Data Assimilation are housed within: pecan/modules/assim.batch.
-
For a detailed usage of the module, please see the vignette under pecan/modules/assim.batch/vignettes.
-
Hierarchical version of the PDA is also implemented, for more details, see the MultiSitePDAVignettepackage vignette and function-level documentation.
-
-
7.2.1.1pda.mcmc.R
-
This is the main PDA code. It performs Bayesian MCMC on model parameters by proposing parameter values, running the model, calculating a likelihood (between model output and supplied observations), and accepting or rejecting the proposed parameters (Metropolis algorithm). Additional notes:
-
-
The first argument is settings, followed by others that all default to NULL.settings is a list used throughout Pecan, which contains all the user options for whatever analyses are being done. The easiest thing to do is just pass that whole object all around the Pecan code and let different functions access whichever settings they need. That’s what a lot of the rest of the Pecan code does. But the flexibility to override most of the relevant settings in settings is there by providing them directly as arguments to the function.
-
The if(FALSE)… : If you’re trying to step through the function you probably will have the settings object around, but those other variables will be undefined. If you set them all to NULL then they’ll be ignored without causing errors. It is there for debugging purposes.
-
The next step calls pda.settings(), which is in the file pda.utils.R (see below). It checks whether any settings are being overridden by arguments, and in most cases supplies default values if it can’t find either.
-
In the MCMC setup section
-
-
The code is set up to allow you to start a new MCMC chain, or to continue a previous chain as specified in settings.
-
The code writes a simple text file of parameter samples at every iteration, which lets you get some results and even re-start an MCMC that fails for some reason.
-
The code has adaptive jump distributions. So you can see some initialization of the jump distributions and associated variables here.
-
Finally, note that after all this setup a new XML settings file is saved. The idea is that the original pecan.xml you create is preserved for provenance, and then periodically throughout the workflow the settings (likely containing new information) are re-saved with descriptive filenames.
-
-
MCMC loop
-
-
Periodically adjust jump distribution to make acceptance rate closer to target
-
Propose new parameters one at a time. For each:
-
-
First, note that Pecan may be handling many more parameters than are actually being targeted by PDA. Pecan puts priors on any variables it has information for (in the BETY database), and then these get passed around throughout the analysis and every step (meta-, sensitivity, ensemble analyses, etc.). But for PDA, you specify a separate list of probably far fewer parameters to constrain with data. These are the ones that get looped over and varied here. The distinction between all parameters and only those dealt with in PDA is dealt with in the setup code above.
-
First a new value is proposed for the parameter of interest.
-
Then, a new model run is set up, identical to the previous except with the new proposed value for the one parameter being updated on this run.
-
The model run is started, and outputs collected after waiting for it to finish.
-
A new likelihood is calculated based on the model outputs and the observed dataset provided.
-
Standard Metropolis acceptance criteria is used to decide whether to keep the proposed parameter.
-
Periodically (at interval specified in settings), a diagnostic figure is saved to disk so you can check on progress.
-
-
This works only for NEE currently
-
-
-
-
-
7.2.1.2pda.mcmc.bs.R
-
This file is basically identical to pda.mcm.R, but rather than propose parameters one at a time, it proposes new values for all parameters at once (“bs” stands for “block sampling”). You choose which option to use by specifying settings\(assim.batch\)method:
-* “bruteforce” means sample parameters one at a time
-* “bruteforce.bs” means use this version, sampling all parameters at once
-* “emulator” means use the emulated-likelihood version
-
-
-
7.2.1.3pda.emulator
-
This version of the PDA code again looks quite similar to the basic “bruteforce” one, but its mechanics are very different. The basic idea is, rather than running thousands of model iterations to explore parameter space via MCMC, run a relatively smaller number of runs that have been carefully chosen to give good coverage of parameter space. Then, basically interpolate the likelihood calculated for each of those runs (actually, fit a Gaussian process to it), to get a surface that “emulates” the true likelihood. Now, perform regular MCMC (just like the “bruteforce” approach), except instead of actually running the model on every iteration to get a likelihood, just get an approximation from the likelihood emulator. Since the latter step takes virtually no time, you can run as long of an MCMC as you need at little computational cost, once you have done the initial model runs to create the likelihood emulator.
-
-
-
7.2.1.4pda.mcmc.recover.R
-
This function is for recovering a failed PDA MCMC run.
-
-
-
7.2.1.5pda.utils.R
-
This file contains most of the individual functions used by the main PDA functions (pda.mcmc.*.R).
-
-
assim.batch is the main function Pecan calls to do PDA. It checks which method is requested (bruteforce, bruteforce.bs, or emulator) and call the appropriate function described above.
-
pda.setting handles settings. If a setting isn’t found, the code can usually supply a reasonable default.
-
pda.load.priors is fairly self explanatory, except that it handles a lot of cases and gives different options priority over others. Basically, the priors to use for PDA parameters can come from either a Pecan prior.distns or post.distns object (the latter would be, e.g., the posteriors of a meta-analysis or previous PDA), or specified either by file path or BETY ID. If not told otherwise, the code tries to just find the most recent posterior in BETY, and use that as prior for PDA.
-
pda.create.ensemble gets an ensemble ID for the PDA. All model runs associated with an individual PDA (any of the three methods) are considered part of a single ensemble. This function does is register a new ensemble in BETY, and return the ID that BETY gives it.
-
pda.define.prior.fn creates R functions for all of the priors the PDA will use.
-
pda.init.params sets up the parameter matrix for the run, which has one row per iteration, and one column per parameter. Columns include all Pecan parameters, not just the (probably small) subset that are being updated by PDA. This is for compatibility with other Pecan components. If starting a fresh run, the returned matrix is just a big empty matrix to fill in as the PDA runs. If continuing an existing MCMC, then it will be the previous params matrix, with a bunch of blank rows added on for filling in during this round of PDA.
-
pda.init.run This is basically a big wrapper for Pecan’s write.config function (actually functions [plural], since every model in Pecan has its own version). For the bruteforce and bruteforce.bs methods this will be run once per iteration, whereas the emulator method knows about all its runs ahead of time and this will be a big batch of all runs at once.
-
pda.adjust.jumps tweaks the jump distributions for the standard MCMC method, and pda.adjust.jumps.bs does the same for the block-sampled version.
-
pda.calc.llik calculates the log-likelihood of the model given all datasets provided to compare it to.
-
pda.generate.knots is for the emulator version of PDA. It uses a Latin hypercube design to sample a specified number of locations in parameter space. These locations are where the model will actually be run, and then the GP interpolates the likelihood surface in between.
-
pda.plot.params provides basic MCMC diagnostics (trace and density) for parameters being sampled.
-
pda.postprocess prepares the posteriors of the PDA, stores them to files and the database, and performs some other cleanup functions.
-
pda.load.data.r This is the function that loads in data that will be used to constrain the PDA. It’s supposed to be eventually more integrated with Pecan, which will know how to load all kinds of data from all kinds of sources. For now, it can do NEE from Ameriflux.
-
pda.define.llik.r A simple helper function that defines likelihood functions for different datasets. Probably in the future this should be queried from the database or something. For now, it is extremely limited. The original test case of NEE assimilation uses a heteroskedastic Laplacian distribution.
-
pda.get.model.output.R Another function that will eventually grow to handle many more cases, or perhaps be replaced by a better system altogether. For now though, it again just handles Ameriflux NEE.
-
-
-
-
7.2.1.6get.da.data.*.R, plot.da.R
-
Old codes written by Carl Davidson. Defunct now, but may contain good ideas so currently left in.
-
-
-
-
7.2.2 State data assimilation (SDA)
-
sda.enkf.R is housed within: /pecan/modules/assim.sequential/R
-
The tree ring tutorial is housed within: /pecan/documentation/tutorials/StateAssimilation
-
More descriptive SDA methods can be found at: /pecan/book_source/adve_user_guide_web/SDA_Methods.Rmd
-
-
7.2.2.1sda.enkf.R Description
-
This is the main ensemble Kalman filter and generalized filter code. Originally, this was just ensemble Kalman filter code. Mike Dietze and Ann Raiho added a generalized ensemble filter to avoid filter divergence. The output of this function will be all the of run outputs, a PDF of diagnostics, and an Rdata object that includes three lists:
-
-
FORECAST will be the ensemble forecasts for each year
-
ANALYSIS will be the updated ensemble sample given the NPP observations
-
enkf.params contains the prior and posterior mean vector and covariance matrix for each time step.
Load data means (obs.mean) and covariances (obs.cov) as lists with PEcAn naming and unit conventions. Each observation must have a date in YYYY/MM/DD format (optional time) associated with it. If there are missing data, the date must still be represented in the list with an NA as the list object.
-
Create initial conditions matrix (IC) that is state variables columns by ensemble members rows in dimension. sample.IC.MODEL can be used to create the IC matrix, but it is not required. This IC matrix is fed into write.configs for the initial model runs.
read.restart.MODEL - read model restart files corresponding to start.time and stop.time that you want to assimilate data into
-
Analysis - There are four choices based on if process variance is TRUE or FALSE and if there is data or not. See explaination below.
-
write.restart.MODEL - This function has two jobs. First, to insert adjusted state back into model restart file. Second, to update start.time, stop.time, and job.sh.
adjustment : [optional] TRUE/FLASE flag for if ensembles needs to be adjusted based on weights estimated given their likelihood during analysis step. The defualt is TRUE for this flag.
-
process.variance : [optional] TRUE/FLASE flag for if process variance should be estimated (TRUE) or not (FALSE). If TRUE, a generalized ensemble filter will be used. If FALSE, an ensemble Kalman filter will be used. Default is FALSE. If you use the TRUE argument you can set three more optional tags to control the MCMCs built for the generalized esnsemble filter.
-
nitrGEF : [optional] numeric defining the length of the MCMC chains.
-
nthin : [optional] numeric defining thining length for the MCMC chains.
-
nburnin : [optional] numeric defining the number of burnins during the MCMCs.
-
q.type : [optional] If the process.variance is set to TRUE then this can take values of Single, Site or PFT.
-
censored.data : [optional] logical set TRUE for censored state variables.
-
sample.parameters : [optional] TRUE/FLASE flag for if parameters should be sampled for each ensemble member or not. This allows for more spread in the initial conditions of the forecast.
-
state.variable : [required] State variable that is to be assimilated (in PEcAn standard format).
-
spin.up : [required] start.date and end.date for initial model runs.
-
NOTE: start.date and end.date are distinct from values set in the run tag because initial runs can be done over a subset of the full run.
-
forecast.time.step : [optional] In the future, this will be used to allow the forecast time step to vary from the data time step.
-
start.date : [optional] start date of the state data assimilation (in YYYY/MM/DD format)
-
end.date : [optional] end date of the state data assimilation (in YYYY/MM/DD format)
-
NOTE: start.date and end.date are distinct from values set in the run tag because this analysis can be done over a subset of the run.
-
-
-
-
7.2.2.6 Model Specific Functions for SDA Workflow
-
-
-
7.2.2.7 read.restart.MODEL.R
-
The purpose of read.restart is to read model restart files and return a matrix that is site rows by state variable columns. The state variables must be in PEcAn names and units. The arguments are:
-
-
outdir - output directory
-
runid - ensemble member run ID
-
stop.time - used to determine which restart file to read (in POSIX format)
var.names - vector with state variable names with PEcAn standard naming. Example: c(‘AGB.pft’, ‘TotSoilCarb’)
-
params - parameters used by ensemble member (same format as write.configs)
-
-
-
-
7.2.2.8 write.restart.MODEL.R
-
This model specific function takes in new state and new parameter matrices from sda.enkf.R after the analysis step and translates new variables back to the model variables. Then, updates start.time, stop.time, and job.sh so that start.model.runs() does the correct runs with the new states. In write.restart.LINKAGES and write.restart.SIPNET, job.sh is updated by using write.configs.MODEL.
-
-
outdir - output directory
-
runid - run ID for ensemble member
-
start.time - beginning of model run (in POSIX format)
new.state - matrix from analysis of updated state variables with PEcAn names (dimensions: site rows by state variables columns)
-
new.params - In the future, this will allow us to update parameters based on states (same format as write.configs)
-
inputs - model specific inputs from split.inputs.MODEL used to run the model from start.time to stop.time
-
RENAME - [optional] Flag used in write.restart.LINKAGES.R for development.
-
-
-
-
7.2.2.9 split.inputs.MODEL.R
-
This model specific function gives the correct met and/or other model inputs to settings\(run\)inputs. This function returns settings\(run\)inputs to an inputs argument in sda.enkf.R. But, the inputs will not need to change for all models and should return settings\(run\)inputs unchanged if that is the case.
start.time - start time for model run (in POSIX format)
-
stop.time - stop time for model run (in POSIX format)
-
-
-
-
7.2.2.10 sample.IC.MODEL.R
-
This model specific function is optional. But, it can be used to create initial condition matrix (IC) with # state variables columns by # ensemble rows. This IC matrix is used for the initial runs in sda.enkf.R in the write.configs.MODEL function.
-
-
ne - number of ensemble members
-
state - matrix of state variables to get initial conditions from
-
year - used to determine which year to sample initial conditions from
-
-
-
-
7.2.2.11 Analysis Options
-
There are four options depending on whether process variance is TRUE/FALSE and whether or not there is data or not.
-
-
If there is no data and process variance = FALSE, there is no analysis step.
-
If there is no data and process variance = TRUE, process variance is added to the forecast.
If there is data and process variance = FALSE, the Kalman filter is used and solved analytically.
-
-
-
-
7.2.2.12 The Generalized Ensemble Filter
-
An ensemble filter is a sequential data assimilation algorithm with two procedures at every time step: a forecast followed by an analysis. The forecast ensembles arise from a model while the analysis makes an adjustment of the forecasts ensembles from the model towards the data. An ensemble Kalman filter is typically suggested for this type of analysis because of its computationally efficient analytical solution and its ability to update states based on an estimate of covariance structure. But, in some cases, the ensemble Kalman filter fails because of filter divergence. Filter divergence occurs when forecast variability is too small, which causes the analysis to favor the forecast and diverge from the data. Models often produce low forecast variability because there is little internal stochasticity. Our ensemble filter overcomes this problem in a Bayesian framework by including an estimation of model process variance. This methodology also maintains the benefits of the ensemble Kalman filter by updating the state vector based on the estimated covariance structure.
-
This process begins after the model is spun up to equilibrium.
-
The likelihood function uses the data vector \(\left(\boldsymbol{y_{t}}\right)\) conditional on the estimated state vector \(\left(\boldsymbol{x_{t}}\right)\) such that
where \(\boldsymbol{R}_{t}=\boldsymbol{\sigma}_{t}^{2}\boldsymbol{I}\) and \(\boldsymbol{\sigma}_{t}^{2}\) is a vector of data variances. To obtain an estimate of the state vector \(\left(\boldsymbol{x}_{t}\right)\), we use a process model that incorporates a process covariance matrix \(\left(\boldsymbol{Q}_{t}\right)\). This process covariance matrix differentiates our methods from past ensemble filters. Our process model contains the following equations
where \(\boldsymbol{\mu}_{forecast_{t}}\) is a vector of means from the ensemble forecasts and \(\boldsymbol{P}_{forecast_{t}}\) is a covariance matrix calculated from the ensemble forecasts. The prior for our process covariance matrix is \(\boldsymbol{Q}_{t}\sim\mathrm{Wishart}(\boldsymbol{V}_{t},n_{t})\) where \(\boldsymbol{V}_{t}\) is a scale matrix and \(n_{t}\) is the degrees of freedom. The prior shape parameters are updated at each time step through moment matching such that
where we calculate the mean of the process covariance matrix \(\left(\bar{\boldsymbol{Q}_{t}}\right)\) from the posterior samples at time t. Degrees of freedom for the Wishart are typically calculated element by element where \(v_{ij}\) are the elements of \(\boldsymbol{V}_{t}\). \(I\) and \(J\) index rows and columns of \(\boldsymbol{V}\). Here, we calculate a mean number of degrees of freedom for \(t+1\) by summing over all the elements of the scale matrix \(\left(\boldsymbol{V}\right)\) and dividing by the count of those elements \(\left(I\times J\right)\). We fit this model sequentially through time in the R computing environment using R package ‘rjags.’
-
Users have control over how they think is the best way to estimate \(Q\). Our code will look for the tag q.type in the XML settings under state.data.assimilation which can take 3 values of Single, PFT or Site. If q.type is set to single then one value of process variance will be estimated across all different sites or PFTs. On the other hand, when q.type` is set to Site or PFT then a process variance will be estimated for each site or PFT at a cost of more time and computation power.
-
-
-
7.2.2.13 Multi-site State data assimilation.
-
sda.enkf.multisite function allows for assimilation of observed data at multiple sites at the same time. In order to run a multi-site SDA, one needs to send a multisettings pecan xml file to this function. This multisettings xml file needs to contain information required for running at least two sites under run tag. The code will automatically run the ensembles for all the sites and reformats the outputs matching the required formats for analysis step.
-
The observed mean and cov needs to be formatted as list of different dates with observations. For each element of this list also there needs to be a list with mean and cov matrices of different sites named by their siteid. In case that zero variance was estimated for a variable inside the obs.cov, the SDA code will automatically replace that with half of the minimum variance from other non-zero variables in that time step.
In general, the idea is that sending, running and monitoring an SDA job should all be done using two functions (SDA_remote_launcher and Remote_Sync_launcher). SDA_remote_launcher checks the XML settings defining the run, sets up the SDA on the remote machine, and then sends a qusb command for running the job. Remote_Sync_launcher, on the other hand, sits on the local machine and monitors the progress of the job(s) and brings back the outputs as long as the job is running.
-
SDA_remote_launcher sets up the job by copying a template SDA workflow R script and a bash file template that are ready for submission to the remote machine. This function checks the paths to all inputs including met, soil, site_pft and etc., testing whether they exists on the remote machine or not. If they do not exist, the function copies the missing paths over and replaces the settings accordingly. After submitting the bash script, the function returns the PID of the job writing the log file, allowing the Remote_Sync_launcher to monitor the progress of the run, checks to see if the job is still alive, and determines if sda.output.rdata has been updated since the last check or not.
-
Additionally, the Remote_Sync_launcher function follows the progress of the remote job by executing a nohup command on a template R script and keeps the R console open for further use. This R script, as mentioned above, constantly pings the given PID every 5 minutes and copies over the SDA output.
-
Several points on how to prepare your xml settings for the remote SDA run:
-1 - In the main pecan workflow.R, if you were able to generate pecan.TRAIT.xml, your settings are ready to be used for an SDA run. All you need to add is your state data assimilation tags.
-2 - Inside the xml setting an <outdir> flag needs to be included and point to a local directory where SDA_remote_launcher will look for either a sample.Rdata file or a pft folder.
-3 - You need to set your <host> tag according to the desired remote machine. You can learn more about this on the Remote execution with PEcAn section of the documentation. Please make sure that the <folder> tag inside <host> is pointing to a directory where you would like to store and run your SDA job(s).
-4 - Finally, make sure the tag inside the tag is set to the correct path on the remote machine.
-
-
-
7.2.4 Restart functionality in SDA
-
If you prefer to run your SDA analysis in multiple stages, where each phase picks up where the previous one left off, you can use the restart argument in the sda.enkf.multisite function. You need to make sure that the output from previous step exists in the SDA folder (in the outfolder), and the <start.date> is the same as <end.date> from the previous step. When you run the SDA with the restart parameter, it will load the output from the previous step and use configs already written in the run folder to set itself up for the next step. Using the restart argument could be as easy as :
Where the new settings, obs.mean and obs.cov contain the relevant information for the next phase.
-
-
-
7.2.5 State Data Assimilation Methods
-
By Ann Raiho
-
Our goal is build a fully generalizable state data assimilation (SDA) workflow that will assimilate multiple types of data and data products into ecosystem models within PEcAn temporally and spatially. But, during development, specifically with PalEON goals in mind, we have been focusing on assimilating tree ring estimated NPP and AGB and pollen derived fractional composition into two ecosystem models, SIPNET and LINKAGES, at Harvard Forest. This methodology will soon be expanded to include the PalEON sites listed on the state data assimilation wiki page.
-
-
7.2.5.1 Data Products
-
During workflow development, we have been working with tree ring estimated NPP and AGB and pollen derived fractional composition data products. Both of these data products have been estimated with a full accounting of uncertainty, which provides us with state variable observation mean vector and covariance matrix at each time step. These data products are discussed in more detail below. Even though we have been working with specific data products during development, our workflow is generalizable to alternative data products as long as we can calculate a state variable observation mean vector and covariance for a time point.
-
-
-
7.2.5.2 Tree Rings
-
We have been primarily been working with the tree ring data product created by Andria Dawson and Chris Paciorek and the PEcAn tree ring allometry module. They have developed a Bayesian model that estimates annual aboveground biomass increment (Mg/ha/yr) and aboveground biomass (Mg/ha) for each tree in a dataset. We obtain this data and aggregate to the level appropriate for the ecosystem model. In SIPNET, we are assimilating annual gross woody increment (Mg/ha/yr) and above ground woody biomass (Mg/ha). In LINKAGES, we are assimilating annual species biomass. More information on deriving these tree ring data products can be found in Dawson et al 201?.
-
We have been working mostly with tree data collected at Harvard Forest. Tree rings and census data were collected at Lyford Plot between 1960 and 2010 in three separate plots. Other tree ring data will be added to this analysis in the future from past PEON courses (UNDERC), Kelly Heilman (Billy’s Lake and Bigwoods), and Alex Dye (Huron Mt. Club).
-
-
-
7.2.5.3 Pollen
-
STEPPS is a Bayesian model developed by Paciorek and McLachlan 2009 and Dawson et al 2016 to estimate spatially gridded fractional composition from fossil pollen. We have been working with STEPPS1 output, specifically with the grid cell that contains Harvard Forest. The temporal resolution of this data product is centennial. Our workflow currently operates at annual time steps, but does not require data at every time step. So, it is possible to assimilate fractional composition every one hundred years or to assimilate fractional composition data every year by accounting for variance inflation.
-
In the future, pollen derived biomass (ReFAB) will also be available for data assimilation. Although, we have not discussed how STEPPS and ReFAB data assimilation will work.
-
-
-
7.2.5.4 Variance Inflation
-
*Side Note: Probably want to call this something else now.
-
Since the fractional composition data product has a centennial resolution, in order to use fractional composition information every year we need to change the weight the data has on the analysis. The basic idea is to downweight the likelihood relative to the prior to account for (a) the fact that we assimilate an observation multiple times and (b) the fact that the number of STEPPS observations is ‘inflated’ because of the autocorrelation. To do this, we take the likelihood and raise it to the power of (1/w) where ‘w’ is an inflation factor.
-
w = D * (N / ESS)
-
where D is the length of the time step. In our case D = 100. N is the number of time steps. In our case N = 11. and ESS is the effective sample size. The ESS is calculated with the following function where ntimes is the same as N above and sims is a matrix with the dimensions number of MCMC samples by number of state variables.
-
ESS_calc <- function(ntimes, sims){
- # center based on mean at each time to remove baseline temporal correlation
- # (we want to estimate effective sample size effect from correlation of the errors)
- row.means.sims <- sims - rowMeans(sims)
-
- # compute all pairwise covariances at different times
- covars <- NULL
- for(lag in 1:(ntimes-1)){
- covars <- c(covars, rowMeans(row.means.sims[(lag+1):ntimes, , drop = FALSE] * row.means.sims[1:(ntimes-lag), , drop = FALSE]))
- }
- vars <- apply(row.means.sims, 1, var) # pointwise post variances at each time, might not be homoscedastic
-
- # nominal sample size scaled by ratio of variance of an average
- # under independence to variance of average of correlated values
- neff <- ntimes * sum(vars) / (sum(vars) + 2 * sum(covars))
- return(neff)
- }
-
The ESS for the STEPPS1 data product is 3.6, so w in our assimilation of fractional composition at Harvard Forest will be w = 305.6.
-
-
-
7.2.5.5 Current Models
-
SIPNET and LINKAGES are the two ecosystem models that have been used during state data assimilation development within PEcAn. SIPNET is a simple ecosystem model that was built for… LINKAGES is a forest gap model created to simulate the process of succession that occurs when a gap is opened in the forest canopy. LINKAGES has 72 species level plant functional types and the ability to simulate some below ground processes (C and N cycles).
-
-
-
7.2.5.6 Model Calibration
-
Without model calibration both SIPNET and LINKAGES make incorrect predictions about Harvard Forest. To confront this problem, SIPNET and LINKAGES will both be calibrated using data collected at the Harvard Forest flux tower. Istem has completed calibration for SIPNET using a parameter data assimilation emulator contained within the PEcAn workflow. LINKAGES will also be calibrated using this method. This method is also generalizable to other sites assuming there is data independent of data assimilation data available to calibrate against.
-
-
-
7.2.5.7 Initial Conditions
-
The initial conditions for SIPNET are sampled across state space based on data distributions at the time when the data assimilation will begin. We do not sample LINAKGES for initial conditions and instead perform model spin up for 100 years prior to beginning data assimilation. In the future, we would like to estimate initial conditions based on data. We achieve adequate spread in the initial conditions by allowing the parameters to vary across ensemble members.
-
-
-
7.2.5.8 Drivers
-
We are currently using Global Climate Model (GCM) drivers from the PaLEON model intercomparison. Christy Rollinson and John Tipton are creating MET downscaled GCM drivers for the Paleon data assimilation sites. We will use these drivers when they are available because they are a closer representation of reality.
-
-
-
7.2.5.9 Sequential State Data Assimilation
-
We are using sequential state data assimilation methods to assimilate paleon data products into ecosystem models because less computation power is required for sequential state data assimilation than for particle filter methods.
-
-
-
7.2.5.10 General Description
-
The general sequential data assimilation framework consists of three steps at each time step:
-1. Read the state variable output for time t from the model forecast ensembles and save the forecast mean (muf) and covariance (Pf).
-2. If there are data mean (y) and covariance (R) at this time step, perform data assimilation analysis (either EnKF or generalized ensemble filter) to calculate the new mean (mua) and covariance (Pa) of the state variables.
-3. Use mua and Pa to restart and run the ecosystem model ensembles with new state variables for time t+1.
-
-
-
7.2.5.11 EnKF
-
There are two ways to implement sequential state data assimilation at this time. The first is the Ensemble Kalman Filter (EnKF). EnKF has an analytical solution, so the kalman gain, analysis mean vector, and analysis covariance matrix can be calculated directly:
-
- K <- Pf %*% t(H) %*% solve((R + H %*% Pf %*% t(H))) ## Kalman Gain
-
- mu.a <- mu.f + K %*% (Y - H %*% mu.f) # Analysis mean vector
-
- Pa <- (diag(ncol(X)) - K %*% H) %*% Pf # Analysis covariance matrix
-
-
The EnKF is typically used for sequential state data assimilation, but we found that EnKF lead to filter divergence when combined with our uncertain data products. Filter divergence led us to create a generalized ensemble filter that estimates process variance.
-
-
-
7.2.5.12 Generalized Ensemble Filter
-
The generalized ensemble filter follows generally the three steps of sequential state data assimilation. But, in the generalized ensemble filter we add a latent state vector that accounts for added process variance. Furthermore, instead of solving the analysis analytically like the EnKF, we have to estimate the mean analysis vector and covariance matrix with MCMC.
-
-
-
7.2.5.13 Mapping Ensemble Output to Tobit Space
-
There are some instances when we have right or left censored variables from the model forecast. For example, a model estimating species level biomass may have several ensemble members that produce zero biomass for a given species. We are considering this case a left censored state variable that needs to be mapped to normal space using a tobit model. We do this by creating two matrices with dimensions number of ensembles by state variable. The first matrix is a matrix of indicator variables (y.ind), and the second is a matrix of censored variables (y.censored). When the indicator variable is 0 the state variable (j) for ensemble member (i) is sampled. This allows us to impute a normal distribution for each state variable that contains ‘missing’ forecasts or forecasts of zero.
7.2.5.14 Generalized Ensemble Filter Model Description
-
Below is the BUGS code for the full analysis model. The forecast mean an covariance are calculated from the tobit2space model above. We use a tobit likelihood in this model because there are instances when the data may be left or right censored. Process variance is included by adding a latent model state (X) with a process precision matrix (q). We update our prior on q at each time step using our estimate of q from the previous time step.
-
tobit.model <- nimbleCode({
-
- q[1:N,1:N] ~ dwish(R = aq[1:N,1:N], df = bq) ## aq and bq are estimated over time
- Q[1:N,1:N] <- inverse(q[1:N,1:N])
- X.mod[1:N] ~ dmnorm(muf[1:N], prec = pf[1:N,1:N]) ## Model Forecast ##muf and pf are assigned from ensembles
-
- ## add process error
- X[1:N] ~ dmnorm(X.mod[1:N], prec = q[1:N,1:N])
-
- #agb linear
- #y_star[1:YN,1:YN] <- X[1:YN,1:YN] #[choose]
-
- #f.comp non linear
- #y_star[1:YN] <- X[1:YN] / sum(X[1:YN])
-
- ## Analysis
- y.censored[1:YN] ~ dmnorm(X[1:YN], prec = r[1:YN,1:YN]) #is it an okay assumpution to just have X and Y in the same order?
-
- #don't flag y.censored as data, y.censored in inits
- #remove y.censored samplers and only assign univariate samplers on NAs
-
- for(i in 1:YN){
- y.ind[i] ~ dconstraint(y.censored[i] > 0)
- }
-
- })
-
-
-
7.2.5.15 Ensemble Adjustment
-
Each ensemble member has a different set of species parameters. We adjust the updated state variables by using an ensemble adjustment. The ensemble adjustment weights the ensemble members based on their likelihood during the analysis step.
There are three diagnostics we have currently implemented: time series, bias time series, and process variance. The time series diagnostics show the data, forecast, and analysis time series for each state variable. These are useful for visually assessing variance and magnitude of change of state variables through time. These time series are also updated throughout the analysis and are also created as a pdf at the end of the SDA workflow. There are two types of bias time series the first assess the bias in the update (the forecast minus the analysis) and the second assess the bias in the error (the forecast minus the data). These bias time series are useful for identifying which state variables have intrinsic bias within the model. For example, if red oak biomass in LINKAGES increases at every time step (the update and the error are always positive), this would suggest that LINKAGES has a positive growth or recruitment bias for red oak. Finally, when using the generalized ensemble filter to estimate process variance, there are two additional plots to assess estimation of process variance. The first is a correlation plot of the process covariance matrix. This tells us what correlations are incorrectly represented by the model. For example, if red oak biomass and white pine biomass are highly negatively correlated in the process covariance matrix, this means that the model either 1) has no relationship between red oak and white pine and they should affect each other negatively or 2) there is a positive relationship between red oak and white pine and there shouldn’t be any relationship. We can determine which of these is true by comparing the process covariance matrix to the model covariance matrix. The second process variance diagnostic plot shows how the degrees of freedom associated with estimating the process covariance matrix have changed through time. This plot should show increasing degrees of freedom through time.
-
-
-
-
7.2.6 MultiSettings
-
(TODO: Under construction…)
-
-
-
7.2.7 Benchmarking
-
Benchmarking is the process of comparing model outputs against either experimental data or against other model outputs as a way to validate model performance.
-We have a suit of statistical comparisons that provide benchmarking scores as well as visual comparisons that help in diagnosing data-model and/or model-model differences.
-
-
7.2.7.1 Data Preparation
-
All data that you want to compare with model runs must be registered in the database.
-This is currently a step that must be done by hand either from the command line or through the online BETY interface.
-The data must have three records:
Model runs can be setup and executed
-- Using the PEcAn web interface online or with a VM (see setup)
-- By hand using the pecan.xml
-
-
-
7.2.7.3 The Benchmarking Shiny App
-
The entire benchmarking process can be done through the Benchmarking R Shiny app.
-
When the model run has completed, navigate to the workflow visualization Shiny app.
-
-
Load model data
-
-
Select the workflow and run id
-
Make sure that your model output is loading properly (i.e. you can see plots of your data)
-
-
Load benchmarking data
-
-
Again make sure that you can see the uploaded data plotted alongside the model output. In the future there will be more tools for double checking that your uploaded data is appropriate for benchmarking, but for now you may need to do the sanity checks by hand.
-
-
-
-
-
7.2.7.4 Create a reference run record
-
-
Navigate to the Benchmarking tab
-
-
The first step is to register the new model run as a reference run in the database. Benchmarking cannot be done before this step is completed. When the reference run record has been created, additional menus for benchmarking will appear.
-
-
-
-
-
7.2.7.5 Setup Benchmarks and metrics
-
-
From the menus select
-
-
The variables in the uploaded data that you wish to compare with model output.
-
The numerical metrics you would like to use in your comparison.
-
Additional comparison plots that you would like to see.
-
-
Note: All these selections populate the benchmarking section of the pecan.BENCH.xml which is then saved in the same location as the original run output. This xml is purely for reference.
-
-
-
7.2.7.5.1 Benchmarking Output
-
-
All benchmarking results are stored in the benchmarking directory which is created in the same folder as the original model run.
-
The benchmaking directory contains subdirectories for each of the datasets compared with the model output. The names of these directories are the same as the corresponding data set’s input id in BETY.
-
Each input directory contains benchmarking.output.Rdata, an Rdata file contianing all the results of the benchmarking workflow. load(benchmarking.output.Rdata) loads a list called result.out which contains the following:
-
-
bench.results: a data frame of all numeric benchmarking scores
-
format: a data frame that can be used to see how the input data was transformed to make it comparable to the model output. This involves converting from the original variable names and units to the internal pecan standard.
-
aligned.dat: a data frame of the final aligned model and input values.
-
-
All plots are saved as pdf files with names with “benchmark_plot-type_variable_input-id.pdf”
-
To view interactive results, naviage to the Benchmarking Plots tab in the shiny app.
The pecan.xml has an optional benchmarking section. Below are all the tags in the benchmarking section explained. Many of these field are filled in automatically during the benchmarking process when using the benchmarking shiny app.
-
The only time one should edit the benchmarking section by hand is for performing clone runs. See clone run documentation.
-
<benchmarking> settings:
-
-
ensemble_id: the id of the ensemble that you will be using - the settings from this ensemble will be saved in a reference run record and then ensemble_id will be replaced with reference_run_id
-
new_run: TRUE = create new run, FALSE = use existing run (required, default FALSE)
-
-
It is possible to look at more than one benchmark with a particular run.
-The specific settings related to each benchmark are in a sub section called benchmark
-
-
input_id: the id of the benchmarking data (required)
-
variable_id: the id of the variable of interest within the data. If you leave this blank, all variables that are shared between the input and model output will be used.
-
metric_id: the id(s) of the metric(s) to be calculated. If you leave this blank, all metrics will be used.
-
-
Example:
-In this example,
-- we are using a pre-existing run from ensemble_id = 1000010983 (new_run = FALSE)
-- the output will be compared to data from input_id = 1000013743, specifically two variables of interest: variable_id = 411, variable_id = 18
-- for variable_id = 411 we will perform only one metric of comparison metric_id = 1000000001
-- for for variable_id = 18 we will perform two metrics of comparison metric_id = 1000000001, metric_id = 1000000002
-
-
-
-
-### 2. Creating an Input Record
-Creating an input record requires some basic metadata about the file that is being ingested. Each entry field is briefly explained below.
-
-
-### 4. Formats_Variables Record
-The final step in the ingest process is to register a formats-variables record. This record links pecan variables with variables from the selected data.
-
-
-
-
-As can be seen in the figure the architecture leverages of two standard containers (in orange). The first container is postgresql with postgis (
