Northwestern University Studies in Russian Literature, Philosophy, and Religious Thought

This repository contains website (journal/) and production (production/) files for Northwestern University Studies in Russian Literature, Philosophy, and Religious Thought.

RPLRT is a yearly or twice-yearly journal.

To start working on a new issue, create a new branch and deploy it to a staging environment by connecting the branch in AWS amplify.

The workflow for publishing an issue involves Pandoc, Hugo, and the py_bats automation scripts. The workflow is described in detail below.

Get Started

Clone this repository to your computer. If you're unfamiliar with Git on the command line, I recommend installing GitHub Desktop for managing git repositories on your computer.

You'll also need a text editor to work with these files. I recommend Visual Studio Code but any plain text editor will work.

• Download the .docx file
• Save the .docx in the relevant production/ folder At the end of this process, your directory structure should look like this:

py_bats/production
├── 1 # Article 1
|   ├── article1title.docx # Article 1
├── 2 # Article 2
|   ├── article2title.docx # Article 2
├── 3 # Article 3
|   ├── article3title.docx # Article 3

You are now ready to start making Markdown files.

Producing Markdown from Manuscripts

We use Pandoc to convert .docx files to .md and .xml formats. Pandoc is a command line program. Install Pandoc for your operating system.

This is an excellent tutorial on using Pandoc and Markdown: https://doi.org/10.46430/phen0041

We have automated the process of converting .docx files to .md files with a script called docx2md.py. This script is in the py_bats/ folder.

docx2md.py searches all subdirectories of the production/ folder for .docx files and converts them to .md files. You will now see a new .md file in each subdirectory.

Convert Manuscripts from Microsoft Word to Markdown

Let's say you downloaded a file called manuscript.docx from OJS and saved it in the production/manuscripts/ folder.

You open a terminal and navigate to the manuscripts directory. You can use Pandoc on the command line convert the manuscript to Markdown with this command:

pandoc manuscript.docx --wrap none --output manuscript.md

This will add a markdown file called manuscript.md in your current directory.

There are hundreds of options you can specify for each conversion. The --wrap none option is a personal preference, meaning Pandoc will not automatically wrap text at 72 characters in width.

Ripping references from manuscripts

Once we have the entire manuscript in Markdown, we can use a regex pattern to extract the references from the manuscript. This is done with the strip_references.py script. This script is in the py_bats/ folder.

The script will search all subdirectories of the production/ folder for .md files and extract the references from each file. You will now see a new .txt file in each subdirectory with each reference on a new line. (You may be required to do some manual cleaning on this, but it should be minimal.)

Converting plaintext citations to biblatex citations

We use a GPT to convert plaintext citations to biblatex citations. This is done with the gpt4bibtex.py script. This script is in the py_bats/ folder.

⚠️ This script will take a very long time to run. It has to call the GPT API once for every citation: Once it has begun, step away from the computer and have a coffee ☕

Converting references.bib to Markdown

We need to convert the references.bib file created by the GPT script above to markdown so that we can add it to the article's metadata. This is done with the bib2md.py script. This script is in the py_bats/ folder.

Add the end of this process, you will have a references.md file in each subdirectory. This file contains the references in YAML front-matter format.

Add Metadata to the Markdown file

We have automated the process of adding metadata to the Markdown files with a script called ref_in_article.py. This script is in the py_bats/ folder.

Article metadata fields are based on Pandoc's article.jats_publishing template. This metadata can be manually filled out on a per-article basis using a text editor.

Here's an empty metadata template for an article:

---
weight: 
title: ''
subtitle: ''
article:
    type: research-article # book-review, brief-report, research-article
    doi: ''
    heading: '' # Original Article, Brief Report, Book Review
    volume: ''
    issue: ''
    fpage: 
    lpage: 
    filename: ''
author:
    -   orcid: 
        surname: ''
        given_name: ''
        name: '' # First Name Last Name
        email: ''
        affiliation: 
            organization: ''
            department: ''
            isni: ''
            ror: '' # info available at https://ror.org/
            bio: |
                
_date:
    day: ''
    month: ''
    year: ''
abstract: |
    
tags:
    - 
---

Here's what each field means:

• weight: the order in which the article should appear in the table of contents

• title: title of the article

• subtitle: subtitle of the article

• article:

  • type: type of article, one of: book-review, brief-report, or research-article
  • doi: doi for the article, represented as a number (not a URL)
  • heading: type of article, one of: Original Article, Brief Report, Book Review
  • volume: the volume number
  • issue: the issue number
  • fpage: page number first page of the article
  • lpage: page number of the last page of the article
  • filename: the name of the pdf file for the article

• author:

  • orcid: Unique identifier for the author, from ORCID
  • surname: last name
  • given_name: first name (and middle name, if applicable)
  • name: full name (first name last name)
  • email: email from OJS
  • affiliation:
    • organization: college or university name, usually
    • department: name of the department, if available
    • ror: unique identifier from the Research Organization Registry
    • isni: unique identifier, also available from the Research Organization Registry
    • bio: biographical statement, usually in the manuscript

• _date:

  • day: day of publication
  • month: month of publication
  • year: year of publication

• abstract: article abstract

• tags: list of keywords supplied by the author (in OJS or manuscript)

We automated this process with a script called add_metadata.py. This script is in the py_bats/ folder.

Bibliographic Citations

Bibliographic citations are data that need to be created from text delievered in manuscripts. We can leverage Pandoc to help structure that data and Hugo to display the data in ways for tools like Google Scholar and OpenAlex to parse. See Helpful Pandoc documentation on Citations.

This process could be supported with student assistance. For each article, someone should review the References section and do the following (Install Zotero as a desktop app and browser extension):

• Create a collection in Zotero for the article
• Check each citation to find a record of it, ideally from the publisher's website
  • If this data is not available on the web, the data can be entered manually in Zotero Desktop app
• Pull the citation data from the website, using the Zotero browser extension, into the Zotero collection
• Check to see that the data looks accurate, correct any bad information

When this collection has been made, export the collection to BibTeX format, which will produce a .bib file.

Use pandoc to convert this file format to Markdown:

pandoc references.bib -s -f biblatex -t markdown -o references.md

Although this will produce a Markdown file, the references will actually be formatted as YAML front-matter. We can append this data (all of the items in the references: array) to the article's Markdown file, like what has been done here for example.

When the website is built, Hugo will read this data and produce <meta> tags for each reference so that these references can be linked by indexing services, like Google Scholar.

Producing JATSXML from Markdown

Articles are "published" when the HTML and PDF are available to the public for download. The JATSXML is for the eventual distribution to database and indexing services, like PubMed Central. See this ticket and this ticket for more information.

Production happens in the production folder. It's essentially a messy workbench for converting files, cleaning them up, and getting them ready for the website. JATSXML is produced from Markdown with Pandoc.

The files currently in the production/jats folder are sample outputs from earlier tests. The article.jats_publishing.xml file in production/templates is used by pandoc to convert Markdown to JATS. There is also journal-level metadata that is combined with article-level metadata that is defined in the jats.yml configuration.

An example conversion of a markdown file would look like this:

pandoc article.md --to jats_publishing+element_citations --template ./templates/article.jats_publishing.xml --metadata-file ./config/jats.yml --output article.jats.xml

These are what the pandoc options mean:

• --to jats_publishing+element_citations: convert to JATS XML with the element_citations extension
• --template ./templates/article.jats_publishing.xml: use the locally supplied template instead of the default template
• --metadata-file ./config/jats.yml: add metadata from config/jats.yml
• --output article.jats.xml: generate a file called article.jats.xml

There is a script in production/bin for producing different output formats from multiple files at once. This hasn't been used yet, so consider it a draft.

Setting up the Next Issue

All of the article files will be organized within the bulletin/content/article directory by volume, issue, and article folders. The basic directory structure is this: bulletin/content/article/[volume]/[issue]/[article]

Here are examples from the inaugural issue:

• Volume 1: bulletin/content/article/1
• Issue 1-2: bulletin/content/article/1/1-2
• Article 1: bulletin/content/article/1/1-2/1

This organizational pattern should be replicated for each additional issue and volume. I created a helper script to automate the generation of these files and folders (with boilerplate metadata in each file) to speed things up.

Adding Markdown to the Website

Website updates happen in the bulletin folder. Markdown files created by during the production process can be copied over to the /bulletin/content/article/[volume_number]/[issue_number]/[article_number] folder. The markdown files contain two elements: the YAML front-matter (i.e. metadata) and the full-text of the article. The YAML is written by hand and the markdown is generated by Pandoc before manual QA checks for things like headings and figures.

Process:

• Copy the files from production to their place in the /bulletin directory
• Create a commit for the new articles in the git repository
• Push the commit to the github repository
• AWS will build and deploy a new version of the site

Adding figures and PDFs to the Website

We use Leaf Bundles to organize Page Resources in the Hugo website.

The organization of an article looks like this:

├── 1-2
│   ├── 2
│   │   ├── index.md
│   │   ├── figure1.jpg
│   │   ├── figure2.png
│   │   └── 2769-2124_v01_i01-02_p19-45.pdf

All that needs to happen is for the files to be added to the folders containing the article's markdown, checked into the repository, pushed to github, and the templates will find these files and add them to the website during the build process.

TODO:

Add date: to each article's metadata in the markdown file. Figure out how to replace _date with date in the metadata. Don't mess with issued.# rplrt