tidyevaluation.Rmd

---
title: "![](img/header.png) Intermediate R"
author: "Brendan Clarke"
date: "2022-10-22"
always_allow_html: true
output:
  html_document: 
    toc_depth: 2
    toc_float:
      collapsed: no
    fig_height: 2
  word_document:
    toc: yes
    toc_depth: '2'
  pdf_document:
    toc: yes
    toc_depth: '2'
    pandoc_args: ["--extract-media", "."]
---

```{r echo=F, warning=F, message=F}
knitr::opts_chunk$set(echo = TRUE, warning = F, message = F, results = 'asis')
install.packages(setdiff("pacman", rownames(installed.packages())))
library("pacman")
p_load(purrr, tidyr, ggplot2,dplyr,lubridate, stringr,NHSRdatasets, rlang, glue, knitr)

# function to add new window links
link<-function(text,url) {
  target<-c('{target="_blank"}')
  glue('[{text}]({url}){target}')
}
```

# tidy evaluation {.tabset}

Tidy evaluation is a `r link("tidyverse-specific approach to programming", "https://dplyr.tidyverse.org/articles/programming.html")`. This training session introduces two different elements of tidy evaluation:

+ tidyselect, which are tidyverse-specific tools for working with columns
+ data masking, which are tools for using data as variables, and vice versa

It is part of a wider training programme designed as next steps for R users who have moved beyond introductory training. This programme covers:

+ iteration
+ functions
+ dplyr
+ purr

This tidyselect training can be taken as a stand-alone session, or integrated into a more involved programme with the other sessions. You can find further details on the `r link("TURAS page", "https://learn.nes.nhs.scot/64644")`, join the `r link("KIND Learning Network", "https://forms.office.com/r/WQdd6HSCEW")` for social learning support, or contact [Brendan Clarke](mailto: brendan.clarke2@nhs.scot?subject=Intermediate R training) with any queries, comments, or suggestions. You can also find the source code for this training on the `r link("GitHub repository", "https://github.com/bclarke-nes/Intermediate-R")`.

## Introduction

```{r echo=F}
# function to display head() using kable()
display <- function(df) {
  if(nrow(df) == 0 | ncol(df) == 0) {cat("No data to display")}
  else{
    df %>%
    head() %>%
    kable()
  }
}

```

This is a session of two halves. The tidyverse brings lots of powerful tools for working with data to R. The first half of the session is about showing the advantages of these tools. We'll look in detail at tidyselection, which is at the heart of the tidyverse approach to selecting data. We'll show how using tidyselect can simplify and speed-up your writing of code.

However, in the second half, we'll concentrate on the downside of the tidyverse approach. This is particularly apparent when using tidyverse tools within your own functions. We'll discuss a group of strategies to ensure that your code works correctly when using tidyverse functions in more advanced ways.

## tidyselect {.tabset}

In this section, we'll look at tidyselect, which is a series of functions that allow you to specify columns by various  patterns. Not only does this give you lots of ways of simplifying and streamlining your code, but it's also a great way of making your code more portable - so that it works well inside functions, or across different data sets. 

We'll look at five groups of functions in this section. We'll start with the logical operators (like `!`) that are used across lots of R functions, before moving on to look at the additional functions found in tidyselect. There's quite a lot of information to absorb, so I would definitely recommend both using the manual pages (using `??tidyselect`) as a reference, and making sure that you try these functions out for yourself. Here are the functions that we'll be exploring:

```{r echo=F}
tribble(
  ~ Type,
  ~ Functions,
  "logical operators",
  "`:`, `!`, `&`, and |",
  "pattern matching",
  "`starts_with()`, `ends_with()`, `contains()`, `matches()`, `num_range()`",
  "match from a character vector",
  "`all_of()`, `any_of()`",
  "selection helpers for specific columns",
  "`everything()`, `last_col()`",
  "selecting with a function",
  "`where()`"
) %>%
  kable()
```

We'll use `stranded_data` dataset from the `NHSRdatasets` package for this section.

```{r}
stranded_data %>%
  display()
```

We're using a little helper function here, `display()`. You can find the code for this in the source - but really it just saves having to write `head() %>% kable()` every time we want to look at our data.

### logical operators

As usual, the `r link("R4DS chapter", "https://r4ds.had.co.nz/transform.html?q=&&#logical-operators")` is extremely helpful if this is completely new to you. The four operators that we'll introduce here are found all over R, and aren't in any way specific to the tidyverse, so we'll review them very briefly before moving on to the more idiosyncratic aspects of tidyselect. 

`:` allows you to select a range of columns. So to select the second to the fifth columns in the `stranded_data` set by name:

```{r}
stranded_data %>%
  select(age:hcop) %>%
  display()
```

Similarly, by index:

```{r}
stranded_data %>%
  select(2:5) %>%
  display()
```

`!` allows you to select the complement - in other words, everything but the specified column:

```{r}
stranded_data %>%
  select(!stranded.label) %>%
  display()
```

Use `c()` if you want to drop multiple columns using `!`:

```{r}
stranded_data %>%
  select(!c(age, stranded.label)) %>%
  display()
```

`&` lets you group together selection helpers. If you're selecting many columns by name, `&` isn't needed. Instead, you can just specify what you need to keep:

```{r}
stranded_data %>%
  select(stranded.label, hcop, age) %>%
  display()
```

Where `&` is really useful is in combination with pattern matching helpers, which we'll look at in more detail in the next subsection:

```{r}
stranded_data %>%
  select(starts_with("m") & ends_with("e")) %>%
  display()
```

`&` returns the columns where both of the patterns are matched. Compare and contrast the `|` operator, which gives the union of both helpers:

```{r}
stranded_data %>%
  select(starts_with("m") | ends_with("e")) %>%
  display()
```

### pattern matching {.tabset}
#### `starts_with()`

`starts_with()` matches columns by string. So to select all the columns that start with "care":

```{r}
stranded_data %>%
  select(starts_with("care")) %>%
  display()
```

By default. `starts_with()` is non-case sensitive:

```{r}
stranded_data %>%
  select(starts_with("Care"))  %>%
  display()
```

You can use the `ignore.case` argument to change this behaviour:

```{r}
stranded_data %>%
  select(starts_with("Care", ignore.case = FALSE))  %>%
  display()
```

#### `ends_with()`
`ends_with()` works as `starts_with`, except matching the endings of column names:

```{r}
stranded_data %>%
  select(age, ends_with("care")) %>%
  display()
```

We can match a range of endings using `c()`:

```{r}
stranded_data %>%
  select(ends_with(c("label", "safe", "care"))) %>%
  display()
```

These are returned in the order they appear inside the `c()`:

```{r}
stranded_data %>%
  select(ends_with(c("care", "label", "safe"))) %>%
  display()
```

#### `contains()`
`contains()` is for strings

```{r}
stranded_data %>%
  select(age, contains("care")) %>%
  display()
```

#### `matches()`
While `starts_with()` and `ends_with()` match supplied strings, `matches()` uses `r link("regular expressions", "https://cran.r-project.org/web/packages/stringr/vignettes/regular-expressions.html")` (regex). Regular expressions are a way of matching all kinds of strings by specifying a particular pattern. For example, you can use regex to search through a piece of text for all the email addresses that it contains. In pseudocode, the regex would work like this:

+ look for an @
+ look for the start and ends of the word that contains that @
+ make sure that the word doesn't contain any forbidden characters

Getting to grips with regex is a bit forbidding, because the syntax used is very concise and impenetrable. For instance, the above pseudocode translates into:

> [[:alnum:].-_]+@[[:alnum:].-]+

Luckily, there are some great tools to help the beginner - such as `r link("the Rstudio cheatsheet", "https://raw.githubusercontent.com/rstudio/cheatsheets/main/regex.pdf")`, or the interactive tool `r link("regular expressions 101", "https://regex101.com/")`. There are also a few very simple examples below that cover some of the common cases:

Match one of a group of letters using `[]`:
```{r}
stranded_data %>%
  select(matches("me[dn]"))  %>%
  display()
```
This returns columns containing either "med" or "men". 

To match characters at the start of a word use `^`:
```{r}
stranded_data %>%
  select(matches("^[am]"))  %>%
  display()
```

`.` matches any character:

```{r}
stranded_data %>%
  select(matches("me."))  %>%
  display()
```

To match columns containing a literal full stop use `\\` to escape the character:
```{r}
stranded_data %>%
  select(matches("\\."))  %>%
  display()
```

#### `num_range()`

`num_range()` selects a group of numbered columns from a character prefix and a range:

```{r}
# using the man page example:
data(billboard)

billboard %>%
  dplyr::select(num_range("wk", 10:15)) %>%
  display()

```

This kind of structure with many predictably-named columns often arises when using `pivot_wider()` to transform data. For example:

```{r}
ons_mortality %>%
  pivot_wider(names_from = week_no, values_from = counts, names_prefix = "week_") %>%
  select(1:12) %>%
  display()
```

(the `select(1:12)` here is just to keep our data narrow enough to display on screen)

To now select the numbered columns:

```{r}
ons_mortality %>%
  pivot_wider(names_from = week_no, values_from = counts, names_prefix = "week_") %>%
  select(1:12) %>%
  select(num_range("week_", 1:9)) %>%
  display()
```

If your numbered suffixes are not single digit (e.g. week_02 rather than week_2), you can add leading zeros to your range using the `width` argument:

```{r}
ons_mortality %>%
  pivot_wider(names_from = week_no, values_from = counts, names_prefix = "week_0") %>%
  select(num_range("week_", 1:9, width=2)) %>%
  display()
```


### matching character vectors {.tabset}

#### `all_of()`
`all_of()` matches all the column names found in some character vector. This can be very useful if you're working with many different sets of data with predictable column names. Let's set up a vector called `my_columns` to play with

```{r results='markup'}
my_columns <- names(stranded_data)[1:4]
my_columns
```

We can now select `all_of` the columns found in `my_columns`:
```{r}
stranded_data %>%
  select(all_of(my_columns)) %>%
  display()
```

As the name suggests, `all_of()` will match *all* the column names in the vector, and will give you an error if a name in the vector is missing from the column names. If we update our vector:

```{r results='markup'}
my_columns <- c(my_columns, "smoking_status")
my_columns
```

`all_of` will now cause an error, because the column `smoking_status` is not found in the data:
```{r results='markup'}
try(stranded_data %>%
  select(all_of(my_columns)) %>%
  display())
```

We can avoid this kind of error using `any_of`.

#### `any_of()`

As the name suggests, `any_of()` will match any column names that appear in the vector, and ignore those that do not. Using our updated vector of column names from above:

```{r} 
stranded_data %>%
  select(any_of(my_columns)) %>%
  display()
```

There's a great tip on the `all_of()` man page - `any_of()` is an especially clever way to drop columns from a tibble because calling it more than once does not cause an error:

```{r}
stranded_data %>%
  select(-any_of(my_columns)) %>%
  select(-any_of(my_columns)) %>%
  display()
```

### selection helpers for specific columns {.tabset}
#### `everything()`

`everything()` selects all columns. This is less useful in combination with `select()` itself, but simplifies some other functions well - particularly with `pivot_longer()`. One of the quirks of `pivot_longer()` is that you need to coerce your data so that the resulting columns don't mix data types. So we'll start with one of the stock R datasets to keep our code simple. Let's have a quick look at that dataset first:

```{r}
mtcars %>%
  display()
```

Now we'll reshape that wide data into a long, narrow, format using `pivot_longer()`:

```{r}
mtcars %>% 
  pivot_longer(everything()) %>%
  display()
```

This kind of reshaping is useful when, for example, we want to plot our entire dataset to look at the distribution of the variables: 

```{r fig.height=4, fig.width=6}
mtcars %>% 
  pivot_longer(everything()) %>%
  ggplot() +
  geom_histogram(aes(x=value, fill=name)) +
  facet_wrap(~ name, scales="free") + 
  theme(legend.position="none")
```

For our LOS_model data, we'll need to trim the organisation names in LOS_model so that our values column is purely numeric:

```{r}
LOS_model %>%
  mutate(Organisation = as.integer(str_replace_all(Organisation, "Trust", ""))) %>%
  pivot_longer(everything()) %>%
  display()
```

Then we can do the same kind of `facet_wrap()` graph as above:

```{r}
LOS_model %>%
  select(Age, LOS) %>%
  pivot_longer(everything()) %>%
  ggplot() +
  geom_histogram(aes(x=value, fill=name)) +
  facet_wrap(~ name, scales="free") + 
  theme(legend.position="none")
```

#### `last_col()`
Finally, `last_col()` selects the last column:

```{r}
stranded_data %>%
  select(last_col()) %>%
  display()
```

You can also specify an offset. This is zero-indexed, so `last_col()` is equivalent to `last_col(0)`:
```{r}
stranded_data %>%
  select(last_col(2)) %>% # selecting the column two before the last column
  display()
```

And you can happily combine `last_col()` with other ways of selecting columns:

```{r}
stranded_data %>%
  select(age:last_col(3)) %>%
  display()
```

`last_col()` is more useful than it might appear at first, because the default behaviour of `mutate()` is to create new columns as the last column. This means that you can `mutate`, and then immediately `select` the `last_col()`:

```{r}
stranded_data %>%
  mutate(age_months = age * 12) %>%
  select(last_col(1), last_col()) %>%
  display()
```

### `where()`
There's a bit more to explain with `where()` than the other tidyselect helpers. Broadly, `where()` lets us use a function to match columns. Let's start simply. We'll use the base R function `is.character()`. This is simple - it returns `TRUE` when a column (or vector, but not whole tibble) contains character data. Let's look at the stranded.label column in stranded_data:


```{r results='markup'}
is.character(stranded_data$stranded.label)
```

This should return true. So we can see how we might use is.character as the basis of selecting all the character columns from this data using `where()`.

```{r}
# select all the character columns from stranded_data:
stranded_data %>% 
  select(where(is.character)) %>%
  display()
```


## Data masking {.tabset}

### Introduction

More so than in other programming languages, R functions bias towards helping the user do common tasks easily. One excellent example is the way that tidyverse functions (like dplyr) make assumptions about what users mean when they refer to variables. As an example:

```{r}
stranded_data %>% 
  select(age) %>%
  display()
```

When we specify the age column in this select function, we don't need to tell R that we specifically mean the age column in the `stranded_data` tibble. That's very helpful, because it saves us having to specify that we want to refer to a specific column in a specific tibble each time we write a line of dplyr. Even if we create another tibble that also has an age column...

```{r}
new_stranded_data <- stranded_data %>%
  select(stranded.label, age)
```

... we can still just refer to the age column of the original `stranded_data` without any risk of confusion. This simplification - which we'll call **data masking** - is a great advantage of using the pipe, and most of the time data masking just works without giving rise to any problems at all. For example, we can write a vector of column names, and then pass it to `select()`, and R will figure out that we want to use those names as column names without any extra effort on our part:

```{r}
my_cols <- c("age", "care.home.referral", "medicallysafe")

stranded_data %>%
  select(any_of(my_cols)) %>%
  display()

```

That means that cases where data masking goes wrong can be enormously frustrating, because most of the time we don't have to think about what are code is *really* doing very often at all. Here's an example of such a problem in a function that picks a specified column from `stranded_data` and displays it:

```{r}
column_displayer <- function(col_name) {
  stranded_data %>%
  select(any_of(col_name)) %>%
  display()
}
```

If we try to use this `column_displayer()` function in the normal way, we'll receive an error:

```{r results='markup'}
try(column_displayer(age))
```

Okay, so we can dodge this error in this case by quoting the column name when we supply it as an argument:

```{r}
column_displayer("age")
```

But this non-standard `r link("kludge", "https://en.wiktionary.org/wiki/kludge#English")` leads to trouble when we want to, for instance, use `column_displayer()` inside another function. A stronger approach is to adjust our function code in the first place, so that we don't have to call our function in a non-standard way (why write *age* in some functions, but *"age"* in others to refer to the same thing).

In this section, we'll give a bit of helpful theoretical background about data masking. We'll then go on to look at four ways of resolving some of the difficulties that data masking can cause.

### Background

The `r link("rlang page on data-masking", "https://rlang.r-lib.org/reference/topic-data-mask.html")` is very helpful here in setting out a key distinction between kinds of variables that we've previously been using synonymously:

+ env-variables (things you create with assignment)
+ data-variables (e.g. imported data in a tibble)

For beginners, this distinction is not that important, particularly because tidyverse functions do lots of helpful blurring between these different types of variable. Note that many base R functions do often require the user to bear this distinction in mind. For instance, in base R you would specify a data variable differently from an environment variable:

```{r eval=F}
mtcars$cyl      # a data variable
cyl <- c(4,6,8) # an environment variable
```

Whereas in tidyverse, you can:

```{r eval=F}
mtcars %>%
  select(cyl) # specifying a data variable like an environment variable inside select
```

Most of the time, data masking doesn't cause any problems. However, when you start wanting to include tidyverse functions inside other functions, that blurring raises a problem. We won't give much of an explanation as to the reasons for this, although do read `r link("this introduction to the topic", "https://rlang.r-lib.org/reference/topic-data-mask.html")` and `r link("this more detailed account", "https://rlang.r-lib.org/reference/topic-data-mask-ambiguity.html")` if you are interested in the technical aspects. Here, we'll concentrate on four strategies for resolving these kind of data masking problems. These strategies are:

```{r echo=F}
tribble(
  ~Problem, ~Solution,
  "data-variable in a function argument", "**embracing** with `{{var}}`",
  "env-variable in a vector", "`.data[[var]]` and `.env[[var]]` **pronouns**", 
  "variables in output", "**injection** with **`:=`**",
  "complex cases", "**quasiquotation** with the injection operator `!!`"
) %>% kable()

```


### Embracing

If you want to use a data variable in the argument of a function, you need to ``{{embrace}}`` the argument. Here's some code to produce a rounded mean of a column in `ae_attendances` in cases where breaches are over 100:

```{r results='markup'}
ae_attendances %>%
  filter(breaches > 100) %>%
  pull(attendances) %>%
  mean() %>%
  round(2)
```

We can generalise this to a function, which won't work properly:

```{r results='markup'}
ae_means <- function(colname) {
  ae_attendances %>%
    filter(breaches > 100) %>%
    pull(colname) %>%
    mean() %>%
    round(2)
}

try(ae_means(breaches))
```

However, if we embrace the argument in the `pull()` call:

```{r results='markup'}
ae_means <- function(colname) {
  ae_attendances %>%
    filter(breaches > 100) %>%
    pull({{colname}}) %>%
    mean() %>%
    round(2)
}
```

We can use that new function in a standard way:

```{r results='markup'}
ae_means(breaches)
ae_means(admissions)

map(ae_attendances %>% select(where(is.numeric)) %>% names(), ae_means)
```

Slightly confusingly, this practice is also referred to as `r link("*tunneling* data variables", "https://www.tidyverse.org/blog/2020/02/glue-strings-and-tidy-eval/")`.

#### Exercise

Take the `ae_means` function, update it so that we can specify:

+ different values for the `filter` cut-off
+ and different columns to apply that cut-off to
+ then use the supplied `pmap` code to test your function on the supplied tibble `testo`

```{r results='markup'}

testo <- expand_grid(c("breaches", "admissions", "attendances"), c("attendances"), 5^(0:3))

#pmap(testo, ~ ae_means(..1, ..2, ..3))

```


### Pronouns

If you want to use an variable that comes from a character vector, then use **pronouns**. Pronouns allow you to specify how a variable should be interpreted. If we create an atomic vector:

```{r}
variable <- c("type")
```

We might then try to use this variable inside `count()`, but there are horrors there:
```{r results='markup'}
try(ae_attendances %>%
  count(variable) %>%
  display())
```

If we now add the `.data[[]]` pronoun:

```{r}
ae_attendances %>%
  count(.data[[variable]]) %>% 
  display()
```

The `.env[[]]` pronoun works in a similar way. Imagine that we happen to have an env-variable that shares the name of one of our data-variables:

```{r}
attendances <- 800
```

If we try to use it to filter our data, we'll run into a problem:

```{r}
ae_attendances %>%
  filter(breaches >= attendances) %>% 
  display()
```

(this is based on an actual problem I manufactured for myself while writing the functions training)

This gives an unexpected result, because there are *definitely* cases where we have more than 800 breaches. In fact we have something like `r nrow(ae_attendances %>% filter(breaches >= 800))` cases with more than 800 breaches. And what's going on here is that there's an ambiguity - which `attendances` do we mean? This is where pronouns come in, by allowing us to be precise about where the variable is coming from:

```{r}
ae_attendances %>%
  filter(.data[["breaches"]] >= .env[["attendances"]]) %>% 
  arrange(breaches) %>%
  display()
```
(you can get away, in this case, without the `.data[[]]`, but included here as an extra example)


### Injection

**`:=`** lets you inject variables into your output. For example, to ensure that the name of a new summary column matches a supplied column name in a function, we can inject the variable into the newly-created column name:

```{r}
col_means <- function(column, cutoff) {

ae_attendances %>%
  filter({{column}} > {{cutoff}}) %>%
  group_by(type) %>%
  summarise("mean_{column}" := round(mean(.data[[column]]), 1)) %>% 
  display()
}

```

The column name is created using `glue()` syntax. `glue()` is a neat replacement for base-R tools like `paste0()`:

```{r}
column <- c("breaches")
cutoff <- 400

cat(paste0("This is how we'd include the column (", column, "), and the cutoff (", cutoff, ") in Rmarkdown using `paste0`  \n  \n"))

cat(glue("This is how we'd include the column ({column}), and the cutoff ({cutoff}) in Rmarkdown using `glue`")) # easier to read

```

The column name is then injected using the `:=` operator. When we call our `col_means()` function, the supplied column name is injected into the new summary column:

```{r}
col_means("attendances", 400)
```

Similar injections can be applied across a range of dplyr functions. We'll demonstrate these below using a vector containing the new column name, but injection is most useful when included as part of a function that you might want to apply across several different aspects of your data:

```{r}

new_column_name <- c("Steve")

ae_attendances %>% 
  mutate("{new_column_name}" := round(attendances ^ 0.5, 2) ) %>%
  display()

ae_attendances %>% 
  rename("{new_column_name}" := attendances) %>%
  display()

ae_attendances %>% 
  group_by(org_code) %>%
  summarise("{new_column_name}" := n()) %>%
  arrange(.data[[new_column_name]]) %>%
  slice(1:10) %>%
  ggplot() +
  geom_col(aes(x=org_code, y=.data[[new_column_name]])) +
  ggtitle(glue("{new_column_name} by org_code"))
  
```

### Quasiquotation

Informally, the `:=` operator that we explored in the previous subsection functions behaved *as if* it were adding quotes around the variable that was passed to it. That meant that we could pass a quoted variable to a function, and yet return the result as expected:

```{r}
quoted_variable <- "Steve"

ae_attendances %>% 
  rename("{quoted_variable}" := attendances) %>%
  display()
```

What if we also need to use this variable in an unquoted way? For example, say we now want to use `select()` to pick out our `r quoted_variable` column? 

In a function where an argument is supplied quoted, you can unquote it with `!!`:

```{r}
ae_attendances %>% 
  rename("{quoted_variable}" := attendances) %>%
  select(!!quoted_variable) %>%
  display()
```

That gives us a useful and clear way of thinking about quasiquotation. To borrow the description from the manual page:

> Quasiquotation is the combination of quoting an expression while allowing immediate evaluation (unquoting) of part of that expression. (`r link("rlang quasiquotation manual page", "https://www.rdocumentation.org/packages/rlang/versions/0.2.1/topics/quasiquotation")`)

And the strength of using quasiquotation is that it grants lots of scope for handling variables in comparatively complicated function. For example, if we want to create a function to take a supplied tibble and column name, and generate a bit of Rmarkdown with a header and summary of that column, quasiquotation (and injection) allow us to wrangle our variables so that they are compatible with the tidyverse functions that we'd like to use:

```{r}
distinct_entries <- function(df, col_name){
  
  cat(glue("#### Results for {col_name}:  \n  \n")) # using glue syntax
  
  df %>% 
    select(!!sym(col_name)) %>% # using quasiquotation to select the supplied column
    rename("distinct_{col_name}" := col_name) %>% # using injection to rename the column
    distinct() %>%
    display()
}

distinct_entries(ae_attendances, "org_code")
```