Reverse dependency `tidytable` fails due to type mismatch in a merge operation

[aitap]

Found while selectively checking reverse dependencies before the 1.18.2 release:

library(tidytable)
df1 <- tidytable(x = 1:3)
df2 <- tidytable(x = c(2, 3, 3), y = c("a", "b", "c"))
out <- nest_join(df1, df2)

Error in bmerge(i, x, leftcols, rightcols, roll, rollends, nomatch, mult,  : typeof x.x (double) != typeof i.x (integer)
Calls: nest_join -> left_join -> [ -> [.data.table -> bmerge
Execution halted

Bisects to 85a4cf5 (#7538).

[aitap]

This works:

library(data.table)
y <- data.table(as.double(2:3), list('foo','bar'))
x <- data.table(1:3)
y[x, on='V1']

But this doesn't:

(y <- tidytable(x = as.double(2:3), y = list('foo','bar')))
# # A tidytable: 2 × 2
#       x y
#   <dbl> <list>
# 1     2 <chr [1]>
# 2     3 <chr [1]>
(x <- tidytable(x = 1:3))
# # A tidytable: 3 × 1
#       x
#   <int>
# 1     1
# 2     2
# 3     3
y[x, on = 'x']
# Error in bmerge(i, x, leftcols, rightcols, roll, rollends, nomatch, mult,  :
#   typeof x.x (double) != typeof i.x (integer)

In the second example, x and y are not completely valid (missing the self-reference attribute, not growable). Installing the development version of tidytable fixes the problem, probably due to markfairbanks/tidytable#840.

[aitap]

The problem is due to bmerge → coerce_col → set changing the data.table in the call frame for coerce_col but not bmerge:

data.table/R/bmerge.R

Line 25 in 35dbf06

set(dt, j=col, value=cast_with_attrs(dt[[col]], cast_fun))

Should bmerge add a check for selfrefok and setalloccol before calling coerce_col?

[aitap]

A more systematic check for internal use of set() reveals that almost all calls are with a properly intialised data.table, usually created a few lines above, or at least something that is not shared with the caller. The only suspect uses are in setdroplevels

data.table/R/fdroplevels.R

Line 26 in 35dbf06

set(x, i=NULL, j=nx, value=fdroplevels(x[[nx]], exclude=exclude))

and bmerge:

data.table/R/bmerge.R

Line 25 in 35dbf06

set(dt, j=col, value=cast_with_attrs(dt[[col]], cast_fun))

Indeed, setdroplevels silently doesn't work on invalid data.tables:

x <- structure(list(factor('a', levels = letters)), class = c('data.table', 'data.frame'), names = 'x')
setdroplevels(x)
levels(x$x)
#  [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s"
# [20] "t" "u" "v" "w" "x" "y" "z"

Should we:

• make the behaviour of set() closer to how it used to work, only requiring setalloccol when changing the number of columns, or
• change setdroplevels and coerce_col to propagate the re-created data.table to the caller?