driver() failing at module_aglu_LB141.ag_Fert_IFA_ctry_crop...

[robbieorvis]

Four of us are getting the same error running the data system on Windows from the main branch:

[1] "module_aglu_LB141.ag_Fert_IFA_ctry_crop"
Error in left_join():
! ... must be empty.
✖ Problematic argument:
• ignore_columns = "item"
Run rlang::last_error() to see where the error occurred.

rlang::last_error()
<error/rlib_error_dots_nonempty>
Error in left_join():
! ... must be empty.
✖ Problematic argument:
• ignore_columns = "item"

---

Backtrace:

1. gcamdata::driver()
2. dplyr:::left_join.data.frame(...)
   Run rlang::last_trace() to see the full context.

rlang::last_trace()
<error/rlib_error_dots_nonempty>
Error in left_join():
! ... must be empty.
✖ Problematic argument:
• ignore_columns = "item"

---

Backtrace:
▆

1. ├─gcamdata::driver()
2. │ └─gcamdata:::run_chunk(chunk, all_data[input_names]) at gcam_data_system/R/driver.R:364:6
3. │ ├─base::do.call(chunk, list(driver.MAKE, all_data)) at gcam_data_system/R/driver.R:16:2
4. │ └─gcamdata:::module_aglu_LB141.ag_Fert_IFA_ctry_crop("MAKE", <named list>)
5. │ └─... %>% select(iso, GTAP_crop, scaler) at gcam_data_system/R/zchunk_LB141.ag_Fert_IFA_ctry_crop.R:108:4
6. ├─dplyr::select(., iso, GTAP_crop, scaler)
7. ├─dplyr::left_join(...)
8. └─dplyr:::left_join.data.frame(...)
9. └─rlang::check_dots_empty0(...)

10. └─rlang::check_dots_empty(call = call)
    

11.   └─rlang:::action_dots(...)
    

12.     ├─base (local) try_dots(...)
    

13.     └─rlang (local) action(...)
    

[russellhz]

Hi @robbieorvis - are you using renv? Package versions are often the source of these sort of errors: https://jgcri.github.io/gcamdata/articles/getting-started/getting-started.html#optional-renv-currently-included-in-gcam-6-0-and-gcamdata-release

[robbieorvis]

That failed for all of us when we tried to follow the instructions online. Happy to try it again and share the output here.

[russellhz]

Probably best to get that working. To be clear, you are using the main GCAM branch from the gcam-core repository: https://github.com/JGCRI/gcam-core?

[robbieorvis]

For the data system, no. We've been using the main branch of the GCAM data system repository: https://github.com/JGCRI/gcamdata

[russellhz]

OK, that shouldn't matter, but good to know in case getting renv working doesn't solve the problem

[robbieorvis]

Okay, when trying renv::restore(), I get the following error:

ERROR: compilation failed for package 'curl'

• removing 'C:/Users/robbie/AppData/Local/Temp/Rtmpmuqk3W/renv-staging-7144522b73a9/curl'
  Error: install of package 'curl' failed

[russellhz]

You can try running devtools::install_version("curl", version = "4.3") -- you will need devtools installed first (install.packages("devtools"))

If you run into problems, can you also share your results when you run sessionInfo()?

[robbieorvis]

Okay, failed again. Here's the latest:

devtools::install_version("curl", version = "4.3")
Downloading package from url: https://cloud.r-project.org/src/contrib/Archive/curl/curl_4.3.tar.gz
Installing package into ‘C:/Users/robbie/Dropbox (Energy Innovation)/My Documents/GCAM Model FIles/gcam_data_system/renv/library/R-4.2/x86_64-w64-mingw32’
(as ‘lib’ is unspecified)
Warning message:
Project requested R version '4.1.0' but '4.2.2' is currently being used

• installing source package 'curl' ...
  ** package 'curl' successfully unpacked and MD5 sums checked
  ** using staged installation
  ** libs
  rm -f curl.dll callbacks.o curl.o download.o escape.o fetch.o form.o getdate.o handle.o ieproxy.o init.o interrupt.o multi.o nslookup.o reflist.o split.o ssl.o typechecking.o utils.o version.o winidn.o writer.o libwinhttp.dll.a winhttp.def
  "C:/PROGRA1/R/R-421.2/bin/x64/Rscript.exe" "../tools/winlibs.R" 7.64.1
  echo '#include <curl/curl.h>' | gcc -E -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -std=gnu99 -xc - | grep "^[ \t]CURLOPT_.," | sed s/,// > ../tools/option_table.txt
  cp winhttp64.def.in winhttp.def
  dlltool --as as -k --dllname winhttp.dll --input-def winhttp.def --output-lib libwinhttp.dll.a
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c callbacks.c -o callbacks.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c curl.c -o curl.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c download.c -o download.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c escape.c -o escape.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c fetch.c -o fetch.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c form.c -o form.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c getdate.c -o getdate.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c handle.c -o handle.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c ieproxy.c -o ieproxy.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c init.c -o init.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c interrupt.c -o interrupt.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c multi.c -o multi.o
  gcc -I"C:/PROGRA1/R/R-421.2/include" -DNDEBUG -I../windows/libcurl-7.64.1/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c nslookup.c -o nslookup.o
  nslookup.c:17:13: error: conflicting types for 'inet_ntop'
  17 | const char *inet_ntop(int af, const void *src, char *dst, socklen_t size);
  | ^~~~~~~~~
  In file included from nslookup.c:16:
  c:\rtools42\x86_64-w64-mingw32.static.posix\include\ws2tcpip.h:401:35: note: previous declaration of 'inet_ntop' was here
  401 | WINSOCK_API_LINKAGE LPCSTR WSAAPI InetNtopA(INT Family, LPCVOID pAddr, LPSTR pStringBuf, size_t StringBufSize);
  | ^~~~~~~~~
  make: *** [C:/PROGRA1/R/R-421.2/etc/x64/Makeconf:253: nslookup.o] Error 1
  ERROR: compilation failed for package 'curl'
• removing 'C:/Users/robbie/Dropbox (Energy Innovation)/My Documents/GCAM Model FIles/gcam_data_system/renv/library/R-4.2/x86_64-w64-mingw32/curl'
• restoring previous 'C:/Users/robbie/Dropbox (Energy Innovation)/My Documents/GCAM Model FIles/gcam_data_system/renv/library/R-4.2/x86_64-w64-mingw32/curl'
  Warning message:
  In i.p(...) :
  installation of package ‘C:/Users/robbie/AppData/Local/Temp/Rtmpmuqk3W/remotes714425f3467d/curl’ had non-zero exit status

sessionInfo()
R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8 LC_CTYPE=English_United States.utf8
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C
[5] LC_TIME=English_United States.utf8

attached base packages:
[1] stats graphics grDevices datasets utils methods base

other attached packages:
[1] gcamdata_5.1 testthat_3.1.6

loaded via a namespace (and not attached):
[1] tidyselect_1.2.0 remotes_2.4.2 purrr_1.0.1 vctrs_0.5.2 generics_0.1.3
[6] miniUI_0.1.1.1 htmltools_0.5.4 usethis_2.1.6 utf8_1.2.3 rlang_1.0.6
[11] pkgbuild_1.4.0 later_1.3.0 pillar_1.8.1 urlchecker_1.0.1 glue_1.6.2
[16] withr_2.5.0 sessioninfo_1.2.2 lifecycle_1.0.3 stringr_1.5.0 devtools_2.4.5
[21] htmlwidgets_1.6.1 memoise_2.0.1 callr_3.7.3 fastmap_1.1.0 httpuv_1.6.8
[26] ps_1.7.2 fansi_1.0.4 Rcpp_1.0.10 xtable_1.8-4 renv_0.12.5
[31] promises_1.2.0.1 cachem_1.0.6 pkgload_1.3.2 mime_0.12 fs_1.6.0
[36] brio_1.1.3 digest_0.6.31 stringi_1.7.12 processx_3.8.0 dplyr_1.1.0
[41] shiny_1.7.4 cli_3.6.0 tools_4.2.2 magrittr_2.0.3 tibble_3.1.8
[46] profvis_0.3.7 crayon_1.5.2 tidyr_1.3.0 pkgconfig_2.0.3 ellipsis_0.3.2
[51] data.table_1.14.6 prettyunits_1.1.1 assertthat_0.2.1 rstudioapi_0.14 R6_2.5.1
[56] compiler_4.2.2

[pkyle]

@robbieorvis please delete the argument , ignore_columns = "item" from the line of code in question, which I think is line 138 but might be slightly different depending on your version. It's not an argument of the left_join() function, so while some environments or dplyr versions might allow extraneous arguments without crashing, there's no reason to have it in there. It'll be gone by the time of the next release!

[robbieorvis]

You're talking about the module where the crash is happening using the non renv approach right? That would be this module:
module_aglu_LB141.ag_Fert_IFA_ctry_crop

And then I should run it the normal way without renv and just using the normal driver() command?

Just confirming that we are abandoning trying to get renv working and instead trying to debug the R module.

[pkyle]

I don't mean to recommend one way or another whether you should use renv; I've never used it myself. But that one specific line of code within module_aglu_LB141.ag_Fert_IFA_ctry_crop contains an argument that is not recognized by the left_join() function, and dplyr/tidyverse functions often get more stringent over time in terms of what's allowed. If you fix this and then crash later on in the code for a similar reason, and renv isn't working out, then you can use devtools::install_version() to get an older version of dplyr and that should work.

[robbieorvis]

Sorry @pkyle didn't realize that was your response and not one from @russellhz.

That fixed the issue with the previous module, but now that I've made it further, another one popped up:

[1] "module_energy_L210.resources"
Error in mutate() at gcam_data_system/R/utils-data.R:32:2:
ℹ In argument: environCost = if_else(resource == "coal" & region %in% L210.low_reg, 0, environCost).
Caused by error in if_else():
! false must be a vector, not NULL.
Run rlang::last_error() to see where the error occurred.

rlang::last_error()
<error/dplyr:::mutate_error>
Error in mutate() at gcam_data_system/R/utils-data.R:32:2:
ℹ In argument: environCost = if_else(resource == "coal" & region %in% L210.low_reg, 0, environCost).
Caused by error in if_else():
! false must be a vector, not NULL.

---

Backtrace:

1. gcamdata::driver()
2. dplyr::if_else(resource == "coal" & region %in% L210.low_reg, 0, environCost)

sessionInfo()
R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19044)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8 LC_CTYPE=English_United States.utf8
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C
[5] LC_TIME=English_United States.utf8

attached base packages:
[1] stats graphics grDevices datasets utils methods base

other attached packages:
[1] gcamdata_5.1 testthat_3.1.6

loaded via a namespace (and not attached):
[1] tidyselect_1.2.0 remotes_2.4.2 purrr_1.0.1 vctrs_0.5.2 generics_0.1.3
[6] miniUI_0.1.1.1 htmltools_0.5.4 usethis_2.1.6 utf8_1.2.3 rlang_1.0.6
[11] pkgbuild_1.4.0 later_1.3.0 pillar_1.8.1 urlchecker_1.0.1 glue_1.6.2
[16] withr_2.5.0 bit64_4.0.5 sessioninfo_1.2.2 lifecycle_1.0.3 stringr_1.5.0
[21] devtools_2.4.5 htmlwidgets_1.6.1 memoise_2.0.1 tzdb_0.3.0 callr_3.7.3
[26] fastmap_1.1.0 httpuv_1.6.8 ps_1.7.2 parallel_4.2.2 fansi_1.0.4
[31] Rcpp_1.0.10 readr_2.1.4 xtable_1.8-4 renv_0.12.5 promises_1.2.0.1
[36] cachem_1.0.6 desc_1.4.2 pkgload_1.3.2 vroom_1.6.1 bit_4.0.5
[41] mime_0.12 fs_1.6.0 brio_1.1.3 hms_1.1.2 digest_0.6.31
[46] stringi_1.7.12 processx_3.8.0 dplyr_1.1.0 shiny_1.7.4 rprojroot_2.0.3
[51] cli_3.6.0 tools_4.2.2 magrittr_2.0.3 tibble_3.1.8 profvis_0.3.7
[56] crayon_1.5.2 tidyr_1.3.0 pkgconfig_2.0.3 ellipsis_0.3.2 data.table_1.14.6
[61] prettyunits_1.1.1 assertthat_0.2.1 rstudioapi_0.14 R6_2.5.1 compiler_4.2.2

[pkyle]

Haha yep, I was worried that was going to happen! When they tighten the rules of functions in tidyverse package updates, we typically have ~5 places that need minor updates to continue to execute. In this case in module_energy_L210.resources this is a bug that was permitted in the older versions of if_else(); to fix it, in that highlighted line of code and the one below, the string environCost should be input.cost. i.e.,
mutate(environCost = if_else(resource == "coal" & region %in% L210.low_reg, 0, input.cost), environCost = if_else(resource == "coal" & region %in% L210.high_reg, 10 * input.cost, input.cost)) %>%
Anyway, unless you'd like to continue walking us through our required modifications for the next dplyr update (I'm not gonna update mine for a while now 😆 ), you probably just want to roll back your local dplyr package to something less stringent. I'm currently using 1.0.10. So, the command to install that one locally would be:
devtools::install_version("dplyr", "1.0.10")

[robbieorvis]

Okay, progress! @pkyle rolling back dplyr gets the driver() command working when not using renv. Still can't use driver__drake().

For @russellhz - there is some issue with curl when running renv::restore(). Curl fails to compile. I can download the binary and install directly in to the renv folder in the gcam-data-system, but when running renv::restore() Rstudio attempts to override and it crashes (I couldn't determine if there is a way to skip trying to restore curl.... as you can probably tell I am an extreme novice with R and RStudio).

Seems like maybe a broader issue than your software with curl generally. All five of us trying to install renv have run into the same issue with curl

[russellhz]

@robbieorvis thanks for catching this - it appears that this is an issue when combining windows with the latest version of R (4.2+) and certain packages (curl, openssl, igraph).

If you would still like to get renv working (which hopefully will also get driver_drake() working), I have created a new renv.lock file for this case. You can simply replace the renv.lock file at the top of your gcamdata directory with this file: https://github.com/JGCRI/gcamdata/blob/renv_lock_R4-2/renv.lock

Then you can try renv::restore() again