Unnest functions fail with lists of exprs

[md0u80c9]

Hi,

I have some code which uses a fair bit of meta programming (so a table of audit measures basically - where we have lists of the audit measure function being applied). I spotted a neat-looking optimisation to use tidyr to unnest the list.

Unfortunately, when using unnest where the list elements are functions, we get errors within the function (and presumably unpredictable behaviour). Ironically it's presumably because internally tidyr is doing exactly what my code is trying to do with the results of the function!

To demonstrate this, I used the tidyr example and converted the 'films' list-column to have functions rather than film titles (imagine we're going to pluck those functions and insert them into a later dplyr query on a table).

---

Brief description of the problem

df <- tibble(
   character = c("Toothless", "Dory"),
   metadata = list(
     list(
       species = "dragon",
       color = "black",
       films = c(
         rlang::expr(.data[["title"]] == "How to Train Your Dragon"),
         rlang::expr(.data[["title"]] == "How to Train Your Dragon 2"),
         rlang::expr(.data[["title"]] == "How to Train Your Dragon: The Hidden World")
        )
     ),
     list(
       species = "clownfish",
       color = "blue",
       films = c(rlang::expr(.data[["title"]] == "Finding Nemo"),
         rlang::expr(.data[["title"]] == "Finding Dory"))
     )
   )
 )

tidyr::unnest_wider(df, metadata)

tidyr::unnest_longer(df, films)

The last command to unnest films results in:

Error in eval_tidy(enquo(var), var_env) : object 'films' not found

[hadley]

Can you please provide a minimal reproducible example using the reprex package? The goal of a reprex is to make it as easy as possible for me to recreate your problem so that I can fix it. If you've never made a minimal reprex before, there is lots of good advice here.

[md0u80c9]

df <- tibble::tibble(
  character = c("Toothless", "Dory"),
  metadata = list(
    list(
      species = "dragon",
      color = "black",
      films = c(
        rlang::expr(.data[["title"]] == "How to Train Your Dragon"),
        rlang::expr(.data[["title"]] == "How to Train Your Dragon 2"),
        rlang::expr(.data[["title"]] == "How to Train Your Dragon: The Hidden World")
      )
    ),
    list(
      species = "clownfish",
      color = "blue",
      films = c(rlang::expr(.data[["title"]] == "Finding Nemo"),
                rlang::expr(.data[["title"]] == "Finding Dory"))
    )
  )
)

tidyr::unnest_wider(df, metadata)
#> # A tibble: 2 x 4
#>   character species   color films     
#>   <chr>     <chr>     <chr> <list>    
#> 1 Toothless dragon    black <list [3]>
#> 2 Dory      clownfish blue  <list [2]>

tidyr::unnest_longer(df, films)
#> Error in eval_tidy(enquo(var), var_env): object 'films' not found

^{Created on 2020-01-03 by the reprex package (v0.3.0)}

[hadley]

I don't think that actually illustrates the problem as df doesn't contain a films column.

[md0u80c9]

Oops - tried again this time pointing at our updated df2.

df <- tibble::tibble(
  character = c("Toothless", "Dory"),
  metadata = list(
    list(
      species = "dragon",
      color = "black",
      films = list(
        rlang::expr(.data[["title"]] == "How to Train Your Dragon"),
        rlang::expr(.data[["title"]] == "How to Train Your Dragon 2"),
        rlang::expr(.data[["title"]] == "How to Train Your Dragon: The Hidden World")
      )
    ),
    list(
      species = "clownfish",
      color = "blue",
      films = list(rlang::expr(.data[["title"]] == "Finding Nemo"),
                rlang::expr(.data[["title"]] == "Finding Dory"))
    )
  )
)

df2 <- tidyr::unnest_wider(df, metadata)

tidyr::unnest_longer(df2, films)
#> Error: `x` must be a vector, not a call

^{Created on 2020-01-03 by the reprex package (v0.3.0)}

[md0u80c9]

Also with the previous reprex there - is there a way to make the missing column error friendlier? It implies that the error is in eval_tidy (which it presumably is within the package); but the user has called tidyr::unnest.

[hadley]

We are working on it; unfortunately the error comes from fairly deep down in the bowels of tidy evaluation, so it is taking some work to bubble it up to a more useful place.

[md0u80c9]

Did that latest reprex make sense? I used the example data frame from the tidyr vignette itself mainly out of lack of imagination (it's not a great example of why you'd want exprs in a nested table - but I'm sure you can imagine the expressions are not all identical and trivial as in this example!)

[hadley]

Yeah, it makes sense. Here's a simpler reprex:

library(tidyr)

df <- tibble(x = 1:2, y = list(list(quote(x)), list(quote(x), quote(y))))
df %>% unnest_longer(y)
#> Error: `x` must be a vector, not a symbol

^{Created on 2020-01-04 by the reprex package (v0.3.0)}

[md0u80c9]

The error seems to be occurring because the simplify_col function in tidyr tries to call vctrs::vec_size on each item in the nested lists - so we can distill the problem, to the following expression which fails:

vec_size(quote(x))

Is this in turn a bug in vctrs? I presume the size of a vector of expressions should be calculable: length(quote(x)) returns 1, which I think is the answer I would expect here.

[hadley]

The heart of the problem is that quote(x) is not a vector; i.e. it's not something that belongs in the column of a data frame.

[md0u80c9]

I've been looking at the simplify_col function in rectangle.r which is where the error occurs (from line 324).

Here's my reasoning of what the function is doing - please feel free to clarify it if I'm wrong.

We're taking x as our input parameter to simplify_col, which must be either a list or a data frame.

We exit early if x is a list of lists.

We are then trying to establish the size of all contents of x by mapping over each item in the list or data frame and calling vec_size. In our case this is a list, each containing a single quote().

But when we are calling map, our vector quote(x) is being simplified to an expression, rather than a vector of expressions of length 1.

The error is because it now sees quote(x) is not a vector, rather than as a vector of length 1.
So I think the failure is occurring here because c(quote(x)) is being wrongly simplified to quote(x).

Testing this:

vec_size(quote(x)) fails with an error.
length(quote(x)) returns 1.
vec_size(c(quote(x)) returns 1 (because it's a vector).
vec_size(c(quote(x), quote(y)) returns 2.

I think we can fix this by modifying line 344 to instead read:

n <- map_int(x, ~ vec_size(c(.x)))

However, we then hit a similar problem a few lines later:

vec_c(!!!x)

Where we're again expecting all of the items in the list to be vectors to be spliced; yet we've had them forced to be expressions again. Is there a way to solve this latter case?