Poor performance of `pivot_wider`

[ashiklom]

The new pivot_wider is roughly twice as slow as the old spread (and reshape2::dcast) were.

library(tidyr)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

mydf <- expand_grid(
  case = sprintf("%03d", seq(1, 4000)),
  year = seq(1900, 2000),
  name = c("x", "y", "z")
) %>%
  mutate(value = rnorm(nrow(.)))

bench::mark(
  pivot = pivot_wider(mydf, names_from = "name", values_from = "value"),
  spread = spread(mydf, name, value),
  dcast = reshape2::dcast(mydf, case + year ~ name)
)
#> Warning: Some expressions had a GC in every iteration; so filtering is
#> disabled.
#> # A tibble: 3 x 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 pivot         1.02s    1.02s     0.984     125MB     1.97
#> 2 spread     457.02ms 461.95ms     2.16      430MB    15.2 
#> 3 dcast      449.15ms 472.34ms     2.12      457MB    11.6

^{Created on 2019-10-10 by the reprex package (v0.3.0)}

This performance penalty is prohibitively severe for some even larger datasets I'm working with.

Based on some basic profvis, most of the time is spent in the actual pivoting operation (pivot_wider_spec), and in there, in the vctrs::vec_match function.

[mschilli87]

I just ran into this were a crucial step of an analysis (one that is repeated in each iteration of an optimization) took over 7 seconds. I replaced a single pivot_wider(names_from = parameter, values_from = value) by spread(parameter, value) and the same process took just above 0.1 seconds (over 50(!) times faster).

---

bench::mark output:

 A tibble: 2 x 13
  expression      min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>
1 pivot         6.72s    6.72s     0.149    48.3MB     3.12     1    21
2 spread     123.36ms 127.31ms     7.90     74.4MB     0        4     0
# … with 5 more variables: total_time <bch:tm>, result <list>,
#   memory <list>, time <list>, gc <list>
Warning message:
Some expressions had a GC in every iteration; so filtering is disabled.

[hadley]

This appears to be resolved in the dev version (thanks to performance improvements in vctrs):

library(tidyr)
library(dplyr, warn.conflicts = FALSE)

mydf <- expand_grid(
  case = sprintf("%03d", seq(1, 4000)),
  year = seq(1900, 2000),
  name = c("x", "y", "z")
) %>%
  mutate(value = rnorm(nrow(.)))

bench::mark(
  pivot = pivot_wider(mydf, names_from = "name", values_from = "value"),
  spread = spread(mydf, name, value),
  dcast = reshape2::dcast(mydf, case + year ~ name)
)
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
#> # A tibble: 3 x 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 pivot         671ms    671ms      1.49     115MB     2.98
#> 2 spread        699ms    699ms      1.43     431MB    10.0 
#> 3 dcast         727ms    727ms      1.38     457MB     9.63

^{Created on 2019-11-13 by the reprex package (v0.3.0)}