complete() on grouping variables gives wrong output

[huftis]

When running complete() on a grouped tibble and some of the variables completed on are also grouping variables, the resulting tibble is incorrect.

Here’s a reprex. First, I’ll create a simple tibble with three factors (gr1, gr2, splitgroup) and one numeric variable (x). The factor splitgroup is identical to gr1, so grouping on either variable should results in identical output. However, it doesn’t (I’ll remove splitgroup from the output just so that it doesn’t effect the ordering of the columns). There’s not even the same number of rows in the output:

library(tidyverse)
#> ── Attaching packages ───────────────────────────────────────────── tidyverse 1.2.1 ──
#> ✔ ggplot2 2.2.1          ✔ purrr   0.2.4     
#> ✔ tibble  1.4.1          ✔ dplyr   0.7.4.9000
#> ✔ tidyr   0.7.2.9000     ✔ stringr 1.2.0     
#> ✔ readr   1.1.1          ✔ forcats 0.2.0
#> ── Conflicts ──────────────────────────────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag()    masks stats::lag()

# Example data
d = tibble(
  gr1 = factor(c("A", "B", "B")),
  gr2 = factor(c(1, 2, 2)),
  x = c(10, 20, 30),
  splitgroup = gr1
)

# Complete on grouping variable
d %>% 
  group_by(gr1) %>% 
  complete(gr1, gr2) %>% 
  select(-splitgroup) # 10 rows
#> # A tibble: 10 x 3
#> # Groups:   gr1 [2]
#>    gr1    gr2        x
#>    <fctr> <fctr> <dbl>
#>  1 A      1       10.0
#>  2 A      2       NA  
#>  3 B      1       NA  
#>  4 B      2       20.0
#>  5 B      2       30.0
#>  6 A      1       10.0
#>  7 A      2       NA  
#>  8 B      1       NA  
#>  9 B      2       20.0
#> 10 B      2       30.0

# Completing on non-grouping but identical variable (should give same output)
d %>% 
  group_by(splitgroup) %>% 
  complete(gr1, gr2) %>% 
  ungroup %>% select(-splitgroup) # 9 rows
#> # A tibble: 9 x 3
#>   gr1    gr2        x
#>   <fctr> <fctr> <dbl>
#> 1 A      1       10.0
#> 2 A      2       NA  
#> 3 B      1       NA  
#> 4 B      2       NA  
#> 5 A      1       NA  
#> 6 A      2       NA  
#> 7 B      1       NA  
#> 8 B      2       20.0
#> 9 B      2       30.0

# Alternative method to find the *expected* results
# (which are identical to the results from the
# `group_by(splitgroup)` approach)
d %>% 
  split(.$gr1) %>% 
  map_df(~complete(., gr1, gr2)) %>% 
  select(-splitgroup) # 9 rows
#> # A tibble: 9 x 3
#>   gr1    gr2        x
#>   <fctr> <fctr> <dbl>
#> 1 A      1       10.0
#> 2 A      2       NA  
#> 3 B      1       NA  
#> 4 B      2       NA  
#> 5 A      1       NA  
#> 6 A      2       NA  
#> 7 B      1       NA  
#> 8 B      2       20.0
#> 9 B      2       30.0

[jnolis]

I was just coming here to post the same issue. It seems that complete ignores the grouping variables entirely. I can imagine some valid reasons for that, but if so then tidyr should output an error or at least a warning. This has caused me issues multiple times.

[kendonB]

There's a natural case here to output a warning/throw an error when trying to complete a grouped tibble that is being completed based on the same groups - the operation would do nothing if complete operates on the group_df like other functions operate on group_dfs.

tbh, I can not see a use case for using complete on a grouped df. I would advocate just ignoring the fact that it's a group_df altogether and have the behavior be identical regardless of initial grouping.

[huftis]

@kendonB Completing based on just the grouping variable(s) (e.g. completing A based on a tibble grouped by A) might be a strange thing do, but completing based on the grouping variables and other variables (e.g. completing A and B based on a tibble grouped by A) may very well be both sensible and useful.

[huftis]

@kendonB, @hadley Hmm, I’m not sure I agree with myself anymore. I was sure I had an example where completing based on a grouping variable made sense. But I can’t find it anymore. And thinking about it, I no longer am convinced that it makes sense. So having an error message seems fine to me (I think an error is better than a warning, as the current behaviour of completing on a grouped variable is buggy).

[hadley]

complete() doesn't have a grouped_df method, so this is the result of expand.grouped_df():

#' @export
expand.grouped_df <- function(data, ...) {
  dots <- quos(...)
  dplyr::do(data, expand(., !!! dots))
}

which TBH I'm not sure about either - the use of do() is a warning sign. I think it should probably translate more like

df %>% group_by(a) %>% expand(b)
# same as
df %>% expand(nesting(a), b)

df %>% group_by(a, b) %>% expand(c)
# ->
df %>% expand(nesting(a, b), c)

df %>% group_by(a, b) %>% expand(b, c)
# ->
df %>% expand(nesting(a), b, c)

I'm not sure that if that would fix the issues with complete() too.

[hadley]

@huftis it's not clear to me that those commands should give the same results

To me:

d %>% 
  group_by(gr1) %>% 
  complete(gr1, gr2)
# Should be equivalent to 
d %>% 
  complete(gr1, gr2) %>%
  group_by(gr1)

d %>% 
  group_by(splitgroup) %>% 
  complete(gr1, gr2)
# Should be equivalent to 
d %>% 
  complete(splitgroup, gr1, gr2)
  group_by(splitgroup)

(That's not what happens currently)

[gvwilson]

I tripped over this one earlier today - took me a while to figure out that complete doesn't respect group_by:

library(tidyverse)

raw <- tribble(
  ~left, ~right, ~value,
  "a",   "x",    1,
  "a",   "x",    2,
  "a",   "y",    1,
  "a",   "y",    2,
  "b",   "x",    1,
  "b",   "x",    2
)

cat("The raw data table\n")
raw

cat("\nCompleting the raw data table (fills in 'b'-'y')\n")
raw %>%
  complete(left, right, fill = list(value = 0))

cat("\nSumming values for distinct combinations of 'left' and 'right'\n")
raw %>%
  group_by(left, right) %>%
  summarize(total = sum(value))

cat("\nTrying to complete summation does _not_ fill in 'b'-'y'\n")
raw %>%
  group_by(left, right) %>%
  summarize(total = sum(value)) %>%
  complete(left, right, fill = list(total = 0))

1. What's the recommended workaround?
2. Should I PR an addition to the docs for complete to warn people about this case?

[hadley]

1. Add the grouping variables to complete() and then regroup? I think this got stuck last time because no one reassured me that my thoughts about the desired behaviour were correct.

2. I'd prefer to just fix it, and the timeline for getting it into CRAN is likely to be the same either way.

[kendonB]

I still think that completing based on a grouping variable is an odd thing to do and should either throw an error or a warning. Responding to @hadley's year-old example:

He says:

d %>% 
  group_by(gr1) %>% 
  complete(gr1, gr2)
# Should be equivalent to 
d %>% 
  complete(gr1, gr2) %>%
  group_by(gr1)

I think that complete should operate on a grouped df just like other functions do. Other functions can see the grouping variables but they always only have a single value within the group. The grouping variables are always then already "complete" within the subdataframes and there would be no reason to include them, which might warrant a warning. The above then should not be equivalent.

d %>% 
  group_by(splitgroup) %>% 
  complete(gr1, gr2)
# Should be equivalent to 
d %>% 
  complete(splitgroup, gr1, gr2)
  group_by(splitgroup)

Again, I would not expect these to be equivalent. In the former, the user is asking tidyr to complete the subdataframes defined by the grouping variable splitgroup. The set of values for gr1 and gr2 need not be same across different values of splitgroup so every value from the big dataframe wouldn't necessarily appear in every group. The latter example asks tidyr to complete the entire dataframe by all three variables, ensuring that all combinations of all three variables appear, then group by splitgroup.

[hadley]

@kendonB I agree with your analysis, although I think that we can just silently ignore grouping variables that are respecified in complete(). (I think detecting them is potentially hard because both complete() and group_by() have mutate() semantics).

[hadley]

Or should

d %>% 
  group_by(splitgroup) %>% 
  complete(gr1, gr2)

# Be equivalent to

d %>% 
  complete(nesting(splitgroup, crossing(gr1, gr2)) %>%