UI elements do not refresh when filter removed

[npaszty]

What happened?

"Select Biomarker" menu does not refresh do display all biomarkers available in data after filter panel filter on PARAMCD is removed.

Manifested in study data we are using to test upversioning and can replicate in teal.gallery app as well but there is different behavior as far as the modules producing a plot. none of the teal gallery app plots are produced if ALT is filtered out as a first step. however see below for what happens in the study data app we are using for testing.

Steps

1. start app
2. go to view data and filter PARAMCD to ALT and F2974907 for example
3. got to boxplot plot where default PARAM is ALBUM. box plot throws validation error. this is actually the sample app behavior. the study app produces a boxplot of ALT
4. got to correlation plot. produces a plot with ALT as x-axis biomarker and ALT as y-axis biomarker
5. got to density dist plot. produces plot with ALT.
6. got to line plot. produces plot with ALT.
7. got to spaghetti plot. prodices plot with ALT.
8. remove the PARAM filter. biomarker UI item does not reset to all biomarkers. just includes the two that were originally filtered

sessionInfo()

R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] tidyr_1.3.1                     teal.goshawk_0.1.15.9027        teal.modules.general_0.3.0.9038 ggmosaic_0.3.3                 
 [5] ggplot2_3.5.0                   teal.modules.clinical_0.9.0     tern_0.9.5.9007                 rtables_0.6.9.9002             
 [9] formatters_0.5.8.9000           teal.transform_0.5.0            teal.connectors.rice_0.2.0      teal.connectors.harbour_0.1.1  
[13] teal_0.15.2                     teal.slice_0.5.0                teal.data_0.5.0                 teal.code_0.5.0                
[17] stringr_1.5.1                   sparkline_2.0                   ricepass_1.1.4                  rice_3.0.1                     
[21] magrittr_2.0.3                  goshawk_0.1.17.9003             dplyr_1.1.4                     googlesheets4_1.0.1            
[25] DescTools_0.99.54               shiny_1.8.1.1                  

loaded via a namespace (and not attached):
  [1] rstudioapi_0.16.0      jsonlite_1.8.8         estimability_1.5       farver_2.1.1           rmarkdown_2.26         ragg_1.2.7            
  [7] geepack_1.3.10         fs_1.6.3               vctrs_0.6.5            memoise_2.0.1          askpass_1.2.0          htmltools_0.5.8.1     
 [13] forcats_1.0.0          curl_5.2.1             haven_2.5.4            broom_1.0.5            cellranger_1.1.0       sass_0.4.9            
 [19] bslib_0.7.0            htmlwidgets_1.6.4      fontawesome_0.5.2      emmeans_1.10.0         rootSolve_1.8.2.4      plotly_4.10.4         
 [25] cachem_1.0.8           mime_0.12              lifecycle_1.0.4        teal.widgets_0.4.2     pkgconfig_2.0.3        Matrix_1.6-1.1        
 [31] R6_2.5.1               fastmap_1.1.1          rbibutils_2.2.16       digest_0.6.35          Exact_3.2              colorspace_2.1-0      
 [37] shinycssloaders_1.0.0  textshaping_0.3.7      crosstalk_1.2.1        labeling_0.4.3         fansi_1.0.6            httr_1.4.7            
 [43] compiler_4.3.1         gargle_1.2.1           proxy_0.4-27           bit64_4.0.5            withr_3.0.0            backports_1.4.1       
 [49] logger_0.3.0           MASS_7.3-60.0.1        openssl_2.1.1          gld_2.6.6              tools_4.3.1            robslopes_1.1.3       
 [55] googledrive_2.0.0      httpuv_1.6.15          shinyvalidate_0.1.3    glue_1.7.0             nlme_3.1-163           promises_1.2.1        
 [61] grid_4.3.1             rsconnect_1.2.2        checkmate_2.3.1        generics_0.1.3         gtable_0.3.4           class_7.3-22          
 [67] data.table_1.15.4      lmom_3.0               hms_1.1.3              utf8_1.2.4             ggrepel_0.9.5          pillar_1.9.0          
 [73] later_1.3.2            splines_4.3.1          tern.gee_0.1.3         lattice_0.21-9         renv_1.0.3             survival_3.5-7        
 [79] bit_4.0.5              tidyselect_1.2.1       knitr_1.45             teal.logger_0.2.0.9005 xfun_0.43              expm_0.999-9          
 [85] DT_0.33                stringi_1.8.3          yaml_2.3.8             lazyeval_0.2.2         boot_1.3-29            shinyWidgets_0.8.4    
 [91] evaluate_0.23          tibble_3.2.1           cli_3.6.2              systemfonts_1.0.6      arrow_16.1.0           xtable_1.8-4          
 [97] Rdpack_2.6             munsell_0.5.1          jquerylib_0.1.4        Rcpp_1.0.12            teal.reporter_0.3.1    readxl_1.4.3          
[103] parallel_4.3.1         assertthat_0.2.1       mcr_1.3.3              viridisLite_0.4.2      mvtnorm_1.2-4          scales_1.3.0          
[109] e1071_1.7-14           crayon_1.5.2           purrr_1.0.2            rlang_1.1.3            cowplot_1.1.3          shinyjs_2.1.0

Relevant log output

INFO [2024-07-24 21:44:22] Shiny input change detected on submit: 0 -> 1
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Removed 6 rows containing missing values or values outside the scale range (`geom_line()`).
Warning: Unknown or uninitialised column: `AVALU`.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Removed 6 rows containing missing values or values outside the scale range (`geom_line()`).
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Removed 3 rows containing missing values or values outside the scale range (`geom_line()`).
`geom_line()`: Each group consists of only one observation.
ℹ Do you need to adjust the group aesthetic?
`geom_line()`: Each group consists of only one observation.
ℹ Do you need to adjust the group aesthetic?
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Groups with fewer than two data points have been dropped.
Warning: Removed 6 rows containing missing values or values outside the scale range (`geom_line()`).

Code of Conduct

• I agree to follow this project's Code of Conduct.

Contribution Guidelines

• I agree to follow this project's Contribution Guidelines.

Security Policy

• I agree to follow this project's Security Policy.

[m7pr]

I think this is a bigger issue, not necesserilly just in modules from teal.goshawk but with teal in general.
I think encoding panel gets created the moment when module is opened, and uses what was currently displayed in the filter panel, for the time being of the module till the end of the app.

Encodings do not get refreshed when filters are edited in the filter panel. I will open an issue in teal or teal.slice when I will create a reproducible example for developers to help.

[m7pr]

This happens for all plots for their all UI elements. I will change the name of the issue.
For boxplot I think this part is responsible - it does not get updated when filter panel adds or remove filters.

https://github.com/insightsengineering/teal.goshawk/blob/main/R/tm_g_gh_boxplot.R#L330-L345

teal.goshawk/R/tm_g_gh_boxplot.R

Lines 330 to 345 in 28735c8

	output$axis_selections <- renderUI({
	env <- shiny::isolate(as.list(data()@env))
	resolved_x <- teal.transform::resolve_delayed(module_args$xaxis_var, env)
	resolved_y <- teal.transform::resolve_delayed(module_args$yaxis_var, env)
	resolved_param <- teal.transform::resolve_delayed(module_args$param, env)
	templ_ui_params_vars(
	session$ns,
	xparam_choices = resolved_param$choices,
	xparam_selected = resolved_param$selected,
	xparam_label = module_args$"Select a Biomarker",
	xchoices = resolved_x$choices,
	xselected = resolved_x$selected,
	ychoices = resolved_y$choices,
	yselected = resolved_y$selected
	)
	})

Other plots have similar code, so once we figure this out for a boxplot, we can figure this out for the rest of the plots in here.

[m7pr]

I was able to create a minimal reproducible example with an open data. I believe the main reason of this behavior is the value_choices function in param = choices_selected(choices = value_choices("ADLB", "PARAMCD", "PARAM")).

To reproduce the issue:

1. Start the app.
2. In View Data tab, create ADLB filter for PARAMCD variable. Pick 2 levels.
3. Go to Box Plot tab, check choices under Select an X-Axis Biomarker - there are only 2.
4. Remove filter you created for ADLB+PARAMCD - there are still 2 choices in Select an X-Axis Biomarker.
5. Close the app.
6. Start the app.
7. Go to Box Plot tab without any filters created, check choices in Select an X-Axis Biomarker - there are 3.
8. If you create a filter for ADLB+PARAMCD now, there will always be 3 choices, no matter how many levels were left in data.

Simple App

library(teal)
library(teal.modules.general)
library(teal.goshawk)

data <- teal_data()
data <- within(data, {
  library(dplyr)
  library(nestcolor)
  library(stringr)
  
  # use non-exported function from goshawk
  h_identify_loq_values <- getFromNamespace("h_identify_loq_values", "goshawk")
  
  # original ARM value = dose value
  arm_mapping <- list(
    "A: Drug X" = "150mg QD",
    "B: Placebo" = "Placebo",
    "C: Combination" = "Combination"
  )
  set.seed(1)
  ADSL <- rADSL
  ADLB <- rADLB
  var_labels <- lapply(ADLB, function(x) attributes(x)$label)
  ADLB <- ADLB %>%
    mutate(
      AVISITCD = case_when(
        AVISIT == "SCREENING" ~ "SCR",
        AVISIT == "BASELINE" ~ "BL",
        grepl("WEEK", AVISIT) ~ paste("W", str_extract(AVISIT, "(?<=(WEEK ))[0-9]+")),
        TRUE ~ as.character(NA)
      ),
      AVISITCDN = case_when(
        AVISITCD == "SCR" ~ -2,
        AVISITCD == "BL" ~ 0,
        grepl("W", AVISITCD) ~ as.numeric(gsub("[^0-9]*", "", AVISITCD)),
        TRUE ~ as.numeric(NA)
      ),
      AVISITCD = factor(AVISITCD) %>% reorder(AVISITCDN),
      TRTORD = case_when(
        ARMCD == "ARM C" ~ 1,
        ARMCD == "ARM B" ~ 2,
        ARMCD == "ARM A" ~ 3
      ),
      ARM = as.character(arm_mapping[match(ARM, names(arm_mapping))]),
      ARM = factor(ARM) %>% reorder(TRTORD),
      ACTARM = as.character(arm_mapping[match(ACTARM, names(arm_mapping))]),
      ACTARM = factor(ACTARM) %>% reorder(TRTORD),
      ANRLO = 50,
      ANRHI = 75
    ) %>%
    rowwise() %>%
    group_by(PARAMCD) %>%
    mutate(LBSTRESC = ifelse(
      USUBJID %in% sample(USUBJID, 1, replace = TRUE),
      paste("<", round(runif(1, min = 25, max = 30))), LBSTRESC
    )) %>%
    mutate(LBSTRESC = ifelse(
      USUBJID %in% sample(USUBJID, 1, replace = TRUE),
      paste(">", round(runif(1, min = 70, max = 75))), LBSTRESC
    )) %>%
    ungroup()
  
  attr(ADLB[["ARM"]], "label") <- var_labels[["ARM"]]
  attr(ADLB[["ACTARM"]], "label") <- var_labels[["ACTARM"]]
  attr(ADLB[["ANRLO"]], "label") <- "Analysis Normal Range Lower Limit"
  attr(ADLB[["ANRHI"]], "label") <- "Analysis Normal Range Upper Limit"
  
  # add LLOQ and ULOQ variables
  ALB_LOQS <- h_identify_loq_values(ADLB, "LOQFL")
  ADLB <- left_join(ADLB, ALB_LOQS, by = "PARAM")
})

datanames <- c("ADSL", "ADLB")
datanames(data) <- datanames

join_keys(data) <- default_cdisc_join_keys[datanames]

app <- init(
  data = data,
  modules = modules(
    tm_data_table(label = "View Data"),
    tm_g_gh_boxplot(
      label = "Box Plot",
      dataname = "ADLB",
      param_var = "PARAMCD",
      param = choices_selected(choices = value_choices("ADLB", "PARAMCD", "PARAM")),
      yaxis_var = choices_selected(c("AVAL", "BASE", "CHG"), "AVAL"),
      xaxis_var = choices_selected(c("ACTARM", "ARM", "AVISITCD", "STUDYID"), "ARM"),
      facet_var = choices_selected(c("ACTARM", "ARM", "AVISITCD", "SEX"), "AVISITCD"),
      trt_group = choices_selected(c("ARM", "ACTARM"), "ARM"),
      loq_legend = TRUE,
      rotate_xlab = FALSE,
      hline_arb = c(60, 55),
      hline_arb_color = c("grey", "red"),
      hline_arb_label = c("default_hori_A", "default_hori_B"),
      hline_vars = c("ANRHI", "ANRLO", "ULOQN", "LLOQN"),
      hline_vars_colors = c("pink", "brown", "purple", "black"),
    )
  )
)
if (interactive()) {
  shinyApp(app$ui, app$server)
}

[m7pr]

@npaszty @gogonzo we are currently running a huge overhaul in teal and filter panael and the way we manage the data. I would recommend revisiting this issue once we are done with this refactor in teal insightsengineering/teal#1253

[npaszty]

@m7pr @gogonzo

thanks for looking into this. sounds like a big deal. this issue can be replicated in the teal.gallery app so that's a working example.

what I noticed in the teal.gallery app is that all the modules fail to produce a plot because the "Select a Biomarker" UI item doesn't default to the first PARAMCD that is filtered to which is CRP. so a validation error is thrown

however in the study app the module behavior is different. for example, the first PARAMCD in the list is ALBUM so alphabetically before ALT. after filtering in the manner described "What happened" section above and navigating to boxplot, it produces a plot for ALT which is the first PARAMCD selected in the filtered PARAMCD. I updated the "What happened" section of this issue.

[npaszty]

@m7pr

removed priority since you think it's a bigger issue with teal itself and may be handled in a current refactoring focus area.

[donyunardi]

Acceptance Criteria

• The X and Y axis Biomarker in encoding panel should still show all biomarkers, even though the biomarker are filtered out.
• Apply update to all modules.

Possible solution from discussion

• the module should receive unfiltered data when resolving the encoding panel choices.
• It should not be reactive from the filter panel.
• Update should be for all modules in teal.goshawk.

[gogonzo]

Note to consider when working on the issue:
I think some encoding elements were build back then on unfiltered data (_raw today).