Skip to content

[ENG-8052] Fixed FilterMixin issue with multiple values of any filter #11150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

ihorsokhanexoft
Copy link
Contributor

@ihorsokhanexoft ihorsokhanexoft commented May 19, 2025

Purpose

Query params filters were limited to handle multiple values starting from this commit 6fd1968 for these fields only: _id, guid._id, journal_id, moderation_state. Over time this list have been updated to these four values. Now all filter classes that inherit from FilterMixin support this feature.
In order to filter by multiple values, FE should use the following syntax:
?filter[title]=text1,text2
Filter uses OR operator for its values while filters use AND operator between themselves

Changes

  1. Updated if statement to support multiple values for all fields
  2. Also when one value was passed for any of the fields above, it was processed as a list. Now operator update was removed for id field in ListFilterMixin.postprocess_query_param so that in lookup is not used for one id that is actually string.
  3. Added tests

Concerns

  1. Would be nice to test different endpoints that use different filters because the base filter mixin was updated, thus it may impact all filters.
  2. Another concern is that now we use __in lookup for fields in these filters instead of __icontains(which was wrong as title1,title2 was passed as one value). It doesn't impact any filtering that FE does using internal values they get from BE, so for the case in the ticket it'll work. But do we have filtering on the platform where user can enter text manually that will be passed as a query parameter together with some filter? If user enters "Dev" and FE filters like ?filter['title']=Dev and it should find all records that contain this word, I assume it won't work. It'll actually require refactoring filters.

Ticket

https://openscience.atlassian.net/jira/software/c/projects/ENG/boards/145?selectedIssue=ENG-8052

Copy link
Collaborator

@brianjgeiger brianjgeiger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly just concerned about database performance.

@@ -292,7 +292,7 @@ def parse_query_params(self, query_params):
query.get(key).update({
field_name: self._parse_date_param(field, source_field_name, op, value),
})
elif not isinstance(value, int) and source_field_name in ['_id', 'guid._id', 'journal_id', 'moderation_state']:
elif not isinstance(value, int) and ',' in value:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I mentioned on Slack, let's just add the field we want to explicitly filter on here rather than opening it up to all fields. Too open may cause people to be able to easily construct complex queries that would cause us database problems without having a clear use case for us.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, okay

@@ -505,7 +505,6 @@ def postprocess_query_param(self, key, field_name, operation):
if issubclass(self.model_class, GuidMixin)
else self.model_class.primary_identifier_name
)
operation['op'] = 'in'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not useful?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see, you mentioned that in your Changes section.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it was broking one value filtering

fields = self.view.parse_query_params(query_params)
parsed_field = fields['filter[bool_field]']['bool_field']
assert parsed_field['source_field_name'] == 'foobar'
assert parsed_field['value'] is False
assert parsed_field['op'] == 'eq'

def test_multiple_values_filter(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably you won't need these specific tests with the change I requested, but you might add one for the notification subscriptions.

Copy link
Collaborator

@brianjgeiger brianjgeiger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good when CI passes.

@brianjgeiger brianjgeiger merged commit 883b530 into CenterForOpenScience:feature/pbs-25-10 May 19, 2025
6 checks passed
Johnetordoff added a commit to Johnetordoff/osf.io that referenced this pull request Jun 3, 2025
…cience/osf.io into refactor-notifications

* 'feature/pbs-25-10' of https://github.com/CenterForOpenScience/osf.io:
  fix issue where trying another already confirmed email threw an uncaught exception (CenterForOpenScience#11161)
  [ENG-8148] Add ArtifactOutcome in annotations to linked nodes  (CenterForOpenScience#11158)
  [ENG-7966] Add "collected-in" relationship for Nodes (CenterForOpenScience#11140)
  fix issue where not having any external identities caused a 500
  [ENG-7965] Add v2 email token confirmation endpoints (CenterForOpenScience#11139)
  [ENG-8052] Fixed FilterMixin issue with multiple values of notification subscription field (CenterForOpenScience#11150)
  support related_counts for view_only links (CenterForOpenScience#11148)
  allow admins change registration providers (CenterForOpenScience#11145)
  [ENG-7927] Improved logging for embargo termination (CenterForOpenScience#11137)
  [ENG-7873] CLONE - SPAM - When Hamming a Spammed user, preprints and registrations remain private (CenterForOpenScience#11125)
  Update changelog and package.json
  fix TypeError when check stucked registration
  revert async email sending (CenterForOpenScience#11134)
  [ENG-7921] Add scopes for applications to full_read and full_write scopes (CenterForOpenScience#11126)

# Conflicts:
#	api_tests/nodes/views/test_node_detail.py
#	api_tests/nodes/views/test_node_linked_registrations.py
#	framework/auth/oauth_scopes.py
#	tests/test_registrations/test_retractions.py
ihorsokhanexoft added a commit to ihorsokhanexoft/osf.io that referenced this pull request Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants