Skip to content

Driver NotificationFilter config #533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
de8a76f
basic setup.
thelonelyvulpes Oct 24, 2022
6338f05
undo commithook change
thelonelyvulpes Oct 24, 2022
8cddfa2
full suite for driver config, undo validation for mapping
thelonelyvulpes Oct 24, 2022
aec9272
driver feature
thelonelyvulpes Oct 24, 2022
fd21236
add multi-filter config
thelonelyvulpes Oct 25, 2022
e463879
formatting for picky-parens
thelonelyvulpes Oct 25, 2022
2c13c5d
session configs
thelonelyvulpes Oct 25, 2022
b86aabc
move
thelonelyvulpes Oct 25, 2022
7b2afbe
dedupes
thelonelyvulpes Oct 25, 2022
d3e281e
rename to make it look nicer
thelonelyvulpes Oct 25, 2022
de188d5
mapping tests
thelonelyvulpes Oct 26, 2022
2d13713
add tx func test
thelonelyvulpes Oct 26, 2022
e157f61
Update nutkit/protocol/feature.py
thelonelyvulpes Oct 28, 2022
f7e8e77
always serialize notification filters
thelonelyvulpes Oct 28, 2022
db71062
Merge remote-tracking branch 'origin/feature/notifications' into feat…
thelonelyvulpes Oct 28, 2022
129eaf6
Feature:API:Driver:NotificationFilters
thelonelyvulpes Oct 28, 2022
52e8173
Stubscript: add bolt alias 5 == 5.0
robsdedude Oct 28, 2022
002028c
Update tests/stub/notification_filters/notification_filters_base.py
thelonelyvulpes Oct 28, 2022
0e251e7
Don't assert notifications order
robsdedude Oct 28, 2022
478a0d2
Replace some super calls
robsdedude Oct 28, 2022
a3c1e32
New `HELLO` message structure in Bolt 5.1
robsdedude Oct 31, 2022
32bbc47
Add tests for new `HELLO` structure in BOLT 5.1
robsdedude Oct 31, 2022
bb996b2
update notification tests
thelonelyvulpes Oct 31, 2022
0ff89ee
Merge remote-tracking branch 'origin/feature/notifications' into feat…
thelonelyvulpes Oct 31, 2022
50bbbce
rotate auth
thelonelyvulpes Oct 31, 2022
c7cd6e5
updated to specification
thelonelyvulpes Nov 4, 2022
2b020c7
remove order independent check from script as it is nonsensical.
thelonelyvulpes Nov 4, 2022
0755d0e
Don't add null notification filters because it can break some json de…
thelonelyvulpes Nov 4, 2022
efad9e3
HELLO: flip auth ane extra dict in version tests
robsdedude Nov 7, 2022
28ac8d6
Adjust notifications default values and combinations
robsdedude Nov 7, 2022
f9d822f
update test_full_notifications
thelonelyvulpes Nov 8, 2022
36077b9
support discard on notification tests
thelonelyvulpes Nov 10, 2022
9d12627
remove redundant pull
thelonelyvulpes Nov 10, 2022
1796354
remove specific pull size
thelonelyvulpes Nov 10, 2022
60a38ee
fix.
thelonelyvulpes Nov 10, 2022
7cfadbf
WIP todo: test renewable_auth
robsdedude Dec 14, 2022
d95e207
WIP: renewable auth token protocol messages
robsdedude Dec 19, 2022
4d3178d
WIP: Re-auth testing timeout
bigmontz Dec 19, 2022
79045be
Add tests for refreshing auth info on errors.
robsdedude Jan 11, 2023
e251d2a
Add docstrings for new protocol messages
robsdedude Jan 12, 2023
cdb32d1
Test LOGOFF/LOGON pipelining
robsdedude Jan 13, 2023
52a294e
Fix Bolt 5.1 auth scheme tests
robsdedude Jan 25, 2023
0ab9bc6
Fix auto lines that should be client lines
robsdedude Jan 26, 2023
a24e928
Merge branch '5.0' into re-auth
robsdedude Jan 31, 2023
93ba0dd
Merge branch '5.0' into feature/notifications
thelonelyvulpes Jan 31, 2023
2f25142
update testkit protocol objects for min severity & disabled categories
thelonelyvulpes Jan 31, 2023
e883262
Add tests for supports_session_auth and verify_authentication
robsdedude Feb 1, 2023
4544a6c
Merge branch '5.0' into re-auth
robsdedude Feb 1, 2023
1088d40
config
thelonelyvulpes Feb 3, 2023
5e08911
Merge branch 'feature/notifications' into notifications-two
thelonelyvulpes Feb 3, 2023
b20c66e
some python
thelonelyvulpes Feb 6, 2023
c1655cc
Fix driver creation
bigmontz Jan 11, 2023
74ca4d7
Add javascript to the exception list
bigmontz Feb 6, 2023
02e83e9
notifications!
thelonelyvulpes Feb 7, 2023
72e70cb
Merge branch 're-auth' into feature/notifications
thelonelyvulpes Feb 7, 2023
865bb32
add some more optimization checks
thelonelyvulpes Feb 7, 2023
2ca41c9
remove redundant test, as it is defined in the base configs
thelonelyvulpes Feb 7, 2023
b6b764d
WIP: test backwards compatibility
robsdedude Feb 3, 2023
711ef2a
Stub server: conditional blocks and python lines
robsdedude Feb 7, 2023
6626c23
Update verifyAuthentication tests
robsdedude Feb 7, 2023
6934a01
Document stubscript
robsdedude Feb 8, 2023
84db82e
Fix types.Feature.OPT_MINIMAL_RESETS assertion
robsdedude Feb 8, 2023
d75a7a9
Move routing from LOGON to HELLO in kerberos test
RichardIrons-neo4j Feb 9, 2023
76d8786
out of scope optimization
thelonelyvulpes Feb 10, 2023
c349e71
re-auth: adopt auth token provider API
robsdedude Mar 6, 2023
6e37294
Add missing feature guard
robsdedude Mar 7, 2023
ad56dba
Add error code for Java driver
robsdedude Mar 7, 2023
4f8fa2e
Allow optional routing to be `null`
robsdedude Mar 8, 2023
31f0c3f
Put user switching polyfill on hold
robsdedude Mar 8, 2023
23a027c
Put auth token manager behind feature flag
robsdedude Mar 8, 2023
0b0ee94
Improve Stubscript docs
robsdedude Mar 9, 2023
5d39f52
Add tests for retryable token expired error
robsdedude Mar 9, 2023
5f3b07f
Add basic Bolt 5.1 tests
robsdedude Mar 9, 2023
f7178b7
Merge branch 're-auth' into feature/notifications
thelonelyvulpes Mar 9, 2023
b9b21c0
add test empty array
thelonelyvulpes Mar 9, 2023
bafa3ef
Remove all re-auth tests; leave only basic bolt 5.2 tests
robsdedude Mar 10, 2023
b8f0ba2
Fix summary tests to include new notification properties
robsdedude Mar 14, 2023
71babad
Check disabled categories regardless of order
robsdedude Mar 15, 2023
83835aa
Merge branch '5.0' into feature/notifications
thelonelyvulpes Mar 15, 2023
da86f42
support pre-notifications change
thelonelyvulpes Mar 15, 2023
89ccf83
replace try with if has new notifications config
thelonelyvulpes Mar 16, 2023
9332c18
fix for python
thelonelyvulpes Mar 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions boltstub/bolt_protocol.py
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,7 @@ class Bolt4x4Protocol(Bolt4x3Protocol):
class Bolt5x0Protocol(Bolt4x4Protocol):

protocol_version = (5, 0)
version_aliases = set()
version_aliases = {(5,)}
# allow the server to negotiate other bolt versions
equivalent_versions = set()

Expand All @@ -479,8 +479,6 @@ class Bolt5x1Protocol(Bolt5x0Protocol):
# allow the server to negotiate other bolt versions
equivalent_versions = set()

server_agent = "Neo4j/5.4.0" # TODO: finalize!

messages = {
"C": {
**Bolt5x0Protocol.messages["C"],
Expand All @@ -489,3 +487,14 @@ class Bolt5x1Protocol(Bolt5x0Protocol):
},
"S": Bolt5x0Protocol.messages["S"],
}

server_agent = "Neo4j/5.1.0"


class Bolt5x2Protocol(Bolt5x1Protocol):
protocol_version = (5, 2)
version_aliases = set()
# allow the server to negotiate other bolt versions
equivalent_versions = set()

server_agent = "Neo4j/5.2.0"
32 changes: 20 additions & 12 deletions nutkit/frontend/driver.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ def __init__(self, backend, uri, auth_token, user_agent=None,
max_tx_retry_time_ms=None, encrypted=None,
trusted_certificates=None, liveness_check_timeout_ms=None,
max_connection_pool_size=None,
connection_acquisition_timeout_ms=None):
connection_acquisition_timeout_ms=None,
notifications_min_severity=None,
notifications_disabled_categories=None):
self._backend = backend
self._resolver_fn = resolver_fn
self._domain_name_resolver_fn = domain_name_resolver_fn
Expand All @@ -25,6 +27,8 @@ def __init__(self, backend, uri, auth_token, user_agent=None,
liveness_check_timeout_ms=liveness_check_timeout_ms,
max_connection_pool_size=max_connection_pool_size,
connection_acquisition_timeout_ms=connection_acquisition_timeout_ms, # noqa: E501
notifications_min_severity=notifications_min_severity,
notifications_disabled_categories=notifications_disabled_categories
)
res = backend.send_and_receive(req)
if not isinstance(res, protocol.Driver):
Expand Down Expand Up @@ -84,54 +88,58 @@ def execute_query(self, cypher, params=None,
req = protocol.ExecuteQuery(self._driver.id, cypher, params, config)
res = self.send_and_receive(req, allow_resolution=True)
if not isinstance(res, protocol.EagerResult):
raise Exception("Should be EagerResult but was: %s" % res)
raise Exception(f"Should be EagerResult but was: {res}")
return res

def verify_connectivity(self):
req = protocol.VerifyConnectivity(self._driver.id)
res = self.send_and_receive(req, allow_resolution=True)
if not isinstance(res, protocol.Driver):
raise Exception("Should be Driver but was: %s" % res)
raise Exception(f"Should be Driver but was: {res}")

def get_server_info(self):
req = protocol.GetServerInfo(self._driver.id)
res = self.send_and_receive(req, allow_resolution=True)
if not isinstance(res, protocol.ServerInfo):
raise Exception("Should be ServerInfo but was: %s" % res)
raise Exception(f"Should be ServerInfo but was: {res}")
return res

def supports_multi_db(self):
req = protocol.CheckMultiDBSupport(self._driver.id)
res = self.send_and_receive(req, allow_resolution=False)
if not isinstance(res, protocol.MultiDBSupport):
raise Exception("Should be MultiDBSupport")
raise Exception(f"Should be MultiDBSupport but was: {res}")
return res.available

def is_encrypted(self):
req = protocol.CheckDriverIsEncrypted(self._driver.id)
res = self.send_and_receive(req, allow_resolution=False)
if not isinstance(res, protocol.DriverIsEncrypted):
raise Exception("Should be DriverIsEncrypted")
raise Exception(f"Should be DriverIsEncrypted but was {res}")
return res.encrypted

def close(self):
req = protocol.DriverClose(self._driver.id)
res = self.send_and_receive(req, allow_resolution=False)
if not isinstance(res, protocol.Driver):
raise Exception("Should be driver")
raise Exception(f"Should be Driver but was {res}")

def session(self, access_mode, bookmarks=None, database=None,
fetch_size=None, impersonated_user=None,
bookmark_manager=None):
bookmark_manager=None,
notifications_min_severity=None,
notifications_disabled_categories=None):
req = protocol.NewSession(
self._driver.id, access_mode, bookmarks=bookmarks,
database=database, fetchSize=fetch_size,
impersonatedUser=impersonated_user,
bookmark_manager=bookmark_manager
bookmark_manager=bookmark_manager,
notifications_min_severity=notifications_min_severity,
notifications_disabled_categories=notifications_disabled_categories
)
res = self.send_and_receive(req, allow_resolution=False)
if not isinstance(res, protocol.Session):
raise Exception("Should be session")
raise Exception(f"Should be Session but was {res}")

return Session(self, res)

Expand All @@ -153,12 +161,12 @@ def get_routing_table(self, database=None):
req = protocol.GetRoutingTable(self._driver.id, database=database)
res = self.send_and_receive(req, allow_resolution=False)
if not isinstance(res, protocol.RoutingTable):
raise Exception("Should be RoutingTable")
raise Exception(f"Should be RoutingTable but was {res}")
return res

def get_connection_pool_metrics(self, address):
req = protocol.GetConnectionPoolMetrics(self._driver.id, address)
res = self.send_and_receive(req, allow_resolution=False)
if not isinstance(res, protocol.ConnectionPoolMetrics):
raise Exception("Should be ConnectionPoolMetrics")
raise Exception(f"Should be ConnectionPoolMetrics but was {res}")
return res
7 changes: 7 additions & 0 deletions nutkit/protocol/feature.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ class Feature(Enum):
# The driver offers a method for driver objects to report if they were
# configured with a or without encryption.
API_DRIVER_IS_ENCRYPTED = "Feature:API:Driver.IsEncrypted"
# The driver supports notification filters configuration.
API_DRIVER_NOTIFICATIONS_CONFIG = "Feature:API:Driver:NotificationsConfig"
# The driver offers a method for checking if a connection to the remote
# server of cluster can be established.
API_DRIVER_VERIFY_CONNECTIVITY = "Feature:API:Driver.VerifyConnectivity"
Expand All @@ -45,6 +47,9 @@ class Feature(Enum):
# If there are more than records, the driver emits a warning.
# This method is supposed to always exhaust the result stream.
API_RESULT_SINGLE_OPTIONAL = "Feature:API:Result.SingleOptional"
# The session supports notification filters configuration.
API_SESSION_NOTIFICATIONS_CONFIG = \
"Feature:API:Session:NotificationsConfig"
# The driver implements explicit configuration options for SSL.
# - enable / disable SSL
# - verify signature against system store / custom cert / not at all
Expand Down Expand Up @@ -81,6 +86,8 @@ class Feature(Enum):
BOLT_5_0 = "Feature:Bolt:5.0"
# The driver supports Bolt protocol version 5.1
BOLT_5_1 = "Feature:Bolt:5.1"
# The driver supports Bolt protocol version 5.2
BOLT_5_2 = "Feature:Bolt:5.2"
# The driver supports patching DateTimes to use UTC for Bolt 4.3 and 4.4
BOLT_PATCH_UTC = "Feature:Bolt:Patch:UTC"
# The driver supports impersonation
Expand Down
17 changes: 14 additions & 3 deletions nutkit/protocol/requests.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,9 @@ def __init__(
domainNameResolverRegistered=False, connectionTimeoutMs=None,
fetchSize=None, maxTxRetryTimeMs=None, encrypted=None,
trustedCertificates=None, liveness_check_timeout_ms=None,
max_connection_pool_size=None,
connection_acquisition_timeout_ms=None
max_connection_pool_size=None, connection_acquisition_timeout_ms=None,
notifications_min_severity=None,
notifications_disabled_categories=None
):
# Neo4j URI to connect to
self.uri = uri
Expand All @@ -87,6 +88,10 @@ def __init__(
self.livenessCheckTimeoutMs = liveness_check_timeout_ms
self.maxConnectionPoolSize = max_connection_pool_size
self.connectionAcquisitionTimeoutMs = connection_acquisition_timeout_ms
if notifications_min_severity is not None:
self.notificationsMinSeverity = notifications_min_severity
if notifications_disabled_categories is not None:
self.notificationsDisabledCategories = notifications_disabled_categories # noqa: E501
# (bool) whether to enable or disable encryption
# field missing in message: use driver default (should be False)
if encrypted is not None:
Expand Down Expand Up @@ -282,7 +287,8 @@ class NewSession:

def __init__(self, driverId, accessMode, bookmarks=None,
database=None, fetchSize=None, impersonatedUser=None,
bookmark_manager=None):
bookmark_manager=None, notifications_min_severity=None,
notifications_disabled_categories=None):
# Id of driver on backend that session should be created on
self.driverId = driverId
# Session accessmode: 'r' for read access and 'w' for write access.
Expand All @@ -292,6 +298,11 @@ def __init__(self, driverId, accessMode, bookmarks=None,
self.database = database
self.fetchSize = fetchSize
self.impersonatedUser = impersonatedUser
if notifications_min_severity is not None:
self.notificationsMinSeverity = notifications_min_severity
if notifications_disabled_categories is not None:
self.notificationsDisabledCategories = notifications_disabled_categories # noqa: E501

if bookmark_manager is not None:
self.bookmarkManagerId = bookmark_manager.id

Expand Down
Empty file.
135 changes: 135 additions & 0 deletions tests/stub/notifications_config/notifications_base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
from nutkit.frontend import Driver
import nutkit.protocol as types
from tests.shared import TestkitTestCase
from tests.stub.shared import StubServer


class NotificationsBase(TestkitTestCase):
_auth = types.AuthorizationToken("basic", principal="neo4j",
credentials="pass")

def setUp(self):
super().setUp()
self._server = StubServer(9010)
self._uri = "bolt://%s" % self._server.address
self._driver = None

def tearDown(self):
self._server.reset()
if self._driver:
self._driver.close()
return super().tearDown()

def _new_driver(self, min_sev=None, disabled_cats=None):
return Driver(self._backend, self._uri, self._auth,
notifications_min_severity=min_sev,
notifications_disabled_categories=disabled_cats)

def _run_test_get_summary(self, parameters, script_params,
script="notifications_config_driver.script"):
self._server.start(self.script_path(script),
vars_=script_params)
self._driver = self._new_driver(parameters["min_sev"],
parameters["dis_cats"])
session = self._driver.session("w", database="neo4j")
cursor = session.run("CREATE (:node)")
result = cursor.consume()
self._server.done()
return result

@staticmethod
def configs():
return [
{
"protocol": {"min_sev": "OFF", "dis_cats": None},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "OFF", '
}
},
{
"protocol": {"min_sev": "INFORMATION", "dis_cats": None},
"script": {
"#NOTIS#": '"notifications_minimum_severity": '
'"INFORMATION", '
}
},
{
"protocol": {"min_sev": "WARNING", "dis_cats": None},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "WARNING", '
}
},
{
"protocol": {"min_sev": None, "dis_cats": []},
"script": {
"#NOTIS#": '"notifications_disabled_categories": [], '
}
},
{
"protocol": {
"min_sev": "INFORMATION",
"dis_cats": ["UNRECOGNIZED"]
},
"script": {
"#NOTIS#": '"notifications_minimum_severity": '
'"INFORMATION", '
'"notifications_disabled_categories": '
'["UNRECOGNIZED"], '
}
},
{
"protocol": {
"min_sev": "WARNING",
"dis_cats": ["UNRECOGNIZED", "UNSUPPORTED"]
},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "WARNING", '
'"notifications_disabled_categories{}": '
'["UNRECOGNIZED", "UNSUPPORTED"], '
}
},
{
"protocol": {
"min_sev": "WARNING",
"dis_cats": ["PERFORMANCE"]
},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "WARNING", '
'"notifications_disabled_categories": '
'["PERFORMANCE"], '
}
},
{
"protocol": {
"min_sev": "WARNING",
"dis_cats": ["DEPRECATION"]
},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "WARNING", '
'"notifications_disabled_categories": '
'["DEPRECATION"], '
}
},
{
"protocol": {
"min_sev": "WARNING",
"dis_cats": ["GENERIC"]
},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "WARNING", '
'"notifications_disabled_categories": '
'["GENERIC"], '
}
},
{
"protocol": {
"min_sev": "WARNING",
"dis_cats": ["HINT"]
},
"script": {
"#NOTIS#": '"notifications_minimum_severity": "WARNING", '
'"notifications_disabled_categories": '
'["HINT"], '
}
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
!: BOLT 5.2

A: HELLO {#NOTIS#"[user_agent]": "*", "[routing]": "*"}
A: LOGON {"{}": "*"}
*: RESET
C: RUN "*" "*" {"[db]": "*"}
S: SUCCESS {"fields": ["t"]}
{{
C: PULL "*"
----
C: DISCARD "*"
}}
S: SUCCESS {"type": "w"}
*: RESET
?: GOODBYE
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
!: BOLT 5.2

A: HELLO {"[user_agent]": "*", "[routing]": "*", "notifications_minimum_severity": "WARNING", "notifications_disabled_categories{}":["UNSUPPORTED", "UNRECOGNIZED", "DEPRECATION", "HINT"]}
A: LOGON {"{}": "*"}
*: RESET
C: BEGIN {#NOTIS#"[mode]": "*", "[db]": "*"}
S: SUCCESS {}
C: RUN "RETURN 1 as n" {} {}
S: SUCCESS {"fields": ["n"]}
{{
C: PULL "*"
----
C: DISCARD "*"
}}
S: SUCCESS {"type": "r"}
C: COMMIT
S: SUCCESS {}
*: RESET
?: GOODBYE
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
!: BOLT 5.2

A: HELLO {"[user_agent]": "*", "[routing]": "*", "notifications_minimum_severity": "WARNING", "notifications_disabled_categories{}":["UNSUPPORTED", "UNRECOGNIZED", "DEPRECATION", "HINT"]}
A: LOGON {"{}": "*"}
*: RESET
C: RUN "RETURN 1 as n" {} {#NOTIS#"[mode]": "*", "[db]": "*"}
S: SUCCESS {"fields": ["n"]}
{{
C: PULL "*"
S: RECORD [1]
----
C: DISCARD "*"
}}
S: SUCCESS {"type": "r"}
*: RESET
?: GOODBYE
Loading