Tests failing with 1.34.1 and 1.34.2

[txtsd]

Describe the bug
Tests are failing

To Reproduce
Steps to reproduce the behavior:

go test ./...

Expected behavior
Tests shouldn't fail

Logs

?   	github.com/tidwall/tile38/cmd/tile38-benchmark	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-benchmark/az	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-cli	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-luamemtest	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-server	[no test files]
ok  	github.com/tidwall/tile38/core	0.007s
ok  	github.com/tidwall/tile38/internal/bing	0.164s
ok  	github.com/tidwall/tile38/internal/buffer	0.008s
?   	github.com/tidwall/tile38/internal/deadline	[no test files]
?   	github.com/tidwall/tile38/internal/endpoint	[no test files]
ok  	github.com/tidwall/tile38/internal/clip	0.005s
?   	github.com/tidwall/tile38/internal/hservice	[no test files]
ok  	github.com/tidwall/tile38/internal/collection	0.679s
ok  	github.com/tidwall/tile38/internal/field	2.005s
ok  	github.com/tidwall/tile38/internal/glob	1.495s
ok  	github.com/tidwall/tile38/internal/log	0.006s
ok  	github.com/tidwall/tile38/internal/object	0.005s
?   	github.com/tidwall/tile38/tests/107	[no test files]
?   	github.com/tidwall/tile38/tests/616	[no test files]
ok  	github.com/tidwall/tile38/internal/server	0.013s
ok  	github.com/tidwall/tile38/internal/sstring	0.002s

�[1mTesting keys�[0m
BOUNDS ...                   [�[32mok�[0m]
DEL ...                      [�[32mok�[0m]
DROP ...                     [�[32mok�[0m]
RENAME ...                   [�[32mok�[0m]
RENAMENX ...                 [�[32mok�[0m]
EXPIRE ...                   [�[32mok�[0m]
FSET ...                     [�[32mok�[0m]
FGET ...                     [�[32mok�[0m]
GET ...                      [�[32mok�[0m]
KEYS ...                     [�[32mok�[0m]
PERSIST ...                  [�[32mok�[0m]
SET ...                      [�[32mok�[0m]
STATS ...                    [�[32mok�[0m]
TTL ...                      [�[32mok�[0m]
EXIST ...                    [�[32mok�[0m]
FEXIST ...                   [�[32mok�[0m]
SET EX ...                   [�[32mok�[0m]
PDEL ...                     [�[32mok�[0m]
FIELDS ...                   [�[32mok�[0m]
WHEREIN ...                  [�[32mok�[0m]
WHEREEVAL ...                [�[32mok�[0m]
TYPE ...                     [�[32mok�[0m]
FLUSHDB ...                  [�[32mok�[0m]
HEALTHZ ...                  [�[32mok�[0m]
SERVER ...                   [�[32mok�[0m]
INFO ...                     [�[32mok�[0m]

�[1mTesting json�[0m
basic ...                    [�[32mok�[0m]
geojson ...                  [�[32mok�[0m]
number ...                   [�[32mok�[0m]

�[1mTesting search�[0m
KNN_BASIC ...                [�[32mok�[0m]
KNN_RANDOM ...               [�[32mok�[0m]
KNN_CURSOR ...               [�[32mok�[0m]
NEARBY_SPARSE ...            [�[32mok�[0m]
WITHIN_CIRCLE ...            [�[32mok�[0m]
WITHIN_SECTOR ...            [�[32mok�[0m]
INTERSECTS_CIRCLE ...        [�[32mok�[0m]
INTERSECTS_SECTOR ...        [�[32mok�[0m]
WITHIN ...                   [�[32mok�[0m]
WITHIN_CURSOR ...            [�[32mok�[0m]
WITHIN_CLIPBY ...            [�[32mok�[0m]
INTERSECTS ...               [�[32mok�[0m]
INTERSECTS_CURSOR ...        [�[32mok�[0m]
INTERSECTS_CLIPBY ...        [�[32mok�[0m]
SCAN_CURSOR ...              [�[32mok�[0m]
SEARCH_CURSOR ...            [�[32mok�[0m]
MATCH ...                    [�[32mok�[0m]
FIELDS ...                   [�[32mok�[0m]
BUFFER ...                   [�[32mok�[0m]

�[1mTesting testcmd�[0m
WITHIN ...                   [�[32mok�[0m]
INTERSECTS ...               [�[32mok�[0m]
INTERSECTS_CLIP ...          [�[32mok�[0m]
ExpressionErrors ...         [�[32mok�[0m]
Expressions ...              [�[32mok�[0m]

�[1mTesting client�[0m
OUTPUT ...                   [�[32mok�[0m]
CLIENT ...                   [�[32mok�[0m]

�[1mTesting scripts�[0m
BASIC ...                    [�[32mok�[0m]
ATOMIC ...                   [�[32mok�[0m]
READONLY ...                 [�[32mok�[0m]
NONATOMIC ...                [�[32mok�[0m]
VULN ...                     [�[32mok�[0m]

�[1mTesting fence�[0m
basic ...                    [�[32mok�[0m]
channel message order ...    panic: test timed out after 10m0s
	running tests:
		TestIntegration (10m0s)

goroutine 21151 [running]:
testing.(*M).startAlarm.func1()
	testing/testing.go:2373 +0x385
created by time.goFunc
	time/sleep.go:215 +0x2d

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd860, {0x5964fe76679e?, 0x0?}, 0x5964fef6deb8)
	testing/testing.go:1751 +0x3ab
testing.runTests.func1(0xc0000cd860)
	testing/testing.go:2168 +0x37
testing.tRunner(0xc0000cd860, 0xc0004b3c70)
	testing/testing.go:1690 +0xf4
testing.runTests(0xc000120690, {0x5964ff814780, 0x1, 0x1}, {0x5964fda7c1b0?, 0x5964fda7be1a?, 0x5964ff841ca0?})
	testing/testing.go:2166 +0x43d
testing.(*M).Run(0xc000504aa0)
	testing/testing.go:2034 +0x64a
main.main()
	_testmain.go:47 +0x9b

goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000203800)
	go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 33 [chan receive, 9 minutes]:
github.com/tidwall/tile38/tests.runTestGroups(0xc0000cda00)
	github.com/tidwall/tile38/tests/tests_test.go:178 +0x1c5
github.com/tidwall/tile38/tests.TestIntegration(0xc0000cda00)
	github.com/tidwall/tile38/tests/tests_test.go:61 +0x13ee
testing.tRunner(0xc0000cda00, 0x5964fef6deb8)
	testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1
	testing/testing.go:1743 +0x390

goroutine 49 [syscall, 10 minutes]:
os/signal.signal_recv()
	runtime/sigqueue.go:152 +0x29
os/signal.loop()
	os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 33
	os/signal/signal.go:151 +0x1f

goroutine 50 [chan receive, 10 minutes]:
github.com/tidwall/tile38/tests.TestIntegration.func1()
	github.com/tidwall/tile38/tests/tests_test.go:42 +0x19
created by github.com/tidwall/tile38/tests.TestIntegration in goroutine 33
	github.com/tidwall/tile38/tests/tests_test.go:41 +0x112

goroutine 316 [sleep]:
time.Sleep(0xee6b280)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/tests.runTestGroups.func2()
	github.com/tidwall/tile38/tests/tests_test.go:175 +0x9e
created by github.com/tidwall/tile38/tests.runTestGroups in goroutine 33
	github.com/tidwall/tile38/tests/tests_test.go:115 +0x1b6

goroutine 1795 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007a52c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1764
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 4960 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc002cfe9d0, 0x0)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc000389078?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveAOF(0xc002225888, 0x0, {0x5964fef92478, 0xc004be8400}, 0xc0038b2050, 0x4?)
	github.com/tidwall/tile38/internal/server/aof.go:527 +0x50f
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc002225888, {0x5964fef7c840, 0x5964ff7ce540}, {0x5964fef92478, 0xc004be8400}, 0xc0038b2050, 0xc0037a20a0, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:103 +0x1f1
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 4928
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15eb

goroutine 1643 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0005b0600)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1637
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 5100 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc002cfe9d0, 0x1)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc0049a8020?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveAOF(0xc002225888, 0x0, {0x5964fef92478, 0xc0034ec0e0}, 0xc000662050, 0x4?)
	github.com/tidwall/tile38/internal/server/aof.go:527 +0x50f
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc002225888, {0x5964fef7c840, 0x5964ff7ce540}, {0x5964fef92478, 0xc0034ec0e0}, 0xc000662050, 0xc000386280, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:103 +0x1f1
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 5219
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15eb

goroutine 2285 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000e8c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2300
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2189 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000b58000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2245
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2165 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d820c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2149
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2199 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0020ee0c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1947
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2357 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0010d2cc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2272
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 453 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000000780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 124
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 196 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007ee000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 113
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1702 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6c8e5f8, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0018dda00?, 0xc0044b0000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0018dda00, {0xc0044b0000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc0018dda00, {0xc0044b0000?, 0x5964fda1e165?, 0x5964fda1e165?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0007ea780, {0xc0044b0000?, 0xc0047f2f00?, 0xc0047f2ec0?})
	net/net.go:189 +0x45
bufio.(*Reader).fill(0xc0034a8ba0)
	bufio/bufio.go:110 +0x103
bufio.(*Reader).ReadSlice(0xc0034a8ba0, 0xa)
	bufio/bufio.go:376 +0x29
github.com/gomodule/redigo/redis.(*conn).readLine(0xc0004e4000)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:518 +0x33
github.com/gomodule/redigo/redis.(*conn).readReply(0xc0004e4000)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:598 +0x1c
github.com/gomodule/redigo/redis.(*conn).ReceiveWithTimeout(0xc0004e4000, 0x5964fed3fb20?)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:688 +0x6c
github.com/gomodule/redigo/redis.(*conn).Receive(0x0?)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:678 +0x17
github.com/gomodule/redigo/redis.PubSubConn.Receive({{0x5964fef8e6b8?, 0xc0004e4000?}})
	github.com/gomodule/redigo@v1.8.3/redis/pubsub.go:100 +0x2b
github.com/tidwall/tile38/tests.fence_channel_message_order_test.func1()
	github.com/tidwall/tile38/tests/fence_test.go:200 +0x2fd
created by github.com/tidwall/tile38/tests.fence_channel_message_order_test in goroutine 243
	github.com/tidwall/tile38/tests/fence_test.go:179 +0xa6

goroutine 1424 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func4()
	github.com/tidwall/tile38/internal/server/server.go:319 +0x45
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:318 +0x1219

goroutine 1251 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d672c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1211
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1068 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0031343c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1062
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1321 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001e54780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1315
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 726 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000001c80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 704
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2311 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0003823d0, 0x0)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc000cb8600?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).startPublishQueue(0xc002225888, 0xc002f823c0?)
	github.com/tidwall/tile38/internal/server/pubqueue.go:59 +0x106
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2296
	github.com/tidwall/tile38/internal/server/server.go:427 +0x19bf

goroutine 1529 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x5964fe75fc84, 0x9}, 0x278b, {0xc002dc2ab0, 0xf}, 0x1, {0xc002079130, 0x6}, {0x0, 0x0}, ...})
	github.com/tidwall/tile38/internal/server/server.go:442 +0x1a5e
github.com/tidwall/tile38/tests.mockOpenServer.func1()
	github.com/tidwall/tile38/tests/mock_test.go:126 +0x13e
created by github.com/tidwall/tile38/tests.mockOpenServer in goroutine 243
	github.com/tidwall/tile38/tests/mock_test.go:112 +0x455

goroutine 2103 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2091
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 3451 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc0000e4b40)
	net/http/transport.go:2519 +0xe7
created by net/http.(*Transport).dialConn in goroutine 3307
	net/http/transport.go:1875 +0x15a5

goroutine 1593 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00435a540)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1587
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 243 [chan receive, 9 minutes]:
github.com/tidwall/tile38/tests.fence_channel_message_order_test(0xc002c96500)
	github.com/tidwall/tile38/tests/fence_test.go:242 +0x325
github.com/tidwall/tile38/tests.(*testGroupSub).run.func2(0xc000694000)
	github.com/tidwall/tile38/tests/tests_test.go:237 +0x8f
github.com/tidwall/tile38/tests.(*testGroupSub).run(0x0?)
	github.com/tidwall/tile38/tests/tests_test.go:238 +0x53
github.com/tidwall/tile38/tests.runTestGroups.runTestGroups.func1.func3.1.1.1()
	github.com/tidwall/tile38/tests/tests_test.go:96 +0x85
created by github.com/tidwall/tile38/tests.runTestGroups.runTestGroups.func1.func3.1.1 in goroutine 242
	github.com/tidwall/tile38/tests/tests_test.go:90 +0xe5

goroutine 1038 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001ef3ec0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1138
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 185 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000390240)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 24
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2120 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d82000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2031
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 189 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000390300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 179
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1722 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00071ecc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1762
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2296 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x5964fe75fc84, 0x9}, 0x27bd, {0xc00434e300, 0xf}, 0x1, {0xc00438a5a8, 0x6}, {0x0, 0x0}, ...})
	github.com/tidwall/tile38/internal/server/server.go:442 +0x1a5e
github.com/tidwall/tile38/tests.mockOpenServer.func1()
	github.com/tidwall/tile38/tests/mock_test.go:126 +0x13e
created by github.com/tidwall/tile38/tests.mockOpenServer in goroutine 303
	github.com/tidwall/tile38/tests/mock_test.go:112 +0x455

goroutine 1776 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x5964fe75fc84, 0x9}, 0x27a9, {0xc00152f840, 0xf}, 0x1, {0xc0012f4380, 0x6}, {0x0, 0x0}, ...})
	github.com/tidwall/tile38/internal/server/server.go:442 +0x1a5e
github.com/tidwall/tile38/tests.mockOpenServer.func1()
	github.com/tidwall/tile38/tests/mock_test.go:126 +0x13e
created by github.com/tidwall/tile38/tests.mockOpenServer in goroutine 296
	github.com/tidwall/tile38/tests/mock_test.go:112 +0x455

goroutine 815 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0023ba840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 888
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 280 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007ee180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 11
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 484 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007ee0c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 418
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 568 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007ee6c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 562
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1907 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002caef00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1869
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1297 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0037cd140)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1214
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 338 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000a60000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 29
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 324 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0003903c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 129
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 715 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000a60300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 871
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1571 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00256fb50, 0x957)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0x5964fda82c34?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).processLives(0xc001f92388, 0xc0020791a0)
	github.com/tidwall/tile38/internal/server/live.go:61 +0x165
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:415 +0x1733

goroutine 369 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d88000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 177
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1194 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001faf080)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1125
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1569 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002f823c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1529
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 650 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000e2ca80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 644
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1707 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc000859f10, 0x1)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0x5964fee2e701?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func9()
	github.com/tidwall/tile38/internal/server/pubsub.go:346 +0xb4
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription in goroutine 1706
	github.com/tidwall/tile38/internal/server/pubsub.go:327 +0x52d

goroutine 2033 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc00036a110, 0x0)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc000fe9f80?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).startPublishQueue(0xc003256a88, 0xc0007e1ea0?)
	github.com/tidwall/tile38/internal/server/pubqueue.go:59 +0x106
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1776
	github.com/tidwall/tile38/internal/server/server.go:427 +0x19bf

goroutine 262 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000e2c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 26
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2295 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003210240)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2289
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2573 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007a5680)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2505
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1570 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6fa66e0, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc002977a00?, 0x10?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc002977a00)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc002977a00)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc002fe9b40)
	net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc002fe9b40)
	net/tcpsock.go:372 +0x30
net/http.(*Server).Serve(0xc000f114a0, {0x5964fef86a50, 0xc002fe9b40})
	net/http/server.go:3330 +0x30c
net/http.Serve(...)
	net/http/server.go:2858
github.com/tidwall/tile38/internal/server.Serve.func8()
	github.com/tidwall/tile38/internal/server/server.go:405 +0x157
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:400 +0x16ac

goroutine 410 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000a60840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 127
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 428 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000390cc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 422
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1978 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6c8e4e0, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0012f3980?, 0x10?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0012f3980)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0012f3980)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00036a0c0)
	net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc00036a0c0)
	net/tcpsock.go:372 +0x30
net/http.(*Server).Serve(0xc0000eed20, {0x5964fef86a50, 0xc00036a0c0})
	net/http/server.go:3330 +0x30c
net/http.Serve(...)
	net/http/server.go:2858
github.com/tidwall/tile38/internal/server.Serve.func8()
	github.com/tidwall/tile38/internal/server/server.go:405 +0x157
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1776
	github.com/tidwall/tile38/internal/server/server.go:400 +0x16ac

goroutine 362 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000cc2180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 356
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 436 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000ac2540)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 318
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2098 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc001379010, 0x7)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc00053c800?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveAOF(0xc003256a88, 0x0, {0x5964fef92478, 0xc001e7c080}, 0xc001b02050, 0x4?)
	github.com/tidwall/tile38/internal/server/aof.go:527 +0x50f
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc003256a88, {0x5964fef7c840, 0x5964ff7ce540}, {0x5964fef92478, 0xc001e7c080}, 0xc001b02050, 0xc00384c8c0, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:103 +0x1f1
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 2069
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15eb

goroutine 2227 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003210000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2143
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1486 [sleep]:
time.Sleep(0xee6b280)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.(*Server).processLives.func1()
	github.com/tidwall/tile38/internal/server/live.go:36 +0x6f
created by github.com/tidwall/tile38/internal/server.(*Server).processLives in goroutine 1571
	github.com/tidwall/tile38/internal/server/live.go:29 +0xf9

goroutine 2292 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2289
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 2232 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0032100c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2264
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 352 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000a60780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 382
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1973 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc0037baeb0)
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:183 +0x55
created by github.com/tidwall/tile38/internal/endpoint.NewManager in goroutine 1776
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:161 +0xc5

goroutine 1362 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001726840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1276
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1398 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001ef2e40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1371
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 679 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d88900)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 600
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 584 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000000a80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 641
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 978 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001e54cc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 953
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 504 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000e2dbc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 388
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 963 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0024e4180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 891
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1704 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6c8e3c8, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc001d68c00?, 0xc000802000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001d68c00, {0xc000802000, 0xffff, 0xffff})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc001d68c00, {0xc000802000?, 0x5964fe6491c5?, 0xc001f92538?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002bf6168, {0xc000802000?, 0xc001f92538?, 0xc000575d30?})
	net/net.go:189 +0x45
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5964fef92070, 0xc002bf6168})
	github.com/tidwall/tile38/internal/server/server.go:555 +0x503
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1422
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1574 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc001f92388, 0x3b9aca00, 0xc0001a3f90)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).watchAutoGC(0xc001f92388, 0xc00049cb00?)
	github.com/tidwall/tile38/internal/server/server.go:731 +0x86
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:421 +0x1877

goroutine 1095 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003276840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1089
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1162 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001234b40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1156
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1230 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002b20e40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1224
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 550 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000e2dc80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 474
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 937 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002980180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 911
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 5219 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc002cfe1c0?)
	runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc001af6630?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5964fef92070, 0xc004cc40f8})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15f8
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 2220
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 611 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000e2dd40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 578
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 536 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007ee900)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 476
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1705 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc000859ec0?)
	runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc002feb810?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5964fef92070, 0xc002bf6170})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15f8
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1422
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 2169 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2141
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 2000 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003276000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2029
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 4928 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc00036b900?)
	runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc002b93320?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5964fef92070, 0xc000089190})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15f8
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 2220
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 693 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000000b40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 660
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1423 [sleep]:
time.Sleep(0xee6b280)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.Serve.func3()
	github.com/tidwall/tile38/internal/server/server.go:314 +0x38
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:311 +0x11a5

goroutine 2106 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d82180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2091
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 745 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.readMemStats.func1()
	github.com/tidwall/tile38/internal/server/stats.go:38 +0x45
created by github.com/tidwall/tile38/internal/server.readMemStats in goroutine 823
	github.com/tidwall/tile38/internal/server/stats.go:31 +0x91

goroutine 899 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0020ee480)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 634
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1491 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002a249c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1475
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 606 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000a61c80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 736
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1113 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002a5c300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1008
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2159 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0023ba000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2180
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 781 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001726780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 743
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1613 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00435ab40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1546
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1658 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0030b0840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1652
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 998 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0032b1080)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 927
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 766 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d82e40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 760
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1015 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0032118c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 976
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 827 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000cc2f00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 628
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1706 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6fa6398, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc001d68d00?, 0xc0007cc070?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001d68d00, {0xc0007cc070, 0xffff, 0xffff})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc001d68d00, {0xc0007cc070?, 0x5964fda1e42b?, 0xc000b5d888?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc002bf6170, {0xc0007cc070?, 0x762af6c12f58?, 0x1a?})
	net/net.go:189 +0x45
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc0007cc050)
	github.com/tidwall/tile38/internal/server/server.go:1577 +0x50
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc001f92388, {0x5964fef92478, 0xc003104a80}, 0xc0007cc050, 0xc0041756d0, 0x0)
	github.com/tidwall/tile38/internal/server/pubsub.go:392 +0x570
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc001f92388, {0x5964fef7c7e0, 0x5964ff8657e0}, {0x5964fef92478, 0xc003104a80}, 0xc0007cc050, 0xc0041756d0, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:105 +0x9c5
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 1705
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15eb

goroutine 852 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0020ee540)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 622
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1181 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002e10a80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1175
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 856 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0020ee600)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 803
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2433 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc000e4a190, 0x99)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc000e60240?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc000e60240)
	github.com/tidwall/tile38/internal/server/hooks.go:623 +0xa7
created by github.com/tidwall/tile38/internal/server.(*Hook).Open in goroutine 2376
	github.com/tidwall/tile38/internal/server/hooks.go:564 +0x128

goroutine 1351 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001e54a80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1392
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2156 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2180
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 840 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000cc2000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 748
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1238 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0024e43c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1143
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1625 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6c76470, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0018dd900?, 0xc000792000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0018dd900, {0xc000792000, 0xffff, 0xffff})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc0018dd900, {0xc000792000?, 0x5964fe6491c5?, 0xc001f92538?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0007ea768, {0xc000792000?, 0xc001f92538?, 0xc001bafd30?})
	net/net.go:189 +0x45
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5964fef92070, 0xc0007ea768})
	github.com/tidwall/tile38/internal/server/server.go:555 +0x503
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1422
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 992 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002b20cc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1009
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2069 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc00196db00?)
	runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc0015ba410?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5964fef92070, 0xc0003767c0})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15f8
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1974
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1428 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002a24000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1389
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1576 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc001f92388, 0x3b9aca00, 0xc004329f98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).backgroundSyncAOF(0x0?, 0x0?)
	github.com/tidwall/tile38/internal/server/server.go:806 +0x5a
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:425 +0x1953

goroutine 1577 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc002fe9b90, 0x1)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc000197200?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).startPublishQueue(0xc001f92388, 0x0?)
	github.com/tidwall/tile38/internal/server/pubqueue.go:59 +0x106
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:427 +0x19bf

goroutine 1446 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002688600)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1410
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1535 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0030e8cc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1603
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2375 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d36780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2317
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1575 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc001f92388, 0x5f5e100, 0xc000a06f98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).backgroundExpiring(0x0?, 0x0?)
	github.com/tidwall/tile38/internal/server/expire.go:18 +0x5a
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:423 +0x18e5

goroutine 1573 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc001f92388, 0x2540be400, 0xc002192f98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).watchLuaStatePool(0xc003257888?, 0xc002192fa8?)
	github.com/tidwall/tile38/internal/server/server.go:798 +0x5f
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:419 +0x180b

goroutine 1572 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc001f92388, 0xee6b2800, 0xc00202df98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).watchOutOfMemory(0x5964fef92070?, 0xc0007eaf70?)
	github.com/tidwall/tile38/internal/server/server.go:791 +0x5a
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:417 +0x179f

goroutine 1883 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002c2fe00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1877
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1378 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001fae000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1337
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1977 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001a82180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1776
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2224 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6c8e080, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000dc3b80?, 0x10?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000dc3b80)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000dc3b80)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000382380)
	net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc000382380)
	net/tcpsock.go:372 +0x30
net/http.(*Server).Serve(0xc0001641e0, {0x5964fef86a50, 0xc000382380})
	net/http/server.go:3330 +0x30c
net/http.Serve(...)
	net/http/server.go:2858
github.com/tidwall/tile38/internal/server.Serve.func8()
	github.com/tidwall/tile38/internal/server/server.go:405 +0x157
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2296
	github.com/tidwall/tile38/internal/server/server.go:400 +0x16ac

goroutine 2172 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002e10000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2141
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1817 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0008783c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1811
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1974 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc0019f8d60?)
	runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc0019f8e00?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func1()
	github.com/tidwall/tile38/internal/server/server.go:484 +0x1b2
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc003256a88)
	github.com/tidwall/tile38/internal/server/server.go:495 +0x4ef
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x25
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1776
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1422 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x762af6c77078, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00387c980?, 0x900000036?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00387c980)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00387c980)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0038429c0)
	net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc0038429c0)
	net/tcpsock.go:372 +0x30
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc001f92388)
	github.com/tidwall/tile38/internal/server/server.go:492 +0x31c
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x25
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1529
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1421 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc0020db450)
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:183 +0x55
created by github.com/tidwall/tile38/internal/endpoint.NewManager in goroutine 1529
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:161 +0xc5

goroutine 1556 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001e54b40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1483
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1963 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0052d9bc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1938
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1517 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00416ba40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1488
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2223 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0005b0240)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2296
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1923 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002b09e00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1757
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 2040 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc005501ec0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2034
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 1806 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0031de050, 0x1)
	runtime/sema.go:587 +0x15a
sync.(*Cond).Wait(0xc0007a55c0?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc0007a55c0)
	github.com/tidwall/tile38/internal/server/hooks.go:623 +0xa7
created by github.com/tidwall/tile38/internal/server.(*Hook).Open in goroutine 1685
	github.com/tidwall/tile38/internal/server/hooks.go:564 +0x128

goroutine 2220 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc0001c8d60?)
	runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc0001c8e00?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func1()
	github.com/tidwall/tile38/internal/server/server.go:484 +0x1b2
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc002225888)
	github.com/tidwall/tile38/internal/server/server.go:495 +0x4ef
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x25
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 2296
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1740 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0005b1380)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1782
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x2ac

goroutine 3450 [select, 9 minutes]:
net/http.(*persistConn).readLoop(0xc0000e4b40)
	net/http/transport.go:2325 +0xca5
created by net/http.(*Transport).dialConn in goroutine 3307
	net/http/transport.go:1874 +0x154f

goroutine 2219 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:300 +0xf4
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc002bb90e0)
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:183 +0x55
created by github.com/tidwall/tile38/internal/endpoint.NewManager in goroutine 2296
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:161 +0xc5
FAIL	github.com/tidwall/tile38/tests	600.033s
FAIL

Operating System (please complete the following information):

• OS: Arch Linuux
• CPU: Ryzen 2400G
• Version: 1.34.1
• Container: None

Additional context
Add any other context about the problem here.

[tidwall]

Looks like there may be a bug in one of the tests.
Did you re-run, and did it happen to work the second time?

[txtsd]

Yes, I ran them several times in a clean chroot and they failed every time.

[txtsd]

Tests are also failing with 1.34.2

?   	github.com/tidwall/tile38/cmd/tile38-benchmark	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-benchmark/az	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-cli	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-luamemtest	[no test files]
?   	github.com/tidwall/tile38/cmd/tile38-server	[no test files]
ok  	github.com/tidwall/tile38/core	0.005s
ok  	github.com/tidwall/tile38/internal/bing	0.074s
ok  	github.com/tidwall/tile38/internal/buffer	0.002s
ok  	github.com/tidwall/tile38/internal/clip	0.003s
ok  	github.com/tidwall/tile38/internal/collection	0.212s
?   	github.com/tidwall/tile38/internal/deadline	[no test files]
?   	github.com/tidwall/tile38/internal/endpoint	[no test files]
ok  	github.com/tidwall/tile38/internal/field	2.020s
ok  	github.com/tidwall/tile38/internal/glob	1.290s
?   	github.com/tidwall/tile38/internal/hservice	[no test files]
ok  	github.com/tidwall/tile38/internal/log	0.003s
ok  	github.com/tidwall/tile38/internal/object	0.002s
ok  	github.com/tidwall/tile38/internal/server	0.014s
ok  	github.com/tidwall/tile38/internal/sstring	0.002s

Testing keys
BOUNDS ...                   [ok]
DEL ...                      [ok]
DROP ...                     [ok]
RENAME ...                   [ok]
RENAMENX ...                 [ok]
EXPIRE ...                   [ok]
FSET ...                     [ok]
FGET ...                     [ok]
GET ...                      [ok]
KEYS ...                     [ok]
PERSIST ...                  [ok]
SET ...                      [ok]
STATS ...                    [ok]
TTL ...                      [ok]
EXIST ...                    [ok]
FEXIST ...                   [ok]
SET EX ...                   [ok]
PDEL ...                     [ok]
FIELDS ...                   [ok]
WHEREIN ...                  [ok]
WHEREEVAL ...                [ok]
TYPE ...                     [ok]
FLUSHDB ...                  [ok]
HEALTHZ ...                  [ok]
SERVER ...                   [ok]
INFO ...                     [ok]

Testing json
basic ...                    [ok]
geojson ...                  [ok]
number ...                   [ok]

Testing search
KNN_BASIC ...                [ok]
KNN_RANDOM ...               [ok]
KNN_CURSOR ...               [ok]
NEARBY_SPARSE ...            [ok]
WITHIN_CIRCLE ...            [ok]
WITHIN_SECTOR ...            [ok]
INTERSECTS_CIRCLE ...        [ok]
INTERSECTS_SECTOR ...        [ok]
WITHIN ...                   [ok]
WITHIN_CURSOR ...            [ok]
WITHIN_CLIPBY ...            [ok]
INTERSECTS ...               [ok]
INTERSECTS_CURSOR ...        [ok]
INTERSECTS_CLIPBY ...        [ok]
SCAN_CURSOR ...              [ok]
SEARCH_CURSOR ...            [ok]
MATCH ...                    [ok]
FIELDS ...                   [ok]
BUFFER ...                   [ok]

Testing testcmd
WITHIN ...                   [ok]
INTERSECTS ...               [ok]
INTERSECTS_CLIP ...          [ok]
ExpressionErrors ...         [ok]
Expressions ...              [ok]

Testing client
OUTPUT ...                   [ok]
CLIENT ...                   [ok]

Testing scripts
BASIC ...                    [ok]
ATOMIC ...                   [ok]
READONLY ...                 [ok]
NONATOMIC ...                [ok]
VULN ...                     [ok]

Testing fence
basic ...                    [ok]
channel message order ...    panic: test timed out after 10m0s
	running tests:
		TestIntegration (10m0s)

goroutine 21221 [running]:
testing.(*M).startAlarm.func1()
	testing/testing.go:2484 +0x394
created by time.goFunc
	time/sleep.go:215 +0x2d

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0003f0e00, {0x5b454c0e9884?, 0xc0004fdb30?}, 0x5b454c9307d8)
	testing/testing.go:1859 +0x431
testing.runTests.func1(0xc0003f0e00)
	testing/testing.go:2279 +0x37
testing.tRunner(0xc0003f0e00, 0xc0004fdc70)
	testing/testing.go:1792 +0xf4
testing.runTests(0xc000026648, {0x5b454d2339f0, 0x1, 0x1}, {0x5b454d264260?, 0x7?, 0x5b454d261f60?})
	testing/testing.go:2277 +0x4b4
testing.(*M).Run(0xc000124fa0)
	testing/testing.go:2142 +0x64a
main.main()
	_testmain.go:47 +0x9b

goroutine 36 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000269800)
	go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 914 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0022760c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 814
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 38 [chan receive, 9 minutes]:
github.com/tidwall/tile38/tests.runTestGroups(0xc0003f0fc0)
	github.com/tidwall/tile38/tests/tests_test.go:178 +0x1c5
github.com/tidwall/tile38/tests.TestIntegration(0xc0003f0fc0)
	github.com/tidwall/tile38/tests/tests_test.go:61 +0x13ee
testing.tRunner(0xc0003f0fc0, 0x5b454c9307d8)
	testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
	testing/testing.go:1851 +0x413

goroutine 40 [syscall, 10 minutes]:
os/signal.signal_recv()
	runtime/sigqueue.go:152 +0x29
os/signal.loop()
	os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 38
	os/signal/signal.go:152 +0x1f

goroutine 41 [chan receive, 10 minutes]:
github.com/tidwall/tile38/tests.TestIntegration.func1()
	github.com/tidwall/tile38/tests/tests_test.go:42 +0x19
created by github.com/tidwall/tile38/tests.TestIntegration in goroutine 38
	github.com/tidwall/tile38/tests/tests_test.go:41 +0x112

goroutine 344 [sleep]:
time.Sleep(0xee6b280)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/tests.runTestGroups.func2()
	github.com/tidwall/tile38/tests/tests_test.go:175 +0x9e
created by github.com/tidwall/tile38/tests.runTestGroups in goroutine 38
	github.com/tidwall/tile38/tests/tests_test.go:115 +0x1b6

goroutine 454 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0001d6b40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 350
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1743 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884cbdb88, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc002d2c580?, 0xc000baf000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002d2c580, {0xc000baf000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc002d2c580, {0xc000baf000?, 0x5b454b382a45?, 0x5b454b493713?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0031526f8, {0xc000baf000?, 0xc001961bb0?, 0x5b454b72c930?})
	net/net.go:194 +0x45
bufio.(*Reader).fill(0xc000707200)
	bufio/bufio.go:113 +0x103
bufio.(*Reader).ReadSlice(0xc000707200, 0xa)
	bufio/bufio.go:380 +0x29
github.com/gomodule/redigo/redis.(*conn).readLine(0xc0001faaa0)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:518 +0x30
github.com/gomodule/redigo/redis.(*conn).readReply(0xc0001faaa0)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:598 +0x1c
github.com/gomodule/redigo/redis.(*conn).ReceiveWithTimeout(0xc0001faaa0, 0x5b454b3d0799?)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:688 +0x67
github.com/gomodule/redigo/redis.(*conn).Receive(0xc0001faa01?)
	github.com/gomodule/redigo@v1.8.3/redis/conn.go:678 +0x17
github.com/gomodule/redigo/redis.PubSubConn.Receive({{0x5b454c952160?, 0xc0001faaa0?}})
	github.com/gomodule/redigo@v1.8.3/redis/pubsub.go:100 +0x2b
github.com/tidwall/tile38/tests.fence_channel_message_order_test.func1()
	github.com/tidwall/tile38/tests/fence_test.go:199 +0x30d
created by github.com/tidwall/tile38/tests.fence_channel_message_order_test in goroutine 274
	github.com/tidwall/tile38/tests/fence_test.go:180 +0xcf

goroutine 10997 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 11017
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1892 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002c72b40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1865
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 12076 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc006b42600)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 11837
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1576 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00477c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1558
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 2145 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002276000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2041
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 20463 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc005ab8300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 20457
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 2106 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0013b61d0, 0x92)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc002276480?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc002276480)
	github.com/tidwall/tile38/internal/server/hooks.go:623 +0xa5
created by github.com/tidwall/tile38/internal/server.(*Hook).Open in goroutine 1887
	github.com/tidwall/tile38/internal/server/hooks.go:564 +0x11f

goroutine 13998 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884cbdca0, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc001cfa500?, 0x5b454c793ec0?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc001cfa500)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc001cfa500)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc006276ac0)
	net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc006276ac0)
	net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc004c91300, {0x5b454c94a2d0, 0xc006276ac0})
	net/http/server.go:3424 +0x30c
net/http.Serve(...)
	net/http/server.go:2932
github.com/tidwall/tile38/internal/server.Serve.func8()
	github.com/tidwall/tile38/internal/server/server.go:405 +0x157
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 13945
	github.com/tidwall/tile38/internal/server/server.go:400 +0x16ac

goroutine 3433 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 3430
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 15988 [select, 9 minutes]:
net/http.(*persistConn).writeLoop(0xc0052650e0)
	net/http/transport.go:2590 +0xe7
created by net/http.(*Transport).dialConn in goroutine 15934
	net/http/transport.go:1945 +0x17a5

goroutine 212 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0001d6000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 113
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 655 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00178c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 572
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 26 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000150480)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 66
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 15987 [select, 9 minutes]:
net/http.(*persistConn).readLoop(0xc0052650e0)
	net/http/transport.go:2395 +0xc5f
created by net/http.(*Transport).dialConn in goroutine 15934
	net/http/transport.go:1944 +0x174f

goroutine 1818 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0020e1ec0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1724
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 21117 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc005ce60c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 21169
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1445 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000bb8fc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1383
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 14037 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc006276b10, 0x0)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0x5?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).startPublishQueue(0xc005815c08, 0xc002ab6e70?)
	github.com/tidwall/tile38/internal/server/pubqueue.go:59 +0x106
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 13945
	github.com/tidwall/tile38/internal/server/server.go:427 +0x19bf

goroutine 431 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0004fa300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 519
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 596 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00071c600)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 511
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1419 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x5b454c0e2d13, 0x9}, 0x278d, {0xc000011fd0, 0xf}, 0x1, {0xc0025ceec0, 0x6}, {0x0, 0x0}, ...})
	github.com/tidwall/tile38/internal/server/server.go:442 +0x1a5e
github.com/tidwall/tile38/tests.mockOpenServer.func1()
	github.com/tidwall/tile38/tests/mock_test.go:126 +0x13e
created by github.com/tidwall/tile38/tests.mockOpenServer in goroutine 274
	github.com/tidwall/tile38/tests/mock_test.go:112 +0x455

goroutine 174 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000416000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 67
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 119 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00071c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 51
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 368 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0001d6840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 483
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 151 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00097c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 129
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 274 [chan receive, 9 minutes]:
github.com/tidwall/tile38/tests.fence_channel_message_order_test(0xc004039310)
	github.com/tidwall/tile38/tests/fence_test.go:249 +0x3c7
github.com/tidwall/tile38/tests.(*testGroupSub).run.func2(0xc00011fc20)
	github.com/tidwall/tile38/tests/tests_test.go:237 +0x8f
github.com/tidwall/tile38/tests.(*testGroupSub).run(0x0?)
	github.com/tidwall/tile38/tests/tests_test.go:238 +0x53
github.com/tidwall/tile38/tests.runTestGroups.runTestGroups.func1.func3.1.1.1()
	github.com/tidwall/tile38/tests/tests_test.go:96 +0x85
created by github.com/tidwall/tile38/tests.runTestGroups.runTestGroups.func1.func3.1.1 in goroutine 273
	github.com/tidwall/tile38/tests/tests_test.go:90 +0xcf

goroutine 11000 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00345a000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 11017
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1712 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x5b454c0e2d13, 0x9}, 0x279b, {0xc002104a70, 0xf}, 0x1, {0xc002279fe0, 0x6}, {0x0, 0x0}, ...})
	github.com/tidwall/tile38/internal/server/server.go:442 +0x1a5e
github.com/tidwall/tile38/tests.mockOpenServer.func1()
	github.com/tidwall/tile38/tests/mock_test.go:126 +0x13e
created by github.com/tidwall/tile38/tests.mockOpenServer in goroutine 324
	github.com/tidwall/tile38/tests/mock_test.go:112 +0x455

goroutine 14999 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002dea180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 15077
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 12073 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 11837
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 21194 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc004a9e3c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 21175
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 74 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d0e000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 161
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1872 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003d766c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1921
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1674 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00047f200)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1746
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 641 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0009d83c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 626
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 20583 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0xc006a34840?)
	runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0xc00240f0b0?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5b454c955ab0, 0xc000012cf0})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15ec
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 13994
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1915 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0xc0055fc780?)
	runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0xc005155a40?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5b454c955ab0, 0xc004b68858})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15ec
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1630
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1402 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00301c000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1427
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 372 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00097c0c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 52
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1629 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc0021305f0)
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:183 +0x55
created by github.com/tidwall/tile38/internal/endpoint.NewManager in goroutine 1712
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:161 +0xc5

goroutine 1029 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0018fdd40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 996
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 300 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000416180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 8
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 263 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d0e840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 417
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 689 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d0f8c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 674
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 615 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00071ccc0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 562
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1592 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc002c81e50)
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:183 +0x55
created by github.com/tidwall/tile38/internal/endpoint.NewManager in goroutine 1419
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:161 +0xc5

goroutine 744 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002045d40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 699
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1616 [sleep]:
time.Sleep(0xee6b280)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.(*Server).processLives.func1()
	github.com/tidwall/tile38/internal/server/live.go:36 +0x6f
created by github.com/tidwall/tile38/internal/server.(*Server).processLives in goroutine 1606
	github.com/tidwall/tile38/internal/server/live.go:29 +0xf9

goroutine 1392 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002debec0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1370
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1507 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003a39380)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1455
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 467 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00097c900)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 390
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1602 [sleep]:
time.Sleep(0xee6b280)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.Serve.func3()
	github.com/tidwall/tile38/internal/server/server.go:314 +0x38
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:311 +0x11a5

goroutine 634 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001d16900)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 721
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1236 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001f8f140)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1142
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 408 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0009d8780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 352
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1793 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0007fc180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1712
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 680 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000151980)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 590
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1695 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002042840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1780
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 887 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0021c5e00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 881
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1801 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc001ab2350, 0x0)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc001573e00?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).startPublishQueue(0xc00095e388, 0xc000020700?)
	github.com/tidwall/tile38/internal/server/pubqueue.go:59 +0x106
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1712
	github.com/tidwall/tile38/internal/server/server.go:427 +0x19bf

goroutine 1046 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002fb6780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 827
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 802 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00212f980)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 792
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1533 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0043803c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1541
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 500 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d0e0c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 449
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 458 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00097c180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 184
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1568 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00071c0c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1663
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 13997 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc005ce6480)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 13945
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1610 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc002ec6708, 0x5f5e100, 0xc0011a6f98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).backgroundExpiring(0xc003a42008?, 0xc000020700?)
	github.com/tidwall/tile38/internal/server/expire.go:18 +0x5a
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:423 +0x18e5

goroutine 1269 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0028c1c80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1220
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 446 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0009d80c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 515
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 771 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d0f440)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 755
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1604 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00459c180)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1419
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 11436 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0020438c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 11340
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 13994 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0xc0000f8d60?)
	runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0xc000012cf0?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func1()
	github.com/tidwall/tile38/internal/server/server.go:484 +0x1b2
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc005815c08)
	github.com/tidwall/tile38/internal/server/server.go:495 +0x4ef
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x25
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 13945
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 963 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001254780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 923
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1020 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00133c240)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1185
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 13993 [sleep]:
time.Sleep(0x3b9aca00)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc005ce9090)
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:183 +0x55
created by github.com/tidwall/tile38/internal/endpoint.NewManager in goroutine 13945
	github.com/tidwall/tile38/internal/endpoint/endpoint.go:161 +0xc5

goroutine 625 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00071c780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 489
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1607 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc002ec6708, 0xee6b2800, 0xc0015d1f98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).watchOutOfMemory(0x5b454b58c8e4?, 0xc000020700?)
	github.com/tidwall/tile38/internal/server/server.go:791 +0x5a
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:417 +0x179f

goroutine 1689 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0006f7150, 0x1)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc000a60c00?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc000a60c00)
	github.com/tidwall/tile38/internal/server/hooks.go:623 +0xa5
created by github.com/tidwall/tile38/internal/server.(*Hook).Open in goroutine 1745
	github.com/tidwall/tile38/internal/server/hooks.go:564 +0x11f

goroutine 659 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000d0f2c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 564
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1635 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003b0e9c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1501
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1008 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00303c6c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1099
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 763 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.readMemStats.func1()
	github.com/tidwall/tile38/internal/server/stats.go:38 +0x45
created by github.com/tidwall/tile38/internal/server.readMemStats in goroutine 811
	github.com/tidwall/tile38/internal/server/stats.go:31 +0x91

goroutine 949 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000df5140)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 929
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1084 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000151200)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1120
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1605 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884eae720, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0045a4a80?, 0x5b454c793ec0?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0045a4a80)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0045a4a80)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0045e1c40)
	net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc0045e1c40)
	net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0043c9400, {0x5b454c94a2d0, 0xc0045e1c40})
	net/http/server.go:3424 +0x30c
net/http.Serve(...)
	net/http/server.go:2932
github.com/tidwall/tile38/internal/server.Serve.func8()
	github.com/tidwall/tile38/internal/server/server.go:405 +0x157
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:400 +0x16ac

goroutine 728 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001efcb40)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 711
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 12367 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000416240)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 12795
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1794 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884eade60, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00288b700?, 0x5b454c793ec0?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00288b700)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00288b700)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc001ab2300)
	net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc001ab2300)
	net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0008b2900, {0x5b454c94a2d0, 0xc001ab2300})
	net/http/server.go:3424 +0x30c
net/http.Serve(...)
	net/http/server.go:2932
github.com/tidwall/tile38/internal/server.Serve.func8()
	github.com/tidwall/tile38/internal/server/server.go:405 +0x157
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1712
	github.com/tidwall/tile38/internal/server/server.go:400 +0x16ac

goroutine 13945 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x5b454c0e2d13, 0x9}, 0x27be, {0xc003ae4ab0, 0xf}, 0x1, {0xc003949838, 0x6}, {0x0, 0x0}, ...})
	github.com/tidwall/tile38/internal/server/server.go:442 +0x1a5e
github.com/tidwall/tile38/tests.mockOpenServer.func1()
	github.com/tidwall/tile38/tests/mock_test.go:126 +0x13e
created by github.com/tidwall/tile38/tests.mockOpenServer in goroutine 331
	github.com/tidwall/tile38/tests/mock_test.go:112 +0x455

goroutine 2537 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00047e000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2514
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1603 [chan receive, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func4()
	github.com/tidwall/tile38/internal/server/server.go:319 +0x45
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:318 +0x1219

goroutine 873 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00284d980)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 979
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 849 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002228300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 833
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1209 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00133d5c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 986
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 14977 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc003b0e000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 14672
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 863 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002228780)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 818
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1196 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0028c00c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1181
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1611 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc002ec6708, 0x3b9aca00, 0xc000b6af98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).backgroundSyncAOF(0xd?, 0x0?)
	github.com/tidwall/tile38/internal/server/server.go:806 +0x5a
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:425 +0x1953

goroutine 1598 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0023b8810, 0x2)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0x5b454c7fdb01?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func9()
	github.com/tidwall/tile38/internal/server/pubsub.go:346 +0xb4
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription in goroutine 1597
	github.com/tidwall/tile38/internal/server/pubsub.go:327 +0x413

goroutine 1984 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0004160c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2023
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 935 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001384840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 910
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1068 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0029169c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1245
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1090 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00276e3c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1073
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1705 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884cbc378, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc001cfa600?, 0xc000b0c000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001cfa600, {0xc000b0c000, 0xffff, 0xffff})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc001cfa600, {0xc000b0c000?, 0x433c5765?, 0xc002ec68b8?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000f94d80, {0xc000b0c000?, 0xc001bf8aa0?, 0x0?})
	net/net.go:194 +0x45
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5b454c955ab0, 0xc000f94d80})
	github.com/tidwall/tile38/internal/server/server.go:555 +0x4fa
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1601
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1163 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0032f29c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1187
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 21120 [chan send, 9 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x3e
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 21142
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1958 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc005253e00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1904
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1338 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00246e9c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1184
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 2114 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc005498000)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2090
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1630 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0xc0011aad60?)
	runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0xc004b68858?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func1()
	github.com/tidwall/tile38/internal/server/server.go:484 +0x1b2
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc00095e388)
	github.com/tidwall/tile38/internal/server/server.go:495 +0x4ef
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x25
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1712
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1123 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0032f3e00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 984
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 21219 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000f92840)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 21142
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 2013 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000416300)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2104
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1612 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0045e1d50, 0x5)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc0001d1200?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).startPublishQueue(0xc002ec6708, 0xc001fd08b8?)
	github.com/tidwall/tile38/internal/server/pubqueue.go:59 +0x106
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:427 +0x19bf

goroutine 1349 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00276fb00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1329
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1626 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0xc0021754c0?)
	runtime/sema.go:110 +0x25
sync.(*WaitGroup).Wait(0xc002279910?)
	sync/waitgroup.go:118 +0x48
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5b454c955ab0, 0xc0005b2ac0})
	github.com/tidwall/tile38/internal/server/server.go:631 +0x15ec
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1601
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1284 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002f7b140)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1149
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1606 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc003ded550, 0x957)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc00189bf7c?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).processLives(0xc002ec6708, 0xc002b4df10)
	github.com/tidwall/tile38/internal/server/live.go:61 +0x165
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:415 +0x1733

goroutine 1608 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc002ec6708, 0x2540be400, 0xc0009c9f98)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).watchLuaStatePool(0x5b454b58c8e4?, 0xc0009c9fb0?)
	github.com/tidwall/tile38/internal/server/server.go:798 +0x5f
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:419 +0x180b

goroutine 1609 [sleep]:
time.Sleep(0xbebc200)
	runtime/time.go:338 +0x165
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc002ec6708, 0x3b9aca00, 0xc000e5ff90)
	github.com/tidwall/tile38/internal/server/server.go:785 +0x45
github.com/tidwall/tile38/internal/server.(*Server).watchAutoGC(0xc002ec6708, 0xc0025ce2b0?)
	github.com/tidwall/tile38/internal/server/server.go:731 +0x86
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:421 +0x1877

goroutine 1597 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884e2bd08, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000481a80?, 0xc000e2c070?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000481a80, {0xc000e2c070, 0xffff, 0xffff})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc000481a80, {0xc000e2c070?, 0xc000af9470?, 0x5b454bf623e2?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005b2ac0, {0xc000e2c070?, 0xc0006a63c0?, 0x1454b3d09f3?})
	net/net.go:194 +0x45
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc000e2c050)
	github.com/tidwall/tile38/internal/server/server.go:1577 +0x50
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc002ec6708, {0x5b454c955f18, 0xc002c6ea80}, 0xc000e2c050, 0xc0001f7bd0, 0x0)
	github.com/tidwall/tile38/internal/server/pubsub.go:392 +0x459
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc002ec6708, {0x5b454c93fc00, 0x5b454d285b60}, {0x5b454c955f18, 0xc002c6ea80}, 0xc000e2c050, 0xc0001f7bd0, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:105 +0x913
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 1626
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15df

goroutine 20584 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc005cbca10, 0x2)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc000012d08?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveAOF(0xc005815c08, 0x0, {0x5b454c955f18, 0xc006574480}, 0xc006f04050, 0x4?)
	github.com/tidwall/tile38/internal/server/aof.go:527 +0x4fb
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc005815c08, {0x5b454c93fc60, 0x5b454c31a940}, {0x5b454c955f18, 0xc006574480}, 0xc006f04050, 0xc006571090, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:103 +0xa05
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 20583
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15df

goroutine 1299 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002f960c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1294
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 2081 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00459c0c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 2006
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1654 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00477d8c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1559
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1733 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00280dc80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1594
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 3436 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002045b00)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 3430
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1775 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001956a80)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1722
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1322 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc001f8f8c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1367
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1601 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884f64f00, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00455e900?, 0x900000036?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00455e900)
	internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00455e900)
	net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00456fa00)
	net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc00456fa00)
	net/tcpsock.go:380 +0x30
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc002ec6708)
	github.com/tidwall/tile38/internal/server/server.go:492 +0x31b
github.com/tidwall/tile38/internal/server.Serve.func2()
	github.com/tidwall/tile38/internal/server/server.go:307 +0x25
created by github.com/tidwall/tile38/internal/server.Serve in goroutine 1419
	github.com/tidwall/tile38/internal/server/server.go:305 +0x1133

goroutine 1475 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc002f978c0)
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:547 +0xaa
created by github.com/tidwall/buntdb.Open in goroutine 1416
	github.com/tidwall/buntdb@v1.2.9/buntdb.go:169 +0x29f

goroutine 1625 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x79a884cbda70, 0x72)
	runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000481a00?, 0xc000daa000?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000481a00, {0xc000daa000, 0xffff, 0xffff})
	internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc000481a00, {0xc000daa000?, 0x4347978f?, 0xc002ec68b8?})
	net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005b2ab8, {0xc000daa000?, 0xc001bf8c80?, 0x0?})
	net/net.go:194 +0x45
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x5b454c955ab0, 0xc0005b2ab8})
	github.com/tidwall/tile38/internal/server/server.go:555 +0x4fa
created by github.com/tidwall/tile38/internal/server.(*Server).netServe in goroutine 1601
	github.com/tidwall/tile38/internal/server/server.go:502 +0x4d2

goroutine 1994 [sync.Cond.Wait, 9 minutes]:
sync.runtime_notifyListWait(0xc0001eff50, 0x7)
	runtime/sema.go:597 +0x15a
sync.(*Cond).Wait(0xc00444d480?)
	sync/cond.go:71 +0x85
github.com/tidwall/tile38/internal/server.(*Server).liveAOF(0xc00095e388, 0x0, {0x5b454c955f18, 0xc005543400}, 0xc005802050, 0x4?)
	github.com/tidwall/tile38/internal/server/aof.go:527 +0x4fb
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc00095e388, {0x5b454c93fc60, 0x5b454c31a940}, {0x5b454c955f18, 0xc005543400}, 0xc005802050, 0xc005275b30, 0x0)
	github.com/tidwall/tile38/internal/server/live.go:103 +0xa05
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
	github.com/tidwall/tile38/internal/server/server.go:620 +0x135
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2 in goroutine 1915
	github.com/tidwall/tile38/internal/server/server.go:618 +0x15df
FAIL	github.com/tidwall/tile38/tests	600.029s
?   	github.com/tidwall/tile38/tests/107	[no test files]
?   	github.com/tidwall/tile38/tests/616	[no test files]
FAIL

[tidwall]

I found what was causing the issue and published a fix.

I initially thought it was a bad test, but it's actually in the Tile38 message sending logic for pubsub channels.

The problem was due to a wait-condition loop that would consume the latest messages, unlock the waiter, send the messages, lock the waiter, and wait for new messages.

The logic was changed to having the wait-condition loop check for new messages, consume the messages, unlock the waiter, send the messages, lock the waiter, check for more new messages, if so then continue consuming/sending until there are no more messages, and only then wait.

Thanks for reporting !

[txtsd]

Thank you! I updated the AUR packages.