error : thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()`

[hellower]

First of all, thank you very much for a great open source product

I am currently using pgbouncer, but I am testing various ways to see if it can be replaced with pgcat.
There was no error at all in haproxy+pgbouncer in the previous setting, and after replacing it with pgcat, pgcat continues to occur.
(I keep getting error messages, but pgcat seems to be working fine.)

I found out that the pgcat error below was caused by this part of the haproxy "...tnt-worker1:6432..." above after a long trace.

1. env
   os: Ubuntu 20.04.6 LTS
   pgcat 1.1.1 (port: 6432)
   postgresql 15.4 (port : 9081)
   HAProxy version 2.8.1-a90123a 2023/07/03 - https://haproxy.org/

2. haproxy configuration

.......................................
listen cluster-rw-poolport
    mode    tcp
    option tcplog                 
    bind *:5000
    timeout client 30m    
    timeout connect 4s
    timeout server 30m
    timeout check 5s
    option httpchk OPTIONS /read-write   # /primary ex) curl -sv  http://tnt-worker1:8008/replica | jq .
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server tnt-worker1 tnt-worker1:6432 maxconn 100 check port 8008                              ############# HERE !!!! #############
    server tnt-worker2 tnt-worker2:6432 maxconn 100 check port 8008 # patroni rest-api 8008      ############# HERE !!!! #############
.......................................

4. PGCAT error

2023-08-23T23:06:10.443007Z  INFO pgcat: Config autoreloader: 86400000 ms    
2023-08-23T23:06:10.443044Z  INFO pgcat::stats: Events reporter started    
2023-08-23T23:06:10.443144Z  INFO pgcat: Waiting for clients    
2023-08-23T23:06:10.442995Z  INFO pgcat::pool: Creating a new server connection Address { id: 9, host: "localhost", port: 9081, shard: 0, database: "postgres", role: Replica, replica_number: 0, address_index: 1, username: "monexp", pool_name: "tx_monexp", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } }    
2023-08-23T23:06:10.443196Z  INFO pgcat::pool: Creating a new server connection Address { id: 8, host: "localhost", port: 9081, shard: 0, database: "postgres", role: Primary, replica_number: 0, address_index: 0, username: "monexp", pool_name: "tx_monexp", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } }    
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-08-23T23:06:11.365586Z  INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TNT-CLUSTER#12    
2023-08-23T23:06:12.421295Z  INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TARANTULA:0.0.0.0:9002/3.4.71@local:b    
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35

3. PGCAT full error message (export RUST_BACKTRACE=full)

.......
2023-08-23T23:06:38.795991Z  INFO pgcat::pool: Creating a new server connection Address { id: 9, host: "localhost", port: 9081, shard: 0, database: "postgres", role: Replica, replica_number: 0, address_index: 1, username: "monexp", pool_name: "tx_monexp", mirrors: [], stats: AddressStats { total: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, current: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 43, bytes_sent: 36, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages: AddressStatFields { xact_count: 0, query_count: 0, bytes_received: 0, bytes_sent: 0, xact_time: 0, query_time: 0, wait_time: 0, errors: 0 }, averages_updated: false } }    
2023-08-23T23:06:38.795465Z  INFO pgcat::stats: Events reporter started    
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
stack backtrace:
   0:     0x560485a9d861 - std::backtrace_rs::backtrace::libunwind::trace::h782cc21a5acaf6cb
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x560485a9d861 - std::backtrace_rs::backtrace::trace_unsynchronized::hc579eb24ab204515
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x560485a9d861 - std::sys_common::backtrace::_print_fmt::h7223525cfdbacda2
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x560485a9d861 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbd7d55b7108d2ab8
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x560485ac506f - core::fmt::rt::Argument::fmt::hb4f4a02b9bd9dd49
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/rt.rs:138:9
   5:     0x560485ac506f - core::fmt::write::h6d54cd7c9e155ec5
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/fmt/mod.rs:1094:21
   6:     0x560485a9aa41 - std::io::Write::write_fmt::h6a453a71c692f63b
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/io/mod.rs:1713:15
   7:     0x560485a9d675 - std::sys_common::backtrace::_print::h1cbaa8b42678f928
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x560485a9d675 - std::sys_common::backtrace::print::h4ddf81241a51b337
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x560485a9e997 - std::panicking::default_hook::{{closure}}::hff91f1f484ade5cd
  10:     0x560485a9e784 - std::panicking::default_hook::h21f14afd59f7aef9
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:288:9
  11:     0x560485a9ee4c - std::panicking::rust_panic_with_hook::h45f66047b14c555c
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:705:13
  12:     0x560485a9ed47 - std::panicking::begin_panic_handler::{{closure}}::h49d1a88ef0908eb4
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:597:13
  13:     0x560485a9dc96 - std::sys_common::backtrace::__rust_end_short_backtrace::hccebf9e57f8cc425
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys_common/backtrace.rs:151:18
  14:     0x560485a9ea92 - rust_begin_unwind
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
  15:     0x560485212d63 - core::panicking::panic_fmt::h54ec9d0e3180a83d
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
  16:     0x560485213203 - core::result::unwrap_failed::h1cd730365d65235f
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/result.rs:1651:5
  17:     0x560485439e9d - pgcat::client::client_entrypoint::{{closure}}::h152ca8414b8b9198
  18:     0x56048547dac9 - pgcat::main::{{closure}}::{{closure}}::h5e6389a6bf4b6203
  19:     0x560485472fd1 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h6a20aed4daa98b35
  20:     0x5604853541f6 - tokio::runtime::task::core::Core<T,S>::poll::h57ff81af1c9d7d81
  21:     0x56048522acae - tokio::runtime::task::harness::Harness<T,S>::poll::h744e0029abcf98ae
  22:     0x560485a4b47a - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h9b544f66c4ef6a0e
  23:     0x560485a4aa5e - tokio::runtime::scheduler::multi_thread::worker::Context::run::h19de324b23b290bd
  24:     0x560485a5bd27 - tokio::runtime::context::scoped::Scoped<T>::set::he66079070aa2bfdd
  25:     0x560485a4dd15 - tokio::runtime::context::runtime::enter_runtime::hbc778c7fd1b00890
  26:     0x560485a4a15c - tokio::runtime::scheduler::multi_thread::worker::run::h6f9959fd400d1aac
  27:     0x560485a58f9b - tokio::runtime::task::core::Core<T,S>::poll::he638b91f0b1532e1
  28:     0x560485a44f5f - tokio::runtime::task::harness::Harness<T,S>::poll::h70ce99c790196f83
  29:     0x560485a632f6 - tokio::runtime::blocking::pool::Inner::run::hbb956397631ab4b3
  30:     0x560485a46875 - std::sys_common::backtrace::__rust_begin_short_backtrace::h192e29676f1f0e03
  31:     0x560485a6fbb9 - core::ops::function::FnOnce::call_once{{vtable.shim}}::he30264600efd9a83
  32:     0x560485aa1875 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::he3e5dbdfabe0b668
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/alloc/src/boxed.rs:1985:9
  33:     0x560485aa1875 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h246f7c7964633611
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/alloc/src/boxed.rs:1985:9
  34:     0x560485aa1875 - std::sys::unix::thread::Thread::new::thread_start::hadf9e3501ff0df23
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/sys/unix/thread.rs:108:17
  35:     0x7fabc7566609 - start_thread
  36:     0x7fabc7336133 - clone
  37:                0x0 - <unknown>
2023-08-23T23:06:39.197253Z  INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TNT-CLUSTER#13    
2023-08-23T23:06:40.239704Z  INFO pgcat::server::cleanup: Server returned with session state altered, discarding state (SET: true, PREPARE: false) for application TARANTULA:0.0.0.0:9001/3.4.71@local:b    
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', /usr/local/src/pgcat-1.1.1/src/client.rs:120:35
stack backtrace:
   0:     0x560485a9d861 - std::backtrace_rs::backtrace::libunwind::trace::h782cc21a5acaf6cb
                               at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/

[hellower]

Can I change the 120th line in the same way? (1.1.2-dev src/client.rs source . .line 120)

109 /// Client entrypoint.
110 pub async fn client_entrypoint(
111 mut stream: TcpStream,
112 client_server_map: ClientServerMap,
113 shutdown: Receiver<()>,
114 drain: Sender,
115 admin_only: bool,
116 tls_certificate: Option,
117 log_client_connections: bool,
118 ) -> Result<(), Error> {
119 // Figure out if the client wants TLS or not.
120 let addr = stream.peer_addr().unwrap(); <======== here!

...

let addr = match stream.peer_addr() {
    Ok(addr) => addr,
    Err(err) => {
        return Err(Error::SocketError(format!(
            "Failed to get peer address: {:?}",
            err
        )));
    }
};

[hellower]

I've confirmed that it's a haporoxy problem and not a pgcat problem.

[levkk]

A PR to replace the unwrap with an error would still be welcome. An error is better than a panic.

[hellower]

Thanks to Zain Kabani!