`Connection` does not reconnect automatically

[jahewson]

When I restart my web socket server, the Connection on the client emits a state event with the newState set to "disconnected", which according to the documentation means:

'disconnected' Connection is closed, but it will reconnect automatically

However, the connection does not reconnect. Given that the socket which was passed to the Connection's constructor is now closed, I don't understand how reconnection would be possible. Don't we need a new WebSocket? What's going on here?

[DetweilerRyan]

The inline documentation you are referring to is incorrect. I recommend using one of the following modules for re-connectable websockets:

websocket.js
reconnectingwebsocket

[jahewson]

Ok, that's good to hear. I had thought that maybe it was just me. Might want to update the README which says:

• Reconnection of document and query subscriptions
• Offline change syncing upon reconnection

[atsepkov]

I too encountered reconnection problems, everything works fine locally, but Heroku seems to kill the socket if no event occurs within a minute. I have now added a ping-pong to help, and replaced my WebSocket with a ReconnectingWebSocket, but running into a weird issue. The socket does indeed reconnect, but the events no longer sync between clients after the reconnection.

Oddly, it seems like they still make it to the server, just not the other client - almost as if the two clients started thinking they're in different rooms until one of the pages is reloaded, or as if the subscription to remote changes broke after the disconnection. Do I need to rerun subscribe events after the reconnection? Do I need to refetch the docs? Do I need to recreate the ShareDB handle on every reconnection? On the main page in the list of features it says that ShareDB handles reconnection, what exactly does that mean and what should we handle ourselves?

Thanks

[curran]

This is addressed in PR #138 .

I've had quite good luck using https://github.com/pladaria/reconnecting-websocket , dropping it in in place of raw WebSocket.

FWIW, initially I tried https://github.com/joewalnes/reconnecting-websocket , but decided not to use it because it depends on window (breaks in a Node environment), and is no longer maintained.

[pedrosanta]

Yep, I've also come across this issue of Connection not reconnecting automatically.

On previous versions of ShareDB/ShareJS the communication was established through browserchannel, but browserchannel comes from a time where WebSockets weren't widely supported and so it resorted to this 'long-polling' protocol to establish the connection. Now, also as stated in its README, we should strongly consider WebSockets as an approach to this.

I also note that ShareDB pretty much requires a WebSocket, or another API-compatible way of transport, that assures in-order-message delivery, it seems.

But I do miss the resilience of browserchannel between connects, disconnects and network issues.

I've been setting up a small (WIP) experiment at pedrosanta/quill-sharedb-cursors and I've resorted to reconnectingwebsocket like @DetweilerRyan suggested, along with some ping/keep-alive mechanism on server-side to help maintain connections. So far seems to be working quite fine.

I recognize that @mbalex99 on #138 updates the documentation, but I don't know if this really shouldn't be taken care of on ShareDB Connection module? I think I would like to have this 'resilience'/reconnection out-of-the-box on ShareDB.

Thoughts?

[pedrosanta]

@atsepkov That's a bit strange. I've been doing a few tests also on Heroku with pedrosanta/quill-sharedb-cursors and, like, the events resume after a reconnect, and even syncs the edits made while offline on reconnect. 😕 LMK if it helps or if you can make it work too somehow.

[DetweilerRyan]

In production we at Retrium no longer use a reconnectable websocket. The logic around how, when and why we attempt to reconnect to our servers is complex and none of the existing reconnecting websocket npm modules provide the flexibility we required. Reconnection logic in a production grade application should take into account a number of things including if the bowser has a network connection, if the browser tab is active and visible, and finally should use an exponentially increasing backoff interval for reconnection attempts. This logic is difficult to get right and the details depend on the needs of the application and therefore ShareDB should not contain reconnection logic.

Instead, we at retrium use the .bindToSocket method on the ShareDB.Connection instance to pass in new WebSockets to ShareDB.

First we instantiate a ShareDB.Connection with a mock WebSocket that is already 'CLOSED'.

const CLOSED = 3;
const mock_web_socket = {
  close: () => {}, 
  readyState: CLOSED
};
const sharedb_connection = new sharedb.Connection(mock_web_socket);

Then each time we instantiate a new WebSocket we pass it into ShareDB:

WebSocketConnector.on('connection', (socket) => {
  sharedb_connection.bindToSocket( socket );
});

Where WebSocketConnector is a proprietary abstraction that encapsulates all of the complexities around reconnecting to our production servers. Note: ShareDb.Connection.bindToSocket must never be called if the current socket referenced by the connection instance has not closed yet.

[pedrosanta]

Nice insight @DetweilerRyan. 👍

So, the way you see it is assume really bare WebSocket interface/API-compatible transport layer, and leave/assume connection/reconnection policy and concern to 'downstream' usage?

I think that can work out too. It would just need IMHO to:

1. Make the documentation clear ( Reconnecting Websocket Example and Updated Readme #138 already does that, although we need to remove another line from README and probably update a few Connection comments, etc);
2. Probably, provide additional documentation on how to achieve simple reconnection capabilities and more info on that topic (probably on a Wiki page or something);

How does this sound?

[pedrosanta]

Also, @DetweilerRyan, ever thought of open-sourcing a version of that WebSocketConnector? 🙂

[curran]

I'm curious, is anyone still using Browserchannel? I'm almost tempted to, because an additional benefit would be the ability to use Cookies set by Express on the req objects.

Currently I need to go through what seems like a bunch of unnecessary hoops in order to access the Express session in the ShareDB middleware that handles WebSocket connections.

In case it might be useful for anyone, and in case there is a better way to do this, I'd like to share some snippets for the approach I'm using now for accessing sessions on WebSockets in ShareDB:

import express from 'express'  
import http from 'http'
import expressSession from 'express-session'
import cookie from 'cookie'    
import cookieParser from 'cookie-parser'
import ShareDB from 'sharedb'  
import { Server as WebSocketServer } from 'ws'
import JSONStream from 'websocket-json-stream'

const app = express()
const server = http.createServer(app)
const wss = new WebSocketServer({server})
const db = sharedbMongo(MONGO_URL)
const sharedb = ShareDB({db})

import connectRedis from 'connect-redis'

import {
  SESSION_SECRET as secret,
  REDIS_HOST as host,
  REDIS_PORT as port
} from './config'

const RedisStore = connectRedis(expressSession)
const store = new RedisStore({ host, port })

app.use(expressSession({ ... }))

// Gets the current session from a WebSocket connection.
// Draws from http://stackoverflow.com/questions/36842159/node-js-ws-and-express-session-how-to-get-session-object-from-ws-upgradereq
export const getSession = (ws, callback) => {

  // If there's no cookie, there's no session, so do nothing.
  if(!ws.upgradeReq.headers.cookie) return callback() 

  // If there's a cookie, get the session id from it.
  const cookies = cookie.parse(ws.upgradeReq.headers.cookie)
  const sessionId = cookieParser.signedCookie(cookies['connect.sid'], secret)

  // Get the session from the store and pass it to the callback.
  store.get(sessionId, callback)
}

wss.on('connection', (ws) => {
  getSession(ws, (err, session) => {
    if(err) return console.log(err)
    ws.upgradeReq.session = session
    sharedb.listen(new JSONStream(ws), ws.upgradeReq)
  })
})

// Expose the session from initial connection as agent.session.
sharedb.use('connect', (request, done) => {
  request.agent.session = request.req.session
  done()
})

sharedb.use('apply', (request, done) => {

  // Unpack the ShareDB request object.
  const {
    op,
    agent: { session },
    snapshot
  } = request

  // Get the id of the currently logged in user from the session.
  const userId = get(session, 'passport.user.id')

  .. your code here ..
}

The above should work well with reconnecting-websockets, but with Browserchannel you can just access req.session and the above code is not necessary.

[DetweilerRyan]

@pedrosanta Agreed, reconnection at the network layer is and probably should remain out-of-scope for the ShareDB project.

Make the documentation clear ( #138 already does that, although we need to remove another line from README and probably update a few Connection comments, etc);

I'm not sure #138 really solves the confusion around the documentation regarding reconnection. The statement in the documentation, "Reconnection of document and query subscriptions", is not incorrect but is just misunderstood. While ShareDB does not initiate reconnection attempts at the network layer it does handle resubscribing to existing Document and Query subscriptions after a disconnect and subsequent reconnect occurs on the network layer. This is important to document because it means that that when using ShareDB and a reconnection occurs at the network layer you do not need to manually resubscribe to Documents and Querys that were previously subscribed to. The internal logic to reconnect to each Document and Query is complex and can be see here in the Connection.prototype._setState method.

Instead, documentation stating that ShareDB does not attempt to open a closed socket but will internally handle resubscribing to all Documents and Queries when the socket open event fires and the socket's readState changes to OPEN may be more appropriate.

Probably, provide additional documentation on how to achieve simple reconnection capabilities and more info on that topic (probably on a Wiki page or something);

The example in #138 using the reconnecting websocket is sufficient to get started with ShareDB. However, my earlier caution regarding websockets in a realtime production grade web application still stand.

Also, @DetweilerRyan, ever thought of open-sourcing a version of that WebSocketConnector? 🙂

Yes; however, our current implementation of WebSocketConnector has some unresolved architectural issues and dependencies that are not suitable for an open source project.

[DetweilerRyan]

@curran At retrium we have been using BrowserChannel in production for two years. For the first year we only used BrowserChannel but then switched over to WebSockets believing that WebSockets had enough browser support that It wouldn't affect many or any of our customers. Unfortunately, despite the almost universal support for WebSockets corporate firewalls and proxies will sometimes block WebSockets. After realizing this we quickly deployed a fallback mechanism from WebSockets to BrowserChannel. We have had no major problems with BrowserChannel and there remains a clear need for a alternative to WebSockets to fallback to despite the near universal support in browsers. That being said, I know little about why or how corporate firewalls and proxies sometimes block WebSockets and any insight here would be appreciated.

Currently I need to go through what seems like a bunch of unnecessary hoops in order to access the Express session in the ShareDB middleware that handles WebSocket connections.

In case it might be useful for anyone, and in case there is a better way to do this, I'd like to share some snippets for the approach I'm using now for accessing sessions on WebSockets in ShareDB:

Sometimes bridging technologies together gets a bit messy, nevertheless your approach for accessing sessions on WebSockets in ShareDB middleware looks pretty solid. Thanks for sharing!

[pedrosanta]

@curran:

I'm curious, is anyone still using Browserchannel?

We're still using Browserchannel, but we might move to WebSockets in the near future. And yea, with WebSockets, handling session can be nightmarish. 😕 Nice snippet, thanks!

[pedrosanta]

@DetweilerRyan:

I'm not sure #138 really solves the confusion around the documentation regarding reconnection. The statement in the documentation, "Reconnection of document and query subscriptions", is not incorrect but is just misunderstood. While ShareDB does not initiate reconnection attempts at the network layer it does handle resubscribing to existing Document and Query subscriptions after a disconnect and subsequent reconnect occurs on the network layer. This is important to document because it means that that when using ShareDB and a reconnection occurs at the network layer you do not need to manually resubscribe to Documents and Querys that were previously subscribed to. The internal logic to reconnect to each Document and Query is complex and can be see here in the Connection.prototype._setState method.

Yap, got it. You're absolutely right. Just network/transport level.

Instead, documentation stating that ShareDB does not attempt to open a closed socket but will internally handle resubscribing to all Documents and Queries when the socket open event fires and the socket's readState changes to OPEN may be more appropriate.

Yap, agreed. This documentation could use some clarification. 👍

Yes; however, our current implementation of WebSocketConnector has some unresolved architectural issues and dependencies that are not suitable for an open source project.

👍 Got it.

Yikes, nice cautionary tale regarding firewalls and proxies @DetweilerRyan... 😟 I think I'll be re-assessing the update from Browserchannel to WebSockets... didn't really thought of those kind of issues.