Skip to content

Conversation

@dubee
Copy link
Member

@dubee dubee commented Jun 4, 2019

An exception thrown when the Doctor is restarting a consumer will result in the Doctor completely shutting down due to the exception not being caught. As a result, exceptions in the Doctor's run loop need to be handled properly.

The exception causing the Doctor to crash occurs when the Doctor restarts a consumer and an attempt to create a database connection fails on consumer init. Since a presistant database connection is not needed by the consumer itself, only create the database connection when it is needed to automatically disable a trigger.

Below is a stacktrace showing the described exception occuring:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/KafkaFeedProvider/thedoctor.py", line 72, in run
    consumer.restart()
  File "/KafkaFeedProvider/consumer.py", line 119, in restart
    self.process = ConsumerProcess(self.trigger, self.params, self.sharedDictionary)
  File "/KafkaFeedProvider/consumer.py", line 182, in __init__
    self.database = Database()
  File "/KafkaFeedProvider/database.py", line 48, in __init__
    self.client.connect()
  File "/usr/local/lib/python2.7/site-packages/cloudant/client.py", line 117, in connect
    self.session_login(self._user, self._auth_token)
  File "/usr/local/lib/python2.7/site-packages/cloudant/client.py", line 173, in session_login
    headers={'Content-Type': 'application/x-www-form-urlencoded'}
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/cloudant/_common_util.py", line 310, in request
    method, url, timeout=self._timeout, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='OMITTED', port=443): Max retries exceeded with url: /_session (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f84d75789d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

@dubee dubee requested a review from jasonpet June 4, 2019 17:06
@dubee dubee removed the wip label Jun 4, 2019
@dubee dubee changed the title Catch Doctor exceptions Catch Doctor exceptions and do not persist consumer database connections Jun 4, 2019
Copy link

@jasonpet jasonpet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jasonpet jasonpet merged commit 02a88dd into apache:master Jun 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants