Scanner abruptly exits with "error: can't start new thread" after certain extent

[jovyn]

Here is an Error I keep getting for larger websites , and the scanner abruptly shuts down after scpidering the URL's to a certain extent.

_Exception in thread Thread-2467:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\site-packages\nyawc\CrawlerThread.py", line 116, in run
    self.__callback(self.__queue_item, new_requests, failed)
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 252, in __request_finish
    self.__spawn_new_requests()
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 105, in __spawn_new_requests
    if self.__spawn_new_request():
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 125, in __spawn_new_request
    self.__request_start(first_in_line)
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 217, in __request_start
    thread.start()
  File "C:\Python27\lib\threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread_

[tijme]

@jovyn Thank you for reporting this issue. I think this is due to a design issue in the crawler.

Currently, the crawler starts new threads in the thread that finished. I'm going to change it so that new threads will be started from the main thread.

[tijme]

@jovyn Do you know after how much requests this happens?

[jovyn]

Referring to my Burp logs (I chained the scanner via Burp) I can see about 8492 requests.
This may not be the exact number everytime this happens. Maeanwhile let me run it against some other site and share the feedback with you.

[tijme]

@jovyn I think I fixed this issue on the develop branch, however, I can't test it since I could not reproduce the issue after ~8500 requests. Could you test if it works for you?

[jovyn]

Hey @tijme I installed the new version from the develop branch and ran the scanner, but I am getting the below errors for larger scanning requests (about 7000 requests) :

_[ERROR]
error : Memory allocation failed : growing buffer
error : Memory allocation failed : growing buffer_

followed by \:

_Traceback (most recent call last):
  File ".\extended.develop.py", line 177, in <module>
    main()
  File ".\extended.develop.py", line 84, in main
    driver.start()
  File "C:\Python27\lib\site-packages\acstis\Driver.py", line 136, in start
    crawler.start_with(startpoint)
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 95, in start_with
    self.__crawler_start()
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 166, in __crawler_start
    self.__spawn_new_requests()
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 111, in __spawn_new_requests
    if self.__spawn_new_request():
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 131, in __spawn_new_request
    self.__request_start(first_in_line)
  File "C:\Python27\lib\site-packages\nyawc\Crawler.py", line 230, in __request_start
    thread.start()
  File "C:\Python27\lib\threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread_

[jovyn]

I did not explicitly specify the number of threads (-mt option) and I the default number of threads the scanner takes is 8.
I think the scanner is trying to create too many threads and is going beyond the default value of 8.

Reference : https://johnsofteng.wordpress.com/2010/03/05/python-thread-error-cant-start-new-thread/

[jovyn]

Tried specifying the number of threads using the -mt option. I still get the same error.

[tijme]

@jovyn Did you update the dependencies? If you run pip freeze the nyawc dependency must be on version 1.7.9.

[jovyn]

Yes I did . The nyawc dependency is on 1.7.9

PS C:\Angular-CSTI-Scanner\Develop_Branch> pip freeze

acstis==3.0.2
alabaster==0.7.10
Babel==2.5.1
beautifulsoup4==4.6.0
certifi==2017.7.27.1
chardet==3.0.4
colorama==0.3.9
colorlog==2.10.0
docutils==0.14
idna==2.5
imagesize==0.7.1
Jinja2==2.9.6
lxml==4.0.0
MarkupSafe==1.0
nyawc==1.7.9
pockets==0.5.1
Pygments==2.2.0
pytz==2017.2
requests==2.18.1
requests-toolbelt==0.8.0
selenium==3.4.3
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.5.5
sphinx-better-theme==0.13
sphinxcontrib-napoleon==0.6.1
urllib3==1.21.1