ValueError: could not convert string to float: b'undefined'

[lukastribus]

Version number

$ pipenv run pip show cfscrape
Name: cfscrape
Version: 2.0.3
Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.
Home-page: https://github.com/Anorov/cloudflare-scrape
Author: Anorov
Author-email: anorov.vorona@gmail.com
License: UNKNOWN
Location: /home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages
Requires: requests
Required-by:
$

Code snippet experiencing the issue

import cfscrape
cfscrape.create_scraper().get('https://pro-src.com/')

Complete exception and traceback

(If the problem doesn't involve an exception being raised, leave this blank)

>>> cfscrape.create_scraper().get('https://pro-src.com/')
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 267, in solve_challenge
    float(result)
ValueError: could not convert string to float: b'undefined'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 108, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 154, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/home/user/.local/share/virtualenvs/project1-4cV4hs0c/lib/python3.5/site-packages/cfscrape/__init__.py", line 270, in solve_challenge
    "Cloudflare IUAM challenge returned unexpected answer. %s" % BUG_REPORT
ValueError: Cloudflare IUAM challenge returned unexpected answer. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
>>>

URL of the Cloudflare-protected page

https://pro-src.com/

URL of Pastebin/Gist with HTML source of protected page

https://pastebin.com/K0LX8KDT

I haven't really taken a good look at this yet but I did add the HTML source as a test fixture. It passes on python 2 and 3 so it's not immediately clear to me as to what is causing the problem.

git clone --single-branch --branch issue_241 https://github.com/pro-src/cloudflare-scrape issue_241
cd issue_241
pip install pytest responses sure
pytest tests

`

[lukastribus]

Well the HTML is not necessarily what cfscrape sees, since cfscrape does not output the HTML code. The pastebin merely contains the output of a parallel curl https://pro-src.com/ which may or may not be what cfscrape sees.

Are you saying you cannot reproduce this? cfscrape 2.0.3 worked fine for me until at least May 12th, 18:10 UTC and started to fail at 19:10 UTC, so I assume cloudflare changed something and cfscrape is broken for everyone?

@lukastribus I'm able to reproduce this error when using a very old version of Node.js v4. If Cloudflare updates usually they'll be a ton of issues flooding into multiple repositories. The most recent version uses more robust methods so maybe not...

• (passing) Node.js v10.15.2
• (passing) Node.js v6.17.1 <-- No longer supported since a couple of weeks ago
• (passing) Node.js v4.9.1 <-- No longer supported since 2018-04-30
• (failing) Node.js v4.0.0 <-- Seriously outdated missing crucial security updates

This works properly on any security patched version of v4 and all versions in between v4 and the most recent Node.js v12.2.0.
The reason this fails is because your version of Node.js is missing the Buffer.from related security update. The error isn't being shown so I'll address that in a PR soon. Logging this particular error raises a security concern.

In conclusion, you should upgrade your version of Node.js. Please read https://github.com/Anorov/cloudflare-scrape#nodejs-dependency

[lukastribus]

Thanks for the analysis.

This is Ubuntu 16.04 (xenial) - which is supported until 2021 and the latest nodejs package 4.2.6~dfsg-1ubuntu4.2 from the official repository. In this case, although the release is 4.2.6, Ubuntu is supposed to backport critical security fixes until the entire OS is EOS. Can you give me a pointer to the security update you are talking about, so I can file a bug against the Ubuntu package?

You're welcome. The security vulnerability was discovered by @feross and other webtorrent contributors. He created this package until the Node.js team had time to backport the proposed fix, also noted here. He raised the Node.js issue here: nodejs/node#4660

CVE-2018-7166 <- (Might just be related)
Search 'em out: https://www.cvedetails.com/vulnerability-list/vendor_id-12113/Nodejs.html

@lukastribus There is no support for v4, v6, v7 or earlier versions even if the Node.js team does receive reports about security vulernabilities, nothing will be done (No support).

Information on all security updates: https://nodejs.org/en/blog/vulnerability/

I would upgrade to bionic(18.04LTS): https://packages.ubuntu.com/search?keywords=nodejs
Or follow these instructions: https://github.com/nodesource/distributions#installation-instructions

But I don't like Ubuntu so... I use Kali.

[lukastribus]

Ok, that works, thanks.

Still don't really understand how this could brake cfscrape 2.0.3 from one moment to the other, only by changes on the cloudflare side.

As we suggest in README that Debian and Ubuntu users can you just apt-get install nodejs, which since yesterday is only true for Debian >= 9/Stretch and Ubuntu >= 18.04/Bionic, we should probably update README.

Can I assume that a PR updating README in that regard would be welcome?

@lukastribus Absolutely, please send one! Also, if we could pin point the exact minimum Node.js version and add that to the README with a note recommending people not to use unsupported Node.js versions for security reasons. Or simply noting v4.9.1 as being the minimum version that is known to work with cfscrape. Just suggestions. 😃

Still don't really understand how this could brake cfscrape 2.0.3 from one moment to the other, only by changes on the cloudflare side.

The IUAM JS challenge can vary from domain to domain, some domains might be locked to an older challenge. The challenges can change on a per request basis as well as the number of challenges required to solve and how often you receive those challenges with/without the cookie. Usually the changes are negligible on a per request basis but in some cases...

This library aims to maintain compatibility with all challenge versions that are currently in play.

[lukastribus]

This has been backported to 4.5, and first released in 4.5.0.

Confirmed by this PR here:
https://github.com/stripe/stripe-node/pull/365/files

So at least node 4.5 is required. Which explains why Ubuntu Xenial (4.2.6) is broken but Debian Stretch (4.8.2) works.

Will send a PR for this later.