Skip to content

Fix HTML crawling #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 4, 2020
Merged

Fix HTML crawling #20

merged 2 commits into from
May 4, 2020

Conversation

ehmicky
Copy link
Contributor

@ehmicky ehmicky commented May 4, 2020

Fixes #19.

HTML crawling fails when one of the paths passed as checkPaths does not exist.

This PR also makes the crawling happen in parallel for all checkPaths (instead of serially) and replace custom crawling by the readdirp library.

@ehmicky ehmicky added the bug Something isn't working label May 4, 2020
@ehmicky ehmicky requested a review from swyxio May 4, 2020 14:07
@ehmicky ehmicky self-assigned this May 4, 2020
Copy link
Contributor

@swyxio swyxio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. should we warn if the path doesnt exist at all? likely to be a user error.

@ehmicky ehmicky mentioned this pull request May 4, 2020
23 tasks
@ehmicky
Copy link
Contributor Author

ehmicky commented May 4, 2020

You're right: fixed in cbb9597.

@ehmicky ehmicky requested a review from swyxio May 4, 2020 15:13
}

if (!(await isFile(fileAndDirPath))) {
build.failBuild(`Input "checkPaths" does not exist: ${fileAndDirPath}`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait.. why fail build? should we just warn instead?

also i think the wording can be better.. maybe Warning: Folder ${fileAndDirPath} was provided in "checkPaths", but does not exist - it either indicates something went wrong with your build, or you can simply delete this folder from your "checkPaths" in netlify.toml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed!

@ehmicky ehmicky force-pushed the fix/html-crawling branch from cbb9597 to 23c42f2 Compare May 4, 2020 17:21
@ehmicky ehmicky force-pushed the fix/html-crawling branch from 23c42f2 to 4bd568e Compare May 4, 2020 17:22
@ehmicky ehmicky requested a review from swyxio May 4, 2020 17:22
@swyxio swyxio merged commit 3fc332f into master May 4, 2020
@swyxio swyxio deleted the fix/html-crawling branch May 4, 2020 21:37
@swyxio
Copy link
Contributor

swyxio commented May 4, 2020

v 0.0.9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error: ENOENT: no such file or directory, stat 'build/public'
2 participants