Fixes OutOfMemory error for large sites#31
Open
pokey909 wants to merge 4 commits intochriskite:masterfrom
Open
Fixes OutOfMemory error for large sites#31pokey909 wants to merge 4 commits intochriskite:masterfrom
pokey909 wants to merge 4 commits intochriskite:masterfrom
Conversation
pokey909
commented
Sep 4, 2011
pokey909
commented
- Added support for external queues via :large_scale_crawl option. (Requires R/W permission for working dir)
- Improved Thread handling. All threads now properly start working on the crawl
added 4 commits
August 31, 2011 21:47
Occurs when crawling larges sites. Issue: link_queue grows faster than threads consume links. Fix: Wait until threads consumed enough links, then continue adding more to the queue.
- OutOfMemory caused by large link/page queues. Added thread safe ExtQueue class which swaps to disk when too much memory is consumed - Improved threading. Most worker threads kept idling when launched simultaneously Signed-off-by: Alexander Lenhardt <alenhard@techfak.uni-bielefeld.de>
External queue storage can be activated via new option :large_scale_crawl Signed-off-by: Alexander Lenhardt <alenhard@techfak.uni-bielefeld.de>
|
Do you have a 0.7.1 version for this pull request ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.