omprompt-test

Crawls a website looking for a token with a breadth-first algorithm. Does not cross domains.

Install

Run the following commands:

• npm i
• npm link

Running

• npm start - this will by default crawl www.bbc.co.uk looking for the term Renault on all crawled pages.

OR

• npx babel-node crawl.js {startPage} {searchTerm} {parallelism} - searches a domain from startPage, with a custom search term searchTerm. parallelism defines the maximum number of concurrent requests the crawler can make to the site at any given time.

Issues

These are (some) of the things I've not yet had time to do within the time budget:

• Static user agent
• Does not take robots.txt into account
• Application does not refill HTTP request slots immediately when they resolve - could be more efficient
• Needs Babel compilation
• Ideally needs some configurable limitations - e.g. max depth of URLs to crawl, max time to crawl for, etc
• CrawlerQueue could probably do with decomposition
• CrawlerQueue needs unit tests!