Protocol plugin which supports retrieving documents via the HTTP and
HTTPS protocols, optionally with Basic, Digest and NTLM authentication
schemes for web server as well as proxy server.
A url filter plugin that validates given urls.
This plugin runs a series of tests for the given url to make sure that given
url is valid and 'fetchable'.
Note: This plugin should only be used for web-related protocols such
as http, https and ftp.
org.apache.nutch.util.domain
This package contains classes for domain analysis.
for information please refer to following urls :
http://en.wikipedia.org/wiki/DNS
http://en.wikipedia.org/wiki/Top-level_domain
http://wiki.mozilla.org/TLD_List
http://publicsuffix.org/