In our effort to track down information on the Web, we employ an arsenal of tools. We may browse the handpicked sites of a directory like Yahoo! or the results of a search engine like Google. Some cast an even wider net by using metasearch tools like Search.com or ixquick. And yet we still don't always find what we need.
This material has been referred to as the Hidden Internet, the Deep Web or the Invisible Web. The Invisible Web is comprised of information stored in databases, according to Chris Sherman, Webmaster of About.com's Web Search. Spiders and robots cannot enter these databases.
"It's as if they've run smack into the entrance of a massive library with securely bolted doors," Sherman said. "Spiders can record the library's address, but can tell you nothing about the books, magazines or other documents."
What else makes up the Invisible Web?
Non-HTML files (PDF files, etc.)
Webbed databases
Sites requiring registration or login
Archives (newspapers and magazines, etc.)
Dynamically created Web pages
Interactive tools (calculators, etc.)
Thankfully, gateways have been developed to mine these hidden troves. Think of them as database or search tool finders. Here are a few of the best:
Direct Search
Maintained by librarian Gary Price, this site provides direct access to the search interfaces of Invisible Web resources.
Invisible Web
A Yahoo!-like directory of more than 10,000 specialized databases (business, legal, travel, etc.). Includes free- and fee-based resources.
Internet Oracle
Search forms and direct links to hundreds of search engines, from general purpose directories to niche topic indexes. Categories of databases include Reference and Research, Medical, Maps, Legal and Government.
Related Articles
Researchers share their secrets to searching the invisible Web at the following sites:
"Invisible Web" Revealed
Danny Sullivan outlines new tools for searching the invisible Web.
Exposing the Invisible Web
Reference librarian Diana Botluk explores information stored in databases and how to access it.
Internet Tips: Searching the Invisible Web
Amigos Library Services provides tips on searching hidden data.
The Invisible Web
Chris Sherman outlines maps of the invisible Web.
Links Provided By Libraryspot