Many have the naive expectation that they can find anything about the World Wide Web using search engines like Google or Yahoo or Ask.com or Bing. The truth is that all of these search engines have indexes of 10% of all web. The remaining 90% is called “Invisible Web", the "Cloaked Web"Or"Deep Web. "
This means that there are huge volumes of data that are available to the public, but they are hidden by the search engines they all know.
It may be difficult for you to understand that billions of websites cannot appear in Google results. But they exist. Robot 'spiders' that scan and archive the world wide web have limited capabilities.
To understand better, let's start with some figures about the size of the services offered by Google.com, Yahoo.com, Cyberatlas, and MIT. The statistics are from the summer of 2013:
Google.com has 40 billions of public webpages in its archives. 100 + Two of these are static websites and publicly available. These pages can easily be found by Google and other search engines.
11 + two-static pages are hidden by the public after they have declared that they contain private content or are on the intranet. These are corporate pages that are open only to employees of these companies.
450 + billions of pages have databases that are completely invisible to Google. For example, government databases with tax information,
Google is considered to have the best search database today. The company's spiders list millions of web pages every week.
So if Google has saved only one 8-10% of the World Wide Web and the other search engines have even smaller databases, then where does the remaining 92% of the content on the Internet hide?
The "Invisible Web" (or "Deep Web" or "Cloaked Web") is the content that does not appear in search engines.
More specifically: The Invisible Web consists of 220+ billion web pages that are not stored as static web pages. The Invisible Web consists of on-demand pages and databases. That is, pages that exist only as reports of changing data. As of August 2007, robotic spiders had not advanced enough to read these private databases. Only people can access it, and only if they have the knowledge.
Technical terminology:
"Spider": An artificial intelligence program, or robot, that has been sent to read millions of static web pages on the public Internet. The information collected by Spiders is stored in databases used by search engines.
Database-Driven Web Content: websites that exist only temporarily, and are only created when readers request answers from a large database. These temporary webpages are dynamic, and usually cannot be bookmarked. They usually have extremely large URLs.
The Invisible Web contains Dynamic Web Pages. This means that a database creates a temporary page to answer your question! Good one huh;
How can I use the Invisible Web?
There are too many who ask exactly the same. Let's look at some valuable databases below.
Humanities
Voice of the Shuttle: Started 1994, and is one of the oldest and largest human data bases on the Web.
Special foundations of the US government
University of Michigan Government Documents Center: You will find a great deal of data, surveys, statistics, and many of the high levels of the US government. The databases offered include Arts, Health Sciences, Social Sciences and International Studies.
USA.gov: A portal station for many agencies of the United States government. Includes government positions, services, and information on finding grants, loans, and financial aid.
Health and Science
PsycNET: Use the American Psychological Association database to find citations and entire journals for various issues psychology.
Scirus: A search tool that works exclusively for scientific information. The amazing search tool has hundreds of millions of scientific, and academic papers to help researchers from around the world.
Healthfinder: It contains information from over a thousand different health databases on the internet.
RXList: If you are looking for reliable information about medicines, then this database is for you.
Mega-Portal
The University of California, Riverside maintains it InfoMine, an incredible source of knowledge that at the last count contained over 100.000 links and access to hundreds, if not thousands, of databases.
There are generally too many websites set up to bring data from the Invisible Web. CompletePlanet.com is one of them. Contains "over 70.000 databases."
Most of the information about the invisible Web is kept by academic institutions. There are "academic gates" that can help you find this information. To find almost every educational resource on the web, simply type the following term into your favorite search engine:
site: .edu "topic I'm looking for"
Your search will return results for edu sites only. If you want to search for something from a specific university use the university URL in your search:
site: www.p university.com "topic I am looking for"
This is just the tip of the iceberg. All that we have mentioned in this article is just beginning to touch the huge resources available on the Invisible Web. As time passes, the Invisible Web becomes larger.
