Many have the naive expectation that they can find anything about the World Wide Web using search engines like Google or Yahoo or Ask.com or Bing. The truth is that all of these search engines have indexes of 10% of all web. The remaining 90% is called “Invisible Web", the "cloaked Web"Or"Deep Web. "
This means that there are huge volumes of data that are available to the public, but they are hidden by the search engines they all know.
It may be difficult for you to understand that billions of websites cannot appear in Google results. But they exist. Robot 'spiders' that scan and archive the world wide web have limited capabilities.
To understand better, let's start with some figures about the size of the services offered by Google.com, Yahoo.com, Cyberatlas, and MIT. The statistics are from the summer of 2013:
Google.com has 40 billion public web pages in the archives her. 100+ billion of these are static web pages and publicly available. These pages can easily be found by Google and other search engines.
11 + billion static pages are hidden from the public, having declared that they contain private content, or are on intranet. These are the company pages that are open only to the employees of the specific companies.
450 + billions of pages have databases that are completely invisible to Google. For example, government databases with tax information,
Google is considered to have the best search database today. The company's spiders list millions of web pages every week.
So, if Google has stored only 8-10% of the World Wide Web and the other search engines have even smaller databases, then where is the remaining 92% of the content on the web hiding?network;
The "Invisible Web" (or "Deep Web" or "Cloaked Web") is the content that does not appear in search engines.
More specifically: The Invisible Web consists of 220+ billion web pages that are not stored as static web pages. The Invisible Web consists of on-demand pages and databases. That is, pages that exist only as reports of changing data. As of August 2007, robotic spiders had not advanced enough to read these private databases. Only people can access it, and only if they have the knowledge.
"Spider": An artificial intelligence program, or robot, that has been sent to read millions of static web pages on the public Internet. The information collected by Spiders is stored in databases used by search engines.
Database-Driven Web Content: websites that exist only temporarily, and are only created when readers request answers from a large database. These temporary webpages are dynamic, and usually cannot be bookmarked. They usually have extremely large URLs.
The Invisible Web contains Dynamic Web Pages. This means that a database creates a temporary page to answer your question! Good one huh;
How can I use the Invisible Web?
There are too many who ask exactly the same. Let's look at some valuable databases below.
Voice of the Shuttle: Started 1994, and is one of the oldest and largest human data bases on the Web.
Special foundations of the US government
University of Michigan Government Documents Center: You will find a lot data, surveys, statistics, and more from high levels of the US government. Databases offered include Arts, Health Sciences, Social Sciences and International Studies.
USA.gov: A portal station for many agencies of the United States governmentehviruses. Includes government positions, services, and information on finding grants, loans, and financial aid.
Health and Science
PsycNET: Use the American Psychological Association database to find excerpts and entire journals on various psychology topics.
Scirus: One tool search engine that works exclusively for scientific information. The amazing search tool has hundreds of millions of scientific and academic documents to help researchers from all over the world.
Healthfinder: It contains information from over a thousand different health databases on the internet.
RXList: If you are looking for reliable information about medicines, then this database is for you.
The University of California, Riverside maintains it InfoMine, an incredible source of knowledge that at the last count contained over 100.000 links and access to hundreds, if not thousands, of databases.
There are generally too many websites set up to bring data from the Invisible Web. CompletePlanet.com is one of them. Contains "over 70.000 databases."
Most of the information about the invisible Web is kept by academic institutions. There are "academic gates" that can help you find this information. To find almost every educational resource on the web, simply type the following term into your favorite search engine:
site: .edu "topic I'm looking for"
Your search will return results for edu sites only. If you want to search for something from a specific university use the university URL in your search:
site: www.p university.com "topic I am looking for"
This is just the tip of the iceberg. All that we have mentioned in this article is just beginning to touch the huge resources available on the Invisible Web. As time passes, the Invisible Web becomes larger.