Many have the naive expectation that they can locate anything on the World Wide Web using search engines like Google or Yahoo or Ask.com or Bing. The truth is that all of these search engines are indexing 10% of the entire web. The remaining 90% is called “Invisible Web", the "Cloaked Web"Or"Deep Web. "
This means that there are huge volumes of data that are available to the public, but they are hidden by the search engines they all know.
It may be difficult for you to understand that billions of websites cannot appear in Google results. But they exist. Robot 'spiders' that scan and archive the world wide web have limited capabilities.
To understand better, let's start with some figures about the size of the services offered by Google.com, Yahoo.com, Cyberatlas, and MIT. The statistics are from the summer of 2013:
Google.com has 40 billions of public webpages in its archives. 100 + Two of these are static websites and publicly available. These pages can easily be found by Google and other search engines.
11 + two-static pages are hidden by the public after they have declared that they contain private content or are on the intranet. These are corporate pages that are open only to employees of these companies.
450 + billions of pages have databases that are completely invisible to Google. For example, government databases with tax information,
Google is considered to have the best base data in searches today. The company's spiders crawl millions of web pages every week.
So if Google has saved only one 8-10% of the World Wide Web and the other search engines have even smaller databases, then where does the remaining 92% of the content on the Internet hide?
The "Invisible Web" (or "Deep Web" or "Cloaked Web") is the content that does not appear in search engines.
More specifically: The Invisible Web consists of 220+ billion web pages that are not stored as static web pages. The Invisible Web consists of on-demand pages and databases. That is, pages that exist only as reports of changing data. As of August 2007, robotic spiders had not advanced enough to read these private databases. Only people can access it, and only if they have the knowledge.
Technical terminology:
"Spider": An artificial intelligence program, or robot, that has been sent to read millions of static web pages on the public Internet. The information collected by Spiders is stored in databases used by search engines.
Database-Driven Web Content: websites that exist only temporarily, and are only created when readers request answers from a large database. These temporary webpages are dynamic, and usually cannot be bookmarked. They usually have extremely large URLs.
The Invisible Web contains Dynamic Web Pages. This means that a database creates a temporary page to answer your question! Good one huh;
How can I use the Invisible Web?
There are too many who ask exactly the same. Let's look at some valuable databases below.
Humanities
Voice of the Shuttle: Started 1994, and is one of the oldest and largest human data bases on the Web.
Special foundations of the US government
University of Michigan Government Documents Center: You will find a great deal of data, surveys, statistics, and many of the high levels of the US government. The databases offered include Arts, Health Sciences, Social Sciences and International Studies.
USA.gov: A portal station for many government agencies in the United States. It includes government posts, services, and information on finding subsidies, loans and financial assistance.
Health and Science
PsycNET: Use its database American Psychological Association to find excerpts and full journals on various psychology topics.
Scirus: A search tool that works exclusively for scientific information. The amazing search tool has hundreds of millions of scientific, and academic papers to help researchers from around the world.
Healthfinder: It contains information from over a thousand different health databases on the internet.
RXList: If you are looking for reliable information about medicines, then this database is for you.
Mega-Portal
The University of California, Riverside maintains it InfoMine, an incredible source of knowledge that at the last count contained over 100.000 links and access to hundreds, if not thousands, of databases.
There are generally too many websites set up to bring data from the Invisible Web. CompletePlanet.com is one of them. Contains "over 70.000 databases."
Most of the information about the invisible web is maintained by academic institutions. There are "academic portals" that can help you find this information. To find almost any educational resource on the web, simply type the term below into your favorite searching machine:
site: .edu "topic I'm looking for"
Your search will return results for edu sites only. If you want to search for something from a specific university use the university URL in your search:
site: www.p university.com "topic I am looking for"
This is just the tip of the iceberg. All that we have mentioned in this article is just beginning to touch the huge resources available on the Invisible Web. As time passes, the Invisible Web becomes larger.
