Welcome to "Quick Search on the Internet." This article was first assembled in 1996 by courtesy of TASC, a major US engineering services firm and my employer at that time (I am no longer with the company). I maintain this article periodically as a public service to the University of California, Santa Barbara. You are welcome to copy the HTML source, but please include this attribution.
I have worked as a systems engineer and technical writer in several technologies, for almost 35 years. Starting in the early 1990s, I developed and taught courses to help bring corporate staff into the new information age that the Internet implied. Of necessity, I learned a fair amount about finding information in this medium. This Web article is intended to share some of that learning with you, the reader.
A lot of people helped me along the way, and I believe in "giving something back". That's the title of my personal home page. If you're having trouble locating what you need on the Net, or you just want to talk with somebody who is reasonably knowledgeable about "this Internet stuff, " please visit me at Giving Something Back. You may also send me email at lawhern@hotmail.com and I'll try to point you in a useful direction.
The Main Idea
The Voice of the Shuttle covers a lot of territory. There are many useful resources in the page from which you may have clicked to get here. But like any Net effort, the Shuttle cannot be kept absolutely current to the world in real time. Links go stale. New links emerge. Web Wizards keep adding more information at more sites. The Wizards at UCSB have only so much time to maintain general resources like the Internet page.
Thus it's not surprising that Internet users sometimes have trouble locating information efficiently. This Quick Search page is intended to help newcomers overcome that disability by collecting and explaining high value resource sites where newcomers can become productive quickly. Many of the links are resources that information miners and market analysts use to earn their living. Each site is supported by some form of local or Network search capability.
The resources fall into four basic categories:
Sites that introduce the Net and survey its resources.
Sites that review and recommend specific resources by subject area and topic.
Net-Wide search engines or meta-engines.
Sites that evaluate search engines against each other and help you determine whether the "facts" you've recovered are really facts or only urban myths.
Internet Introductions -- Places to Learn about the Net
The following sites offer good "introductory" materials and links to Internet guides (sites that attempt to organize the information on the Net into broad hierarchies). I won't claim that these are the "best of the best" (a judgment that changes all the time), but each has a good reputation for value added. Some sites provide very large subject hierarchies or indices, that you may browse if you have enough time on your hands. Others are search-oriented.
The main idea here, however, is essential if you are ever going to become really productive in using this medium beyond a very basic level: you need more than a search engine to find the good stuff. You should eventually start to build relationships or email correspondence with people who are writing the good stuff. Experts can always tell you more than the individual documents you locate. In time, you may become one of these people.
So let's start with some links where you might learn more about how information is organized on the net and where you might find the people who write it:
Argus Clearinghouse started out life as the Clearinghouse for Subject-Oriented Internet Resource Guides, at the University of Michigan. This resource provides a large archive of Information Guides to help you find Internet resources that pertain to various subjects. Each guide is rated against five different criteria by the managers of the Clearinghouse. The site is a "must see" for anyone who needs to learn where in the world a subject area is taught, archived, or discussed.
Ohio State FAQ Archive USENET is a collection of thousands of topical discussions in "special interest groups" or "newsgroups". USENET is supported by email and read by software built into Web browsers or email applications. People active in each newsgroup often publish a periodic posting or "FAQ" -- Frequently Asked Questions -- file. Such a file surveys the subjects discussed in the group and reminds participants of any rules concerning what belongs there and what doesn't. The Ohio State FAQ Archive collects such files under a search engine. It's a good place to find a discussion group or a person who shares specifically the subject of your interest.
FAQ postings are also available from The MIT USENET FAQ Archive. The MIT link takes you to a directory tree, starting from the most general USENET domains (such as comp. or talk.) and working down to specific interest groups. As an "exercise for the student," try to find other archives of USENET FAQ files. Several exist. Try a search engine, below...
The Internet and Computer Mediated Communications -- John December's on-line guide to Internet services. December is one of the first "Internet media experts", and he's been organizing and investigating resources in this medium since the early 1990s. This particular resource has a number of good introductory links for finding your way around the Net and learning what this medium is capable of. It can also help you learn unusual things such as how to use email to gain access to an ftp site (or even what is meant by the term "ftp"). This is a place you go to study the Internet itself -- as distinct from the information to which the Net provides access.
WWW Review Sites -- places which evaluate the "best" places in the Net.
A number of sites on the Internet are well known for digesting, reviewing, and providing links to "cool places" and authoritative, well designed sites. Here are some helpful places to learn. Some are very large, and some are smaller.
About - The Human Internet - Very large index of sites by subject area, with 750 "guide sites" supported by resident experts. Click here for the current list of guide sites. This resource is particularly well known for excellent healthcare information.
Excite Web Guides - Provides a directory where you can drill-down through broad subject areas, or search the entire site. Listings of resources provide short (3 line) descriptions of what each resource provides. This project has grown so large that Excite no longer supports a star-rating system like its predecessor (McKinley's Magellan)
Point Communications Best 5 Percent of the Web. At this site (now operated by Lycos), when you browse the Best of the Web in a list of broad knowledge domains -- you get reviewer notes on content and page design. The index as a whole is searchable. Browsing by general subject category is useful as an overview.
Select Surf -- In the words of the site management, " Select Surf is designed to be the most efficient easy-to-use guide to Internet containing only the very best sites online. Select Surf offers point and click access to over 10,000 of the web's top sites."
Yahoo - search for hundreds of thousands of commercial and academic sites by subject area. Yahoo staff review and select sites submitted for listing. The subject hierarchy is very extensive and quite useful for somebody who wants to browse. The local directory search engine is efficient and helpful. Many information services are linked from the page menus, including maps, weather, "ask the experts" etc.
WWW Search Sites -- Places to Find Documents from All Over
To find the latest information -- or to discover what information is pertinent to you personally, "Inter-nauts" need knowledge servants capable of slicing through and sniffing terabytes of data on the Net. This section provides gateways to several sites where such knowledge servants are being born and raised. Enjoy (in more or less alphabetical order).
Some of the engines below provide search not only of the World Wide Web, but also of specialized materials assembled by the site owners. Although none of the engines below charges a fee for basic service, there are a few others not listed, that do. Other engines (called "meta search engines") submit your inquiries to multiple Internet search sites, and collate the results for presentation in a combined output form.
First, a "Meta Page of pages" -- a site from which you can access a large number of search engines, including the individual engines listed below.
All-In-One Search Page. A creation of William Cross, providing direct search submission and links to hundreds of search engine gateways organized in general categories. There are gateways here to specialized resources as well as general search engines.
Next, some search engine sites that I've found useful over the years.
Altavista -- for over one billion Web documents and growing all the time. To focus your searches, go to "Power Search" and learn now to weed out the wheat from the chaff.
Google -- a serious competitor to Altavista, with a similarly large index. Like an increasing number of other search engines, this one will look for "close matches" to the spellings of your target words, as well as finding documents that have some of the words from your target "phrase search" but not all.
Google Groups -- the inheritor and current custodian of several years of postings to thousands of USENET newsgroups, formerly hosted at Deja News. This is a good place to go when you want to locate discussions between experts in a field, or current opinion on news events.
Lycos Home Page. Lycos started out as the original long-legged Web spider. After a number of mergers with other brand-name search services, the new Tera Lycos continues to be a serious contender in the web indexing sweepstakes. This index is based on abstracts of a huge number of documents, not full text: headers, titles, the first few words of some paragraphs. I have found in the past that Lycos seemed to work best for simple search terms. However all search engines continue to evolve, so my observation is subject to change.
Northern Light Search. This service, begun in 1995, combines Internet and Web search, with local full-text materials from over 7000 publications that do not appear on the Internet. Likewise, some of the Northern Light information directories can be searched in narrower topic areas, notably "business".
WebCrawler This index was the first broadly successful search engine on the Net. It hasn't historically been the largest, but it's been improved continually as it passed through the hands of AOL to Excite. In the past, I've found that "hits" returned from the engine seem to read as somewhat "terse". However, the Crawler provides listings which are easily re-mounted as bookmark pages. Boolean search is supported.
Evaluating Search Engines
With so many search engines competing for acceptance (and commercial market) as "the best on the Web", it is understandable that site would emerge that attempt to independently compare and evaluate the performance of the competitors. Also understandably, the responses of service owners to these reviews have ranged from enthusiastic to pointedly critical (depending on the relative ranking of their site). Without claiming any particular "rigor" in my own evaluations of various search resources, I suggest you review the following documents:
Hope Tillman maintains an on-line version of a comprehensive paper discussing the problem of "Evaluating Quality in the Internet". Ms. Tillman is an experienced librarian with wide experience in both paper and on-line research and sources. Her web work is widely published and acknowledged for both quality and thoughtfulness.
Search Engine Watch -- Because search engines are constantly evolving and competing for commercial notice, it was inevitable that newsletters or archives would emerge to keep track of which engines are doing what, and to provide detailed information on how the engines work. Search Engine Watch is widely regarded as one of the best of these services. If you intend to use engines frequently, the Search Engine Watch daily or weekly newsletter is a useful tool. Subscription is free.
Web Search Service Features -- A top level summary comparison of how several major search engines work and what features they support.
So let's say you've done an Internet search. AltaVista came up with 4,531 hits, and Yahoo found seventeen companies doing business in the areas of your interest. At least a few gems in this pile seem relevant to your project...
NOW WHAT? Simply put, you face the same problem that researchers have long confronted in traditional media: determining the validity or truth of the (ever more voluminous and -- these days -- no longer publisher-moderated) "information" you have recovered. Your process is much the same as in traditional research:
Compare different sources -- on-line and off-line.
Read up on specialist opinions concerning "authoritative" sources.
Assess the depth and breadth of author publications -- Internet and periodicals.
Find a specialist discussion and listen in (called "lurking").
Find an expert and ask in person -- by phone or email.
Document and attribute your sources.
Internet sources can help with parts of your data validation process, but not all. What you need most is often going to be a good library. The following links will help you find one in your neighborhood or across the world, and to determine if it has an information source you need that isn't directly available on the Net.
The HYTELNET Library Lists. The On-Line Publicly Available Catalogs (OPACs) of hundreds of University and College libraries across the world can be accessed via the Internet. The HYTELNET project collected many of the WWW gateways to such resources. This list is still useful despite being no longer actively maintained. The author of HYTELNET also recommends the Library Index as an alternative.
An alternate collection is Libweb - Library Servers via World Wide Web - maintained by Thomas Dowling of Ohio Link, on a University of California Berkeley server. The site maintains links to libraries across the world.
Liszt -- The Master list of Email Discussion Groups can provide you with a less timely but often very useful gateway to something more important for real research than any document you'll ever find in the Net: PEOPLE WHO KNOW. Various email discussion groups provide forums for focused subjects and practitioners of professional or technical specialties. Some of the LISTS are also carried as newsgroups on the USENET. Others are not. This link will provide a starting point for learning how to use and participate in such discussions.
This concludes your introduction to efficient searching on the Internet. If you need further help, email me at lawhern@hotmail.com and we'll see what we can do. ENJOY.
A Word About TASC on the Internet
When I first developed this page, I worked for TASC Inc. They are an engineering consulting firm with broad capabilities in information systems technology, research, integration, and architecture assessment for commercial and government customers. Though I am no longer affiliated, I include this reference as a professional courtesy to my former colleagues: To learn more about TASC, point your browser to:
