THE WEB is one crowded place. This summer estimates pegged the Internet's size at close to 2 billion sites -- a number that's expected to double by the first months of 2001. As in any big city, success in the World Wide Metropolis depends on potential customers' ability to find you.
According to Forrester Research, 81 percent of consumers use search engines as their primary tool for finding sites. But it can take up to six months for search engines to track you down on their own -- if they ever find you at all. Search engines and Web crawlers (search engines that search other search engines) like Yahoo, AltaVista, and others have mapped out only a portion of the Internet. To map, or spider, the Web, these engines and crawlers use software that automatically explores the Internet, collects text from pages it's found, and then stores collected information in a master database. "Google has about 25 to 50 percent of the Web that could be spidered, and the other ones are probably around the 25 to 15 percent range," says Danny Sullivan, editor of SearchEngineWatch.com. Google claims its crawler has collected 560 million full text indexed Web pages and 500 million partially indexed URLs.
Many sites adjust their HTML code to try to qualify for higher ratings from Web crawlers. But companies shouldn't get bogged down in HTML voodoo when there are much better options, Sullivan says.
First, build a really good site, one that includes content that will draw customers back, not just products for sale. Next, submit your URL to major Web crawlers like Google, Altavista, Excite, Go, and Hotbot. Then register the site with the major places that use human beings to list Web sites, such as Yahoo and Looksmart. These sites will also give you the option to pay to get listed. According to Sullivan, it's well worth the $200 fees, because people with good sites do get in, and for those who don't, the search engines provide helpful feedback.
Last, register with the Open Directory Project, www.dmoz.org. Like Yahoo and Looksmart, dmoz.org uses human indexers, but listing is free. The Open Directory Project is important because it feeds into the main search results at Netscape, Lycos, and AOL.