Can Searchbots Find Web Pages That Aren't Linked To?

Problem Statement

Suppose I put a web page on my web site, but don't link to it. Will searchbots find it and index it?

Experiment

I created an XHTML document and an XML document. They both contained the same data. The XHTML document was marked up using the XHTML tags. The XML document mimicked the XHTML document, but used tags that I created. The XHTML document was validated using the W3C Markup validator Service. The XML document was checked for well-formedness by dropping it into Internet Explorer, version 7.

The documents were very simple, containing a definition of progressive enhancement. The title of each document was "Definition of Progressive Enhancement"

February 27, 2008

I uploaded the XHTML and XML versions onto my public website (xfront.com). I did not link to them.

March 9, 2008

I used seven different online web search tools and searched for "Definition of Progressive Enhancement" to see if they had found my documents.

XHTML Version

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>
    <meta http-equiv="content-type" content="application/xhtml+xml;charset=UTF-8" />
    <meta http-equiv="content-language" content="en" />
    <meta name="author" content="Roger L. Costello" />
    <meta name="description" content="Definition of Progressive Enhancement" />
    <meta name="keywords" content="Definition, Progressive Enhancement" />
    <title>Definition of Progressive Enhancement</title>
</head>
<body>

    <h1>Definition of Progressive Enhancement</h1>

    <p><dfn>Progressive Enhancement</dfn> is adding layers of enhancements on top of core content. Applications
       process whatever enhancements they have support for, and ignore those enhancements for which they do 
       not support.</p>
    
</body>
</html>

This HTML was placed in a file, with the filename: ProgressiveEnhancement.html

XML Version

<?xml version="1.0" encoding="UTF-8"?>
<Progressive_Enhancement>

    <Metadata>
        <Content_Language>en</Content_Language>
        <Author>Roger L. Costello</Author>
        <Description>Definition of Progressive Enhancement</Description>
        <Keywords>Definition, Progressive Enhancement</Keywords>
        <Title>Definition of Progressive Enhancement</Title>
    </Metadata>
    <Word_Definition>

        <Header>Definition of Progressive Enhancement</Header>

        <Definition>Progressive Enhancement is adding layers of enhancements on top of core content. Applications
        process whatever enhancements they have support for, and ignore those enhancements for which they do 
        not support.</Definition>
    
    </Word_Definition>
</Progressive_Enhancement>

This XML was placed in a file, with the filename: ProgressiveEnhancement.xml

Results

I used seven different online web search tools and searched for "Definition of Progressive Enhancement" and recorded how the XHTML and XML versions were ranked in the results. Here are the results:

Google: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.
Yahoo! Search: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.
Altavista: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.
Dogpile: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.
Live Search: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.
Excite: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.
Go.com: the HTML version did not appear in the list of results. The XML version did not appear in the list of results.

None of the search engines listed either the HTML document or the XML document in its list of results.

Conclusions

Searchbots can't find your web page if it's not linked to by another web page.