Not many people know that ColdFusion ships with a HTTP spider that integrates with Verity. Unfortunately, this spider will only work with localhost as a server. This means if you want to spider multiple sites, you can't. Well, not without playing with your host headers. (More information on the Verity Spider and ColdFusion may be found here.)

What I worked on today was a way to work around this limitation. It turns out - if you have a sitemap, you already have a "spider" of your site. BlogCFC supports sitemaps out of the box, and I've blogged in the past a simple UDF to generate sitemaps. Let's look at how we can convert a sitemap into Verity data.

To begin with - let's take a look at some very simple sitemap data.
   view plainprintabout
1  <?xml version="1.0" encoding="UTF-8"?>
2  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
3  <url>
4  <loc>http://www.foo.com/index.cfm</loc>
5  </url>
6  <url>
7  <loc>http://www.foo.com/index2.cfm</loc>
8  </url>
9  <url>
10  <loc>http://www.foo.com/index3.cfm</loc>
11  </url>
12  </urlset>
This sample is missing many of the features that you can include with a sitemap, but it gives you an idea of the structure. As you could guess - a sitemap contains a collection of URLs. So let's look at how we can parse this XML. (Note - I'll be using ColdFusion 8 code throughout this demonstration, but you can easily downgrade this to CF7, 6, or even 5.)
   view plainprintabout
1  <!--- read in xml --->
2  <cfset myxml = fileRead(expandPath("./sitemap.xml"))>
3  <!--- convert to xml --->
4  <cfset myxml = xmlParse(myxml)>
The first thing I do is read in my sitemap and convert it to XML.
   view plainprintabout
1  <!--- place to store data --->
2  <cfset request.data = structNew()>
I'm going to be using threading, so I create a Request variable to store my information.
   view plainprintabout
1  <!--- now loop through.... --->
2  <cfloop index="x" from="1" to="#min(20,arrayLen(myxml.urlset.url))#">
3  
4      <cfset tname = "thread#x#">
5      <cfthread name="#tname#" url="#myxml.urlset.url[x].loc.xmltext#">
6          <cfhttp url="#attributes.url#" result="result">
7          <cfset request.data[attributes.url] = structNew()>
8          <cfset request.data[attributes.url].title = getHTMLTitle(result.filecontent)>
9          <cfset request.data[attributes.url].body = getHTMLBody(result.filecontent)>
10          <!--- remove all html from body --->
11          <cfset request.data[attributes.url].body = rereplace(request.data[attributes.url].body, "<.*?>", "", "all")>
12          <cfset headers = getMetaHeaders(result.filecontent)>
13          <cfset request.data[attributes.url].keywords = "">
14          <cfset request.data[attributes.url].description = "">
15  
16          <cfset request.data[attributes.url].x = headers>
17          
18          <!--- find description and keywords --->
19          <cfloop index="x" from="1" to="#arrayLen(headers)#">
20              <cfif structKeyExists(headers[x], "name")>
21                  <cfif headers[x].name is "description">
22                      <cfset request.data[attributes.url].description = headers[x].content>
23                  <cfelseif headers[x].name is "keywords">
24                      <cfset request.data[attributes.url].keywords = headers[x].content>
25                  </cfif>    
26              </cfif>
27          </cfloop>
28      </cfthread>
29  
30  </cfloop>
Ok, I have a lot going on here, so let me take it bit by bit. First off - I'm looping over the XML packet by treating the URL tag as an array. Note the use of "min". I did this to simply make my testing run quicker. My current blog's site map is over 2k URLs. (The cool thing is that it only took ColdFusion about 9 minutes to process all 2000 when I tried it.) For each URL, I suck down the content with CFHTTP. Now for the interesting part. I could simply provide the HTML to Verity. While this "works", it doesn't provide as much information to Verity as I would like. Also - Verity doesn't "get" that is indexing HTML data. It thinks it is just working with simple strings. So in order to make things a bit nicer, I do some cleaning. First off I use a few UDFs: getHTMLTitle, getHTMLBody, getMetaHeaders. The first and last are from CFLib and the middle one is a modified version of getHTMLTitle. These simply parse the HTML string for the title, body, and the meta tags. (I provide the full code at the end.) I remove all HTML from the body. I then look at the meta tags and specifically try to find a description and keywords meta tag. All of this gets stored into the Request scope Data structure I created. Now I need to tell ColdFusion to wait for the threads to end:
   view plainprintabout
1  <!--- join the threads --->
2  <cfthread action="join" name="#structKeyList(cfthread)#" />
I then convert my structure into a query:
   view plainprintabout
1  <!--- make a query for the data --->
2  <cfset info = queryNew("url,body,title,keywords,description")>
3  <cfloop item="c" collection="#request.data#">
4      <cfset queryAddRow(info)>
5      <cfset querySetCell(info, "url", c)>
6      <cfset querySetCell(info, "body", request.data[c].body)>
7      <cfset querySetCell(info, "title", request.data[c].title)>
8      <cfset querySetCell(info, "keywords", request.data[c].keywords)>
9      <cfset querySetCell(info, "description", request.data[c].description)>
10  </cfloop>
There isn't anything too fancy there - I'm just copying from the structure into a new query. Next I insert into the Verity collection:
   view plainprintabout
1  <!--- insert data --->
2  <cfindex collection="sitemaptest" action="refresh" query="info" title="title" key="url" body="body" urlpath="url" custom1="keywords" custom2="description" status="status">
Note how I tell Verity what column means what. I stored the description and keywords into the custom columns. That's it! To make sure things worked, I dumped some information at the end:
   view plainprintabout
1  <cfoutput>
2  <p>
3  Done indexing. Did #info.recordCount# rows. Took #totaltime# ms.
4  </p>
5  </cfoutput>
6  
7  <cfdump var="#status#">
All in all this worked ok - but it has a few problems/places for improvement:
  • First off - you can really tell that Verity wasn't 100% sure what to do with my data. That's why I removed the HTML. I could have considered taking the data I sucked down, saving it to an HTML file, and then running a file based index. While this would be slower, it could have resulted in better indexing.
  • Second - my code, ignoring the Mind(), will suck down every URL and index it. As I mentioned, sitemaps can store more than just URLs. They can also store the last time they were modified. If I were reading my XML data once a day, then it would make sense to only suck down URLs that were modified today. This would greatly improve the speed of the indexing.
Here is the complete index file:
   view plainprintabout
1  <cfsetting requesttimeout="600">
2  
3  <cfset thetime = getTickCount()>
4  
5  <cfscript>
6  /**
7   * Parses an HTML page and returns the title.
8   *
9   * @param str      The HTML string to check.
10   * @return Returns a string.
11   * @author Raymond Camden (ray@camdenfamily.com)
12   * @version 1, December 3, 2001
13   */
14  function GetHTMLTitle(str) {
15      var matchStruct = reFindNoCase("<[[:space:]]*title[[:space:]]*>([^<]*)<[[:space:]]*/title[[:space:]]*>",str,1,1);
16      if(arrayLen(matchStruct.len) lt 2) return "";
17      return Mid(str,matchStruct.pos[2],matchStruct.len[2]);    
18  }
19  
20  function GetHTMLBody(str) {
21      var matchStruct = reFindNoCase("<.*?body.*?>(.*?)<[[:space:]]*/body[[:space:]]*>",str,1,1);
22      if(arrayLen(matchStruct.len) lt 2) return "";
23      return Mid(str,matchStruct.pos[2],matchStruct.len[2]);    
24  }
25  
26  function GetMetaHeaders(str) {
27      var matchStruct = structNew();
28      var name = "";
29      var content = "";
30      var results = arrayNew(1);
31      var pos = 1;
32      var regex = "<meta[[:space:]]*(name|http-equiv)[[:space:]]*=[[:space:]]*(""|')([^""]*)(""|')[[:space:]]*content=(""|')([^""]*)(""|')[[:space:]]*/{0,1}>";     
33      
34      matchStruct = REFindNoCase(regex,str,pos,1);
35      while(matchStruct.pos[1]) {
36          results[arrayLen(results)+1] = structNew();
37          results[arrayLen(results)][ Mid(str,matchStruct.pos[2],matchStruct.len[2])] = Mid(str,matchStruct.pos[4],matchStruct.len[4]);
38          results[arrayLen(results)].content = Mid(str,matchStruct.pos[7],matchStruct.len[7]);
39          pos = matchStruct.pos[6] + matchStruct.len[6] + 1;
40          matchStruct = REFindNoCase(regex,str,pos,1);
41      }
42      return results;
43  }
44  </cfscript>
45  
46  <!--- create collection if needed --->
47  <cfcollection action="list" name="mycollections">
48  
49  <cfif not listFindNoCase(valueList(mycollections.name), "sitemaptest")>
50      <cfoutput><p>Creating collection.<p></cfoutput>
51      <cfcollection action="create" collection="sitemaptest" path="#server.coldfusion.rootdir#/collections">
52  </cfif>
53                      
54  <!--- read in xml --->
55  <cfset myxml = fileRead(expandPath("./sitemap.xml"))>
56  <!--- convert to xml --->
57  <cfset myxml = xmlParse(myxml)>
58  <!--- place to store data --->
59  <cfset request.data = structNew()>
60  
61  <!--- now loop through.... --->
62  <cfloop index="x" from="1" to="#min(20,arrayLen(myxml.urlset.url))#">
63  
64      <cfset tname = "thread#x#">
65      <cfthread name="#tname#" url="#myxml.urlset.url[x].loc.xmltext#">
66          <cfhttp url="#attributes.url#" result="result">
67          <cfset request.data[attributes.url] = structNew()>
68          <cfset request.data[attributes.url].title = getHTMLTitle(result.filecontent)>
69          <cfset request.data[attributes.url].body = getHTMLBody(result.filecontent)>
70          <!--- remove all html from body --->
71          <cfset request.data[attributes.url].body = rereplace(request.data[attributes.url].body, "<.*?>", "", "all")>
72          <cfset headers = getMetaHeaders(result.filecontent)>
73          <cfset request.data[attributes.url].keywords = "">
74          <cfset request.data[attributes.url].description = "">
75  
76          <cfset request.data[attributes.url].x = headers>
77          
78          <!--- find description and keywords --->
79          <cfloop index="x" from="1" to="#arrayLen(headers)#">
80              <cfif structKeyExists(headers[x], "name")>
81                  <cfif headers[x].name is "description">
82                      <cfset request.data[attributes.url].description = headers[x].content>
83                  <cfelseif headers[x].name is "keywords">
84                      <cfset request.data[attributes.url].keywords = headers[x].content>
85                  </cfif>    
86              </cfif>
87          </cfloop>
88      </cfthread>
89  
90  </cfloop>
91  
92  <!--- join the threads --->
93  <cfthread action="join" name="#structKeyList(cfthread)#" />
94  
95  <!--- make a query for the data --->
96  <cfset info = queryNew("url,body,title,keywords,description")>
97  <cfloop item="c" collection="#request.data#">
98      <cfset queryAddRow(info)>
99      <cfset querySetCell(info, "url", c)>
100      <cfset querySetCell(info, "body", request.data[c].body)>
101      <cfset querySetCell(info, "title", request.data[c].title)>
102      <cfset querySetCell(info, "keywords", request.data[c].keywords)>
103      <cfset querySetCell(info, "description", request.data[c].description)>
104  </cfloop>
105  
106  <!--- insert data --->
107  <cfindex collection="sitemaptest" action="refresh" query="info" title="title" key="url" body="body" urlpath="url" custom1="keywords" custom2="description" status="status">
108  
109  <cfset totaltime = getTickCount() - thetime>
110  
111  <cfoutput>
112  <p>
113  Done indexing. Did #info.recordCount# rows. Took #totaltime# ms.
114  </p>
115  </cfoutput>
116  
117  <cfdump var="#status#">