Due to the excellent performance by sentence rank, a further experiment is conducted: applying sentence rank on real news web pages. In this section, due to the length of report, only implement undirected graph and 10 terms per query, the following success retrieve rate shows a high percentage value when the cosine similarity on 2 web pages is applied by using 4.1and 4.2. 10 terms a query means only take first 10 words in the selected sentence including stop words which is consistent with section 4.6. Unlike locating the exact address of a web page itself, this comparison leads to find similar topic document by comparing 2 different URL web pages, the details are all introduced in section 3.4.

  4.1

 4.2

Meanwhile, there are 3 search engines employed in this section: Yahoo News Search, Yahoo Web Search and Google Web Search. Unlike from section 4.6 to section 4.1 which only count URL string match as success retrieval, section 4.7 take document similarity into consideration, and if equation 4.2’s value is bigger than 0.9, which is also permitted in S. T Park and Xiaojun Wang’s research, a success retrieval is considered effective. There are 183 pages in this section which are all from May 4, 2009, Yahoo News, and all related URL addresses are listed in Appendix D.

Table4.29

(a)                                                                                        (b)

Figure4.32

As Figure4.32’s (b) shows, the success rate is above 90% which satisfies the project’s initial requirements by applying a single text retrieval method.

As stated in chapter 3, the sentence rank can significantly improve the linguistic summarization other than the traditional TF or DF methods. Considering the complexity in sentence rank, randomly pick a sentence and take the first 3 to 15 words from the sentence within its original order as search query can avoid the iterations in graph-based ranking algorithm, and the results below show that even the sentences are randomly picked, when the number of terms up to 10, the performance increases enormously, some of them are higher than 75%, which cannot be accomplished easily by the previous carefully designed retrieval algorithms.

Table4.24

(a)                                                                                        (b)

Figure4.26 Random Sentence Pick from Google and Yahoo results

The text in HTML’s title tag is always playing a vital role in web page retrieval. During the beginning of this project, an extensive amount of experiments were conducted by using the title method. It was believed that the success rate would reach 90% from using title text as a query if the query could be composed carefully and properly. Figure4.12 shows that the title method also has a good stability along with the words number in a query. It is important to mention that, from Figure4.1 to Figure4.10, although the classic methods have better results, it only means the HTML extractions have good performance, which filter the structural HTML tags and functional scripts which could be big distractions in the following application on the target page, because all the basic retrieval process is only designed for pure text without structural tags. For example, HTML tags like ‘td’ and ‘tr’ will have a big term frequencies and the function or variable names in Javascript will cause a very low document frequencies, if they are not filtered or removed in the pre-processing step. However, by using title method, it is much easier to extract the text information only between <title> and </title>.

Table4.11

(a)                                                                                                  (b)

Figure4.12 Use title terms as search query

 A real implementation from this project is whether the ability of testing on general search engine can be applied on testing deep web search engine. The general search engines such as Google and Yahoo have been widely approved in their proper results and links. However, many sites may not allow their documents to be indexed but instead may allow the documents to be accessed through their search engines only, these sites are part of the so-called Deep Web ^[1][17]. The deep web search engines which only focus on their own data base and pages, data or documents which are kept privately and cannot be searched by general search engines. Take www.taobao.com as an example, it is a online commercial trading site like www.ebay.com, Taobao apparently abandons general search engines such as www.baidu.com and www.google.com to access its commodities results after the negotiations broken with the big search engine companies. This leads people who want commodity and price information have to go directly to Taobao’s own search engine interface and browse result items in Taobao’s website. Obviously, search engines in Taobao are probably developed by their own or contract consultant software teams, the performance then will be an interesting topic rather than the ones generally accepted by the public such as Google and Yahoo. The specific introduction for deep web and implementation of deep web search engines are not part of this project, but the practical value from this project can offer a feasible way in testing local and small search engines embedded in their own web sites.

If there is a URL match or content match, a success retrieval is established. If the URL does not match, due to URL’s changing all the time ^[2][3], comparison between original page and retrieved pages is indispensable and taken by 2 ways, manually and automatically. Manually checking all the content between original page and retrieved pages is time consuming but it can guarantee the precise results. In this project, we pick around 200 pages from the data source for manual checking. Rather than by brute force, automatic comparison between the result pages from search engine and each test page also needs HTML page preprocessing as in step 2.

         3-1

          3-2

In 3-2, TF_w is the word’s term frequency in document 1 or document 2. In this project, some necessary removing are applied on pages, therefore, the comparison between 2 pages is only focusing on the main content which means all the advertisement, copyrights information, sponsor’s links and information are removed. It can be concluded as finding a similar topic within 2 different pages. Here are 3 pairs of example pages listed from Figure3.22 to Figure3.24. By using undirected weighted sentence rank algorithm, the highest ranking sentence can be picked up, input as a query into SE and then compared to the result page.

Figure3.22 (a) and (b) is an example of proving the validity of cosine comparison. The post time is shown in the red circle. In Figure3.22 (a), it doesn’t show the date but “34 mins ago”. In Figure3.22 (b), it shows “Mon Mar2, 11:57pm ET”. Actually, Figure3.22 (a) was downloaded in the morning on March 2, 2009 and Figure3.22 (b) was downloaded at noon on the same day. Apparently, Yahoo news editors keep updating and modifying the same news, so the later one gives some differences in the content but actually they are talking about the same issue.

Figure3.22 shows the downloaded HTML file images and (a)’s URL is

http://news.yahoo.com/s/ap/20090302/ap_on_re_us/winter_storm

The retrieval URL is http://news.yahoo.com/s/ap/20090303/ap_on_re_us/winter_storm_43

By comparing the different URL, it is obviously that even about the same content, yahoo news changes URL by adding “_43” in the end.

 

(a)                                                    (b)

Figure3.22

 

(a)                                                                      (b)

Figure3.23

Figure3.23 (a) and (b) is an example of finding a similar content web page, according to a downloaded local page Figure3.23 (a). Obviously, they are both talking about the missing NFL player in Florida’s Gulf which is one of the most popular news at the time of this experiment.

Figure3.23 (a)’s URL is

http://news.yahoo.com/s/ap/20090302/ap_on_re_us/missing_boaters_nfl

Figure3.23 (b)’s URL is

http://www.npr.org/templates/story/story.php?storyId=101375823&ft=1&f=1003

The documents’ similarity is 98.38% by 3-2

Figure3.24 (a) and (b) is another example of finding a similar content web page according to a downloaded local page. They are both talking the children’s blood lead level.

Figure3.24 (a)’s URL is:

http://news.yahoo.com/ /s/ap/20090302/ap_on_bi_go_ec_fi/economy

Figure3.24 (b)’s URL is

http://www.ajc.com/services/content/health/stories/2009/03/02/children_lead_level.html?cxtype=rss&cxsvc=7&cxcat=9

The documents similarity is 94% by 3-2.

(a)                                                                                        (b)

Figure3.24

In chapter2, section2.1, S. T. Park adopted 5 terms a query. However, a wider range of term numbers in a query is adopted in this project: the length of LS from 3 to 15 versus the success rate is compared together while sentence rank, take first N words in the selected sentence, from 3 to 15, even including stop words from the top ranked sentences as a search query and remove the rest of them left in the sentences. This procedure does not follow the traditional ways in text retrieval, however, in chapter 5, the experiments show even better results when the terms number are more than 10, compared to the same terms number in traditional ways.

It is necessary at this point to clarify why LS extraction cannot be applied directly on the raw web page which are downloaded in full size without any parser. The first is, other than pure text information retrieval, the web pages have their unique feature, HTML tags, which help to construct page template, font format, font size, images insertion and other components for a fancy appearance. However, these good looking gadgets in the web pages actually are the sources of distractions and interferences when the applications are trying to analyze them. Because only the showing text part in a web page is useful in common sense. How to transfer the HTML page to pure text by removing all kinds of hidden tags is a key issue to the following steps and decide the final results. The text in the page must be all extracted at first, meanwhile, the tags information behind the text can not be simply discarded, for example, in Michal’s research, she classified and saved the text into 6 different categories, each category takes a unique weight. The second is, the link information is also a powerful hint in deciding the unique feature of a particular web page. For example, the commercial search engines largely depend on the algorithms like page-rank and authority and hubs. Even for searching and retrieval studies in academic papers, the citation rank algorithm is also widely accepted. However, not same like academic papers, which contain the citations in the end of each paper as a references chapter, web pages’ link information hides in the anchor tags, which leads to more complicated data-source preprocessing before LS extraction. Construct a query with extracting the link information, such as the domain that the page belongs to, combined with LS could be another study but not included in this report.

Pros: simple REST API which can be directly accessed via Javascript or virtually any server-side language; offers up to 1,000 results in chunks of up to 50 at a time; offers results in at least three different formats: xml, JSON, and serialized php.

Cons: no access to Google search services, only Yahoo; only offers up to 1,000 results in chunks of up to 50 at a time; per-IP address rate limit of 5,000 queries/24-hour period; absolutely no UI elements. It brings difficulty in programming, such as if people want to build a client-side approach, they have to build it from scratch almost. People are expected to route any links through Yahoo’s servers so they can track traffic.

Figure3.9 Yahoo Web Search API Java snippet

Figure3.9 is an implementation of Yahoo web search API, with typical address in red box. While applying Yahoo news search API, the address in red square can be changed into

String se = "http://search.yahooapis.com/NewsSearchService/V1/newsSearch?appid="+uid+"&query="+query+"&results="+total+"&language=en";

Both Google Ajax and Google Base Data API have restriction on the number of results and the client application does not have a free control like manually browsing the web page. It appears that Google is checking the User-Agent header to make sure that people are doing this from a browser that it knows about, otherwise it will deny people’s access (i.e. they don't want people doing this from an application unless they use their API service such as section 3.2.1 and section 3.2.2. It blocks the client’s java program access). Now, one method is found by simply changing some parts of java code. By following the piece of code in the red box can solve this problem.

Figure3.8

Google Base Data API is another way for programmers who want to write client applications that interact with Google Base. With the query like this:

http://base.google.com/base/feeds/snippets?bq=query&key=xxxxxxx

The value of “bq” is the input query and the value of “key” is a unique ID provided by Google after registered in Google. The result is returned in clean XML format. However, there are still some limits, with the same query, Google Base Data API returns different results compared to its original web interface.

Google’s SOAP Search API would have the ability to access Google’s results, but it has been deprecated since late 2006 and no new license keys have been distributed. Instead of SOAP API, Google released its new Ajax for researchers and the search query should like this:

http://ajax.googleapis.com/ajax/services/search/web?start=1&rsz=large&v=1.0&q=“query”

Figure3.6 Google Ajax code snippet

As 2 of the most powerful search engines, Google and Yahoo have the strongest abilities in searching the surface web and they also provide all kinds of different special search functions such as web search, news search, image search which are familiar by the people all over the world. A general experiment in exploring the search abilities without testing these 2 search engines is certainly not conclusive.

Meanwhile, the detailed search ability test implementation on Google, Yahoo or other search engine needs programming according to their result pages, specifically speaking, they are shown in different HTML templates. For example, Figure3.3 and Figure3.5 are the returned pages from Google and Yahoo, with the same query “job search”, currently, the differences or quality are not compared in this section. Although the 2 result pages Figure3.3 and Figure3.5 show a very similar format such as the search engine input interface at the top, the main content is in left and takes more than 2/3 spaces and leaving the right 1/3 to commercial websites as advertisements, the HTML behind the pages show quite different grammar and make extracting the result title, result summary, result URL not a single general template, but specifically one extracting algorithm for one search engine.

Here is a segment from Google result page, the texts shown in Figure3.3 are also in the red boxes in Figure3.2.

Figure3.2

Figure3.3

Here is one segment from Yahoo web search result page.

Figure3.4

Figure3.4 is a segment from Yahoo result page and the text shown in Figure3.5 is also in the red square in Figure3.4.

Before parsing the HTML in Figure3.2 and Figure3.4, both Google and Yahoo provide easier ways for the developers to parse and extract results information. It is quite necessary to carefully examine their API first. Meanwhile different kinds of API provided by Google and Yahoo show quite different capabilities in returning the result links.

Figure3.5

The project is designed to find a best query (LS query) which can represent the web page and be searched by search engine easily. The web pages are picked up from the surface web which is easily accessed without any authority restrictions. The experiments are arranged in following general steps:

(1)     Find a large amount of web pages which will be used for LS extraction and re-finding/relocation, download them as test pages into local disk and save their URLs at the same time. The source of web pages used for experiments requires careful considerations as all kinds of web sites showing up today, some of them are ill-formatted or have poor information. Obviously, they are not the ideal sources in experiments. The detail of web pages selection and crawling is non-trivial and will be introduced in section 3.3.1 specifically.

(2)     HTML parsing and text extraction are needed before extracting LSs. The preprocessing reasons and steps will be in section 3.3.2.

(3)     Apply the algorithms in chapter 2 which are designed to extract lexical signatures from these downloaded pages. In chapter 4, TF, DF, TFIDF, TF3DF2, TF4DF1, TFIDF3DF2, TFIDF4DF1 as well as word rank and sentence rank will be applied to the pages after step2. But different from S. T. Park’s experiments, in this project, varying term numbers in a query is accepted, which make such as TF3DF2 and TFIDF3DF2 to  and , the ratio in selecting between DF order terms and TF or TFIDF order terms is not changed. For convenience, the selections on TF, DF and their mixed forms are listed from (a) to (h) which have been precisely described in S. T. Park’s paper. Although the topic of how many terms are going to help getting better searching results is studied and unveiled by Martin and Michael, they claim that 5 terms a query is the most efficient number in getting the desired result appeared in top 10 from SE ^[3]. More terms in a query means obviously more feasible in sentence-rank. Meanwhile, Google raised their web search limit to 32 words 4 years ago back to 2005. In this project, up to 15 words are allowed per query, with succeeding words being ignored. The performance with different number of terms in a query is an interesting topic and they will be explored in chapter 4.

a)         TF: Select terms in decreasing term frequency (TF) order. If there is a tie, then pick terms based on increasing document frequency (DF). If there is another tie, randomly select the terms ^[5].

b)         DF: Select terms in increasing DF order. If there is a tie, then pick terms based on decreasing TF order. If there is another tie, randomly select the terms ^[5].

c)         TFIDF: Select terms in decreasing term-frequency inverse-document frequency (TFIDF) order. If there is a tie, then pick terms based on increasing DF order. If there is another tie, randomly select the terms ^[5].

d)         PW: Select terms based on Phelps and Wilensky’s ^[3] method: First, list terms in TF order and then pick LS in decreasing TFIDF order (i.e., decreasing TFIDF order where the TF term is capped at five). If there is a tie, then pick terms based on increasing DF order. If there is another tie, randomly select the terms ^[5].

e)         TF3DF2: Select two terms in increasing DF order. If there is a tie, then pick terms based on decreasing TF order. If there is another tie, randomly select the terms. Then filter out all terms which have DF value 1. Select three terms maximizing TF. If there is a tie, it is resolved the same way as with TF method ^[5].

f)          TF4DF1: Select one term based on increasing DF order first. Then filter out all terms which have DF value 1. Select four terms maximizing TF ^[5].

g)         TFIDF3DF2: Select two terms based on increasing DF order first. Then filter out all terms which have DF value 1. Select three terms maximizing TFIDF ^[5].

h)         TFIDF4DF1: Select one term based on increasing DF order first. Then filter out all terms which have DF value 1. Select four terms maximizing TFIDF ^[5].

(4)     Select several search engines which support developer mode. Google web search and news search, Yahoo web search and news search are picked as general search engines in the test. More details are in section 3.2.

(5)     Download the result links in first page returned by SE as well as their corresponding HTML pages. If there is a URL match between test URL and result URL, a successful retrieval is recorded. However if there is no match between test URL and first 10 result URLs, then, comparison between the test page and result pages from search engine is done to see if they have the same topic or content. It is similar to step 2 because both of them need HTML page parsing, extraction and converting them to understandable pure text without noises from HTML tags. Also, the criterion of considering whether

(6)     Repeat step 2 to step 5 on all test pages downloaded by step 1, count all the success retrievals and compute a success retrieval rate which is in 3.1. This is the most straightforward measurement which is also adopted in both S. T. Park and Xiaojun Wang’s researches. Martin and Michael’s LS score is also a good way to evaluate the query generation, however, their score is not as straightforward as 3.1

      3.1

(7)     A stable and reliable LS extraction algorithm should help the success retrieval rate be higher than 90% as the ultimate requirement. The traditional ways which do not maintain the linguistic meaning such as TF shows quite different results comparing to graph-based rank algorithms which maintain the linguistic meaning. For example, after step 3, all terms are listed in alphabetic order and query’s terms are picked from a to z. In chapter 4, the results from Google and Yahoo show both of those search engines have a strong ability in returning desired pages by meaningful query rather than discrete terms simply ordered from TF, IDF and so on, even some stop words are included. The results and analysis will be in chapter 4.

Figure3.1

Figure3.1 is the flow chart showing the overall procedure in experiments including from step 1 to step 7. The URL in the left top rectangle of Figure3.1 is a directory built with a large amount of effective URL which can be accessed online. The corresponding pages are then downloaded and saved on local disk.

The algorithms in extracting lexical signatures from a page are in 3 rectangles in the right top of Figure3.1. They are the core in this project. Another key programming issue is 2 rectangles in button of Figure3.1 page content comparison and results record. The middle rectangles in Figure3.1’s left and right are a pipeline showing all the operations.

Similar to Xiaojun Wang’s applying word rank on web pages, this project applies sentence rank on web page. The passages in the web pages can be extracted as shown in Figure2.19’s red square.

Figure2.19

However, there are conditions sentence rank can not work. Some web pages may not have sentences or passages, which make sentence rank on those pages not effective when there are only titles or phrases. Take Figure2.20 as an example, there is not any passage in Yahoo’s home page, and although there are several sentences in the center 3 red squares which are shown in anchor tags separately, it brings difficulties to construct connections among those independent sentences, because they actually come from completely different topics. Meanwhile, there are a bunch of simple words and phrases in the left blue squares, such as “answer”, “auto” and “finance”. It brings challenges in combining the terms and sentences as well as applying sentence rank. Therefore, the page like Figure2.20 is not a typical type can be applied by sentence rank.

Figure2.20 A typical example of link-based page

There is a simple way to exclude the pages which are not suitable for sentence rank. A threshold p is defined to separate the pages into 2 categories linked-based page and content-based page after using formula 2-11.

  2-11

The pages like Figure2.20 can be concluded as a linked-based page which has a high portion with text in link. The linked-based pages are easily found from the websites’ home page and index page. Compared to Figure2.19, a content-based page has high portion in plain text without link such as Figure2.20.

In previous section, a graph-based ranking algorithm on words is introduced. In this section, the vertex in the graph is applied on the sentences. Compared to word-rank, sentence rank preserves much more linguistic information and the results from sentence rank are more understandable because now the LS is composed by whole sentence rather than terms and phrases.

Similar to the word rank graph, without considering the un-weighted graph, there are 2 kinds of graph construction for sentences in a document: undirected weighted graph and directed weighted graph. Moreover, Rada and Paul proposed another form of directed weighted graph and they classify directed weighted graph into directed forward weighted graph and directed backward weighted graph ^[9]. In undirected weighted graph, the assumption is that every sentence in the document has connection with all other sentences. In directed forward weighted graph, every sentence points to all the following sentences in text, while every sentence receives all the previous sentences in text. In directed backward weighted graph, every sentence points to all the previous sentences in text, while every sentence receives all the following sentences in text.

Figure2.17

Figure2.18 shows a 5-sentences passage. Figure2.18 shows the 3 graphs in detail. In Figure2.18 (a), as an undirected graph, every sentence has connection with other 4 sentences with a pair of in coming pointer and out coming pointer. In Figure2.18 (b), as a directed forward graph, sentence 1 points to sentences 2, 3, 4, 5; sentence 2 points to sentences 3, 4, 5; sentence 3 points to sentences 4, 5; sentence 4 points to sentence 5 and sentence 5 does not have out coming pointer. As a directed backward graph, Figure2.18 (c) shows the (b)’s reversed order, sentence 5 points to sentences 1, 2, 3, 4; sentence 4 points to sentences 1, 2, 3; sentence 3 points to sentences 1, 2; sentence 2 points to sentence 1 and sentence 1 does not have out coming pointer.

                                                                                                                         a. undirected       b. directed forward     c. directed backward

Figure2.18

Therefore, after the direction being decided, the weight between the connections is about computing the 2 sentences’ similarity, as in equation 2-10, the common terms shared by both sentences are divided by the length of both sentences in log form. Length can be simply considered as the terms number in sentence.

  ^[10] 2-10

Generally, W_k is a word appearing in both S_i and S_j. W_k can also be the words with same meaning but different forms such as “interested” and “interesting” which could finally prove the ability in finding the similar or related documents. If there is no common word W_k, then the edge between S_i and S_j can be removed.

After iterations on 2-8 and 2-9, the sentences from text with the highest rank are selected and taken as an abstraction. In [9], 4 highest ranked sentences are extracted after the iteration, resulting in a summary of about 100 words. Rada and Paul also evaluated the abstraction from this graph ranking algorithm with other methods such as HITS and Positional Power Function (POS) by ROUGE evaluation ^[15]. It turned out the sentence rank from page rank provided good performance. The details are not included in this report.

After all, Rada and Paul concluded that, based on this sentence rank algorithm, the abstraction from the extracted sentences are more informative for the given text and it does not require deep linguistic language nor domain or language specific annotated corpora ^[10].

Xiaojun Wang applied the word-rank algorithm to the LS extraction on web pages for finding lost or related web pages ^[11]. He also compared the results from this graph-based ranking algorithm with the traditional ways in extracting the terms from documents, such as TF, DF, TFIDF, PW, TF3DF2, TF4DF1, TFIDF3DF2 and TFIDF4DF1. He pointed out, with word rank algorithm, which takes the semantic relation between words into account and chooses the most representative and salient words as Lexical Signatures, the highly relevant web pages can be found when the desired web pages can not be retrieved ^[11].

In their experiments, Wang not only used the basic word-rank algorithm, but also combined it with DF to select the terms. In [11], Wang only constructed undirected weighted graph model G=(V, E), V is the vertex set containing all words except stop words. E is a set of undirected and weighted edges. Each vertex’s initial score is the normalized  value and set the damping factor d=0.85. Wang did not use window size but WordNet ^[14] to recognize the semantically related words and Wang did not mention the value of weight on edges. These 2 detailed implementations are definitely related to a large amount of work but they were not listed in their paper “WordRank-Based Lexical Signatures for Finding Lost or Related Web Pages” ^[11].

   ^[11] 2-8

   ^[11] 2-9

Similar to 2-7 and set the convergence threshold to 0.0001, 2-8 is run on the graph until it converges based on 2-9.

In Wang’s experiments, he selected Google and MSN Search, randomly crawled 2000 URLs from domain DMOZ.org and kept 1337 pages after filtering out the unqualified pages, such as too short in content, non-HTML format like .pdf, .ps and .doc. Wang constructed each query with 5 terms by implementing TF, DF, TFIDF, PW, TF3DF2, TF4DF1, TFIDF3DF2, TFIDF4DF1, WordRank, WordRank3DF2 and WordRank4DF1. Including the unique returned by SE, top 1 and top 10, the average success rate among 1337 pages are generally from 40% - 60%, for Google, except WordRank3Df2 which is a little higher than 60%. Meanwhile, the results from MSN show poor performance in TF, which is lower than 30%, TFIDF is lower than 40%, the others are between 40% and 60%.

Figure2.15 Retrieval performance of LS from Google search ^[11]

Figure2.16 Retrieval performance of LS from MSN live search ^[11]

In the summarization, Wang concluded that DF was the best method for uniquely identifying the desired documents; TF was easy to compute and did not need to be updated unless documents were modified; TFIDF and the hybrid method combining TFIDF and DF were good candidates for extracting the desired documents ^[11]. By computing the average cosine similarity values of top 10 returned pages with the desired page, WordRank based methods such as WordRank3DF2 are best for retrieving highly relevant documents ^[11].

Word-rank is one implementation of weighted graph ranking algorithm including undirected weighted (on edges) graph and directed weighted (on edges) graph when a single word/term is considered as a vertex and all content is a graph. A window size parameter ‘w’ is introduced for implementing connection among vertices. In undirected weighted graph, each word has connection with other words only in the window size distance, including previous w words and following w words. In directed weighted graph, each word has connection with the following words only in the window size distance. Take Figure2.14 as an example and set window size to 2, ‘packing’ has connections with ‘ferocious’, ‘storm’, ‘freezing’ and ‘rain’ in undirected weighted graph, while it only has connections with ‘freezing’ and ‘rain’ in directed weighted graph. The score associated with each vertex is set to an initial value of 1 and ranking algorithm, 2-6 for undirected weighted graph and 2-7 for directed weighted graph, is run on graph repeatedly until it converges – usually for 20-30 iterations, at a threshold of 0.0001 ^[9].

Figure2.14

The expected end result for this application is a set of words or phrases that are representative for a natural language text. The terms to be ranked are therefore sequences of one or more lexical units extracted from text, and these represent the vertices that are added to the text graph. If more than 1 term happened to be neighbors, they can be connected as a key phrase. Thus, the language consistency in the content is preserved. Rada and Paul in their paper “TextRank: Bring Order into Texts” gave a clear view of this passage “Compatibility of systems of linear constraints over the set of natural numbers. Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types systems and systems of mixed types.” and extracted the following terms as results: “linear constraints”, “linear Diophantine equations”, “natural numbers”, “nonstrict inequations”, “strict inequations” and “uper bounds” ^[9].

In the previous sections, all the methods of extracting LSs disregard the natural language’s consistency in the web pages by only considering the discrete terms. The document’s terms are arranged in alphabetic order before applying TF or DF, which totally destroys the linguistic information in web pages. Take an example from Thomas A. Phelps and Robert Wilensk’s paper ^[3],

http://www.cs.berkeley.edu/˜wilensky/NLP.html cannot be located using this query “texttiling wilensky disambiguation subtopic iago” in Google at current time, as shown in Figure2.8, Google only return 4 results with highly related content but none of them have the same URL as required, however it was claimed as successful in January, 2008. Meanwhile it can be returned by Yahoo search with the same query but a different address like this in Figure2.9:

http://www.eecs.berkeley.edu/Faculty/Homepages/wilensky.html/NLP.html?lexical-signature=texttiling+wilensky+disambiguation+subtopic+iago

This shows the fact that 2 different URLs can open the same page. About different URLs binding on the same web page, it is not discussed in this project. It is supposed to be a successful retrieval if document similarity is taken as a measurement, however, the URL matching measurement will clearly put it into a false retrieval.

Figure2.8

Figure2.9

From Figure2.8 and Figure2.9, we can see the un-stable performance from traditional LS generation techniques even when they were as typical samples as in papers years before. The studies on URL and its page content change have been researched before, such as Martin Klein and Michael L. Nelson studied the pages ranging from 1996 to 2007 ^[3], but they are not included in this project. One single page bound by different URL actually happens quite often, taking Binghamton University’s home page as an example, http://www.binghamton.edu/index.html and http://www2.binghamton.edu actually connect to the same page. In chapter 3, section 3.4 shows typical examples in Yahoo news pages that Yahoo changes the same news page’s URL all the time.

One approach of solving such kind of difficulty, which mainly focuses on finding relative or similar web pages rather than locating web pages only by URLs matching, is referenced by automatically generating key words and summarizations for academic papers. They are introduced as having capabilities by reserving both the underlying language information and relatively stable performance, because there are fewer chances to have two documents with the same content but different URLs, and even this happens, Martin and Michael studied the graph-based algorithms and concluded that they actually had the ability to improve the relative/similar web page re-finding/re-location when the original copy is lost ^[3].

Graph-based ranking algorithm is a way of deciding the importance of a vertex within a graph, by taking all text as global information and recursively computing from the entire graph rather than relying only on local vertex-specific information ^[9][10].The basic idea implemented by a graph-based ranking model is a vertex can receive and cast “voting” or “recommendation” to the others ^[12][13]. When one vertex links to another one, it is basically casting a vote to the other in the graph. The higher the number of votes is received by a vertex, the higher the importance of the vertex is taken ^[9]. Figure2.10 (a) is an example showing how it works when a vertex casts all its weight to the other vertices. Figure2.10 (b) is an example a vertex casts 90% of its weight to the others while it keeps 10%. Page-Rank is a typical implementation of this graph-based ranking algorithm. The score of a vertex V_i is defined as:

 ^[11] 2-5

Where d is a damping factor that can be set between 0 and 1. In 2-5, j is the vertex points to i and Out(V_j) is the score delivered from j to i.

(a)                                                                      (b)

Figure2.10

Meanwhile, graph-based ranking algorithm can also be split into 2 groups as Figure2.11 shows the combinations: weighted (on edges) graph and un-weighted (on edges) graph, undirected-graph and directed-graph. One group’s condition can be combined with the other group’s condition.

Figure2.11

Because undirected un-weighted (on both edges and vertices) graph does not have actual meaning in this project, it is not discussed. Figure2.10 (a) and (b) are 2 examples of directed graph, with weights on vertex, but without value on the edges. Figure2.12 is an example of undirected weighted (on edges) graph. It is the case that assuming out-degree of a vertex is equal to the in-degree of the vertex, take undirected edges like bi-directions edges ^[9][10].^. The weight from i to j and from j to i are same. The weight’s recursive computation formula:

  ^[9] 2-6

 and  show V_i, V_j and V_k are connected but cannot show any direction among V_i, V_j and V_k. Figure2.13 is an example of directed weighted graph. It is the case that a weight is added according to the direction from one vertex to another. The weight from i to j is w_ij, but the weight from j to i is 0.

Figure2.12

Figure2.13

  ^[9] 2-7

Compared to the un-directed weighted graph’s formula 2-6,  and  in formula 2-7 show the direction between V_j, V_i and V_k, V_j.

In 1997, Michal Cutler proposed a method that makes use of structures and hyperlinks of HTML documents to improve the effectiveness of retrieving HTML documents ^[6]. She classified the HTML into categories based on HTML’s tags, such as Title, H1, H2, H3, H4, H5, H6 and so on, and claimed that the terms in different HTML tags have different weight. Based on this idea, a new method for extracting lexical signatures from a web page can use the terms that have the highest weights that are computed with the HTML tag structures taken into consideration ^[6].

It is quite necessary to outline Cutler’s two papers both: “Using the Structure of HTML Documents to Improve Retrieval” ^[6] and “A New Study on Using HTML Structures to Improve Retrieval” ^[7].

First of all, she raised an excellent idea of differentiating the term weights for the different HTML tags. The first paper classified an HTML page into following categories in Table2.1. The detailed specifications and functions of each tag are not listed here in this section. She also mentioned that the tag importance is Anchor > H1 – H2 > H3 – H6 > Strong > Title > Plain Text ^[6].

Table2.1 ^[6]

The second paper classified an HTML page into following categories in Table2.2. The later paper combined all the header tags together but split the strong tags into 2 categories: list and strong. Meanwhile, the second paper considered the text in Title tag and Header tag to be more important than the others rather than Anchor and Header tags are the 2 most important categories in Table2.1 ^[6]

The tags <dl>, <ol> and <ul>’s functions are listed in Appendix A.

Table2.2 ^[6]

The basic ideas behind the two papers’ categories are the same: split the text into different classes based on their tags and then associate them with different weights. When a term appears in more than one class, it only counts terms which appear in higher level. For example, <H1><A href=”http//www.binghamton.edu”>university</A><H1>, ‘university’ is classified into Header category rather than Anchor directory according to Table2.2 ^[6], but it is in Anchor category according to Table2.1 ^[6].

Figure2.5 is a snapshot from http://research.binghamton.edu/. The text in the squares is either in Strong tag or Anchor tag, they are highlighted with either in bigger font size or different color rather than regular black. Apparently, it is consistent with the author’s intention that he/she wants people to notice these lines which should draw more attention to the highlighted content and have more weight than the other un-highlighted text.

Figure2.5

However, difficulties come along with applying different weight to different HTML tags. Take the following piece of HTML as an example, in Figure2.6, which is from Yahoo news page:

Figure2.6

Take a careful look at the red square and orange square, “Mario left a comment: Obama’s ….”, is separated into 2 different parts, the terms in blue are in Anchor tag which have HREF links to the other pages, while, ‘left a comment’ in orange square is taken off from the Anchor tag, and clearly showed in a Strong text style as compared to “to see what your Connections are…”. However, Yahoo put ‘left a comment’ into a pre-defined <P> tag and set it into a Strong style. This can lead the conventional ways in parsing HTML becoming inaccurate and destroy the original order in the text. As Figure2.7 shows, the <P> tags and <A> tags are mixed together, which can lead to confusion in differentiating the text in those 2 kinds of tags if the program is not designed carefully.

Figure2.7

On the other hand, because these 2 papers focused on their test search engine WEBOR ^[7] which was developed by Weiyi Meng and Michal Cutler, Culter’s theory and research were apparently going on with clearly understanding of the working mechanism in WEBOR. Meanwhile, Cutler also had the access to control and modify WEBOR itself according to the requirement of changing CIV ^[6][7].

‘Robust hyperlinks’ is a typical implementation from applying LSs in locating web pages. URL combined with LSs can not only re-find the desired web page, but also discover the most relevant pages if the desired page is missed or lost. Thomas A. Phelps and Robert Wilensky in their “Robust Hyperlinks Cost Just Five Words Each” ^[2] exhibited the problem when the desired page was deleted, renamed, moved, or changed, and demonstrated a novel approach to this issue by argumenting LSs in URLs so that they themselves became robust hyperlinks ^[2]. A novel compatible with traditional URL called “robust hyper link aware” URL can be like:

http://www.something.dom/a/b/c?lexical-signature="w1+w2+w3+w4+w5"

where w1, w2, w3, w4 and w5 are 5 terms extracted from the original page by TF-IDF. They are probably the first ones who raised the idea of lexical signature on a typical web page and explore its the application value.

Researchers have spent a lot of efforts in exploring how many LSs can give a best result. Martin Klein and Michael L. Nelson conclude 5 to 7 LSs are good enough in robust hyperlinks ^[2] after extensive experiments. Martin and Michael did not only conclude LS is a small set of terms derived from a document which can capture the “aboutness” of that document ^[3], but also defined a LS from a web page can discover the page at a different URL as well as to find relevant pages on internet ^[3]. Through their experiments on huge amount of web pages from 1996 – 2007 which were downloaded from Internet Archive, http://www.archive.org/index.php, they claimed that 5-, 6- and 7-term LSs performed the best in returning the interested URLs among the top 10 from Google, Yahoo, MSN live, Internet Archive, European Archive, CiteSeer and NSDL ^[3]. By apply equation 2-1 to 2-2, the LS score versus number of terms in each query were derived in Figure2.4.

Figure2.4 LS Performance by Number of Terms ^[3]

Their experiments also showed that 50% URLs are returned as the top1 result, and 30% URLs were failed to re-locate/find by choosing LS in decreasing TF-IDF order ^[3] when they were reviewing Phelps and Wilensky’s research. Meanwhile, they also carefully studied the techniques for estimating IDF values which is a non-trivial issue in generating LS for the web pages. In their recent paper, 2008, “A comparison of techniques for estimating IDF values to generate lexical signatures for the web” ^[19], they introduced 3 quite different ways to estimate terms’ IDF and carefully examined their performances.

1.         Local universe which was a set of pages downloaded from 98 websites, starting from 1996 to September, 2007 in each month ^[19].

2.         Screen scraping Google web interface which was generated in January, 2008 ^[19].

3.         Google N-Gram (NG) which was distributed in 2006 ^[19].

They compared these 3 IDF estimation techniques and claimed that local universe based data as well as the screen scraping based data is similar compared to their baseline, Google N-Gram based data.

Besides listing the detail percentage of success and fail to retrieve a URL, they used the following 2 equations in paper ^[3] to evaluate the score of LSs: fair score and optimistic score.

  ^[3] 2-1

  ^[3] 2-2

R(i) shows the ith page’s rank returned by SE after sending the query, when it gets bigger value, the fair score will be lower, N is the total sample pages in their experiments which is 98 and is the average value.

  ^[3] 2-3

  ^[3] 2-4

In the optimistic score equation, S_opt is different from S_fair which is only determined by pages’ rank. is the average fair score value.

1.       Weiyi Meng, and Hai He.  Data Search Engine. In Encyclopedia of Computer Science and Engineering (Benjamin Wah, ed.), John Wiley & Sons, pp.826-834, January 2009.

2.       Thomas A. Phelps, Robert Wilensky, 2000. Robust Hyperlinks Cost Just Five Words Each. Technical Report: CSD-00-1091. Publisher: University of California at Berkeley.

3.       Martin Klein, Michael L. Nelson. 2008. Revisiting Lexical Signatures to (Re-)Discover Web Pages. Proceedings of the 12th European conference on Research and Advanced Technology for Digital Libraries Pages: 371 – 382

4.       Seung-Taek, David M. Pennock, C. Lee Giles, Robert Krovetz, 2002. Analysis of Lexical Signatures for Finding Lost or Related Documents SIGIR' 02, August 11-15, 2002, Tampere, Finland.

5.       Seung-Taek, David M. Pennock, C. Lee Giles, Robert Krovetz, 2004. Analysis of Lexical Signatures for Improving Information Persistence on the World Wide Web. ACM Transactions on Information Systems, Vol. 22, No. 4, October 2004, Pages 540–572.

6.       M. Cutler, Y. Shih, W. Meng. 1997. Using the Structure of HTML Documents to Improve Retrieval. USENIX Symposium on Internet Technologies and Systems.

7.       M. Cutler, H. Deng, S. S. Maniccam, W. Meng. Tools with Artificial Intelligence, 1999. A new study on using HTML structures to improve retrieval. Proceedings. 11th IEEE International Conference on Volume , Issue , 1999 Page(s):406 – 409.

8.       J. Lu, Y. Shih, W. Meng and M. Cutler. 1996. Web-based Search Tool for Organization Retrieval. http://nexus.data.binghamton.edu/~yungming/webor.html

9.       Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring Order into Texts. Proceedings of EMNLP 2004, pages 404–411, Barcelona, Spain.

10.    Rada Mihalcea. 2004. Graph-based Ranking Algorithms for Sentence Extraction, Applied to Text Summarization. In Proceedings of the 20th International Conference on Computational Linguistics (COLING 2004), Geneva, Switzerland.

11.    Xiaojun Wang, Jianwu Yang. 2006. WordRank-Based Lexical Signatures for Finding Lost or Related Web Pages. APWeb 2006, LNCS 3841, pp. 843-849.

12.    Larry Page. 1998. The PageRank Citation Ranking: Bringing Order to the Web. Computer Networks and ISND Systems.

13.    Jon M. Kleinberg. 1999. Authoritative Sources in a HyperLinked Environment. Journal of the ACM, 46(5): 604-632.

14.    WordNet, http://wordnet.princeton.edu/

15.    C.Y.Lin and E.H.Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of Human Language Technology Conference (HLT-NAACL 2003), Edmonton, Canada, May.

16.    Weiyi Meng, Clement Yu, and King-Lup Liu. 2002. Building Efficient and Effective Metasearch Engines. ACM Computing Surveys, Vol. 34, No. 1, March 2002, pp.48-89.

17.    Weiyi Meng, Zonghuan Wu, Clement Yu, and Zhuogang Li. 2001. A Highly-Scalable and Effective Method for Metasearch. ACM Transactions on Information Systems 19(3), pp.310-335, July 2001.

18.    Michael K. Bergman. 2001. White Paper: The Deep Web: Surfacing Hidden Value. BrightPlanet. Ann Arbor, MI: Scholarly Publishing Office, University of Michigan, University Library vol. 7, no. 1, August, 2001

19.    Martin Klein, Michael L. Nelson. 2008. A comparison of techniques for estimating IDF values to generate lexical signatures for the web. Workshop on Web Information and Data Management. Proceeding of the 10th ACM workshop on Web information and data management. Napa Valley, California, USA. SESSION: System issues. Pages 39-46.

20.    Crunch, http://www.psl.cs.columbia.edu/crunch/

Before referencing the related studies and works, the terminology “Lexical Signature” (LS) is quite necessary to be mentioned first. LS is simply considered as an equivalent term to “key words/terms/phrases” in chapter 1. There are many related works have given LS various descriptions. Thomas A. Phelps and Robert Wilensky made this definition: a relatively small set of such terms can effectively discriminate a given document from all the others in a large collection ^[2]. They also proposed a way to create LS that meets the desired criteria which is selecting the first few terms of the document that have the highest "term frequency-inverse document frequency" (TF-IDF) values ^[2]. Martin Klein and Michael L. Nelson introduced the LS as a small set of terms derived from a document that capture the “aboutness” of itself ^[3]. S. T. Park studied and analyzed Phelps and Wilensky’s theory, and he claimed that, LS had following characteristics by concluding from Phelps and Wilensky’s paper ^[4][5]:

(1) LSs should extract the desired document and only that document ^[5].

(2) LSs should be robust enough to find documents that have been slightly modified ^[5].

(3) New LSs should have minimal overlap with existing LSs ^[5].

(4) LSs should have minimal search engine dependency ^[5].

Seung-Park also raised his own perspective about LS to help the user finding similar or relevant documents:

(1) LSs should easily extract the desired document. When a search engine returns more than one document, the desired document should be the top-ranked documents ^[5].

(2) LSs should be useful enough to find relevant information when the precise documents being searched for are lost ^[5].

After all, S. T. Park’s studies on LS are very insightful and helpful in this project. If type “Lexical Signature” as a search query into Google, then the first 10 results are most likely going to have both of his 2 papers “Analysis of lexical signatures for finding lost or related documents” ^[4] and “Analysis of lexical signatures for improving information persistence on the www” ^[5].

S. T. Park conducted a large amount of experiments with TF, DF, TFIDF, PW, TF3DF2, TF4DF1, TFIDF3DF2, TFIDF4DF1 separately and combined them synthetically ^[4][5], then, compared the results from Yahoo, MSN and AltaVista all in histograms. Including unique result, 1^st-rank result and top 10 results ^[5], the success re-finding rate is more than 60% but less than 70% when take both 2 URLs match and 2 documents’ cosine value > 0.95 as a success re-finding into consideration. Thus, if only taking 2 URLs comparison as a measurement and having a success when they are matched, the success re-finding/re-locating rate would be probably lower.

Figure2.1

Figure2.2

Figure2.3 ^[5]

In this project, the LS’s definition follows S. T. Park’s theory: LSs are the key terms from the web page and can help to both identify the web page from others uniquely and retrieve the most relevant page effectively by search engines. Meanwhile, in experiments, LS cannot be simply considered as the unchanged terms (words) from the documents. Some necessary pre-procedures and transformations must be taken before starting to process the web pages/documents in the information retrieval ways, such as removing the stop words or transforming the words in different forms but close meanings into one unique term, like “lexica” and “lexical” to “lex”. Other than this, picking out only nouns and verbs or nouns and adjectives from the text is also feasible based on word form data base. These steps are implemented in Chapter 4 particularly by LUCENE and WORDNET, 2 open source Java projects well accepted in practical industry world.