Deploying Magnify Search Crawler

Magnify Search Crawler is packaged in the following directory of your WebFOCUS installation:

drive:\ibi\WebFOCUS82\utilities\magnify\crawler

The crawler folder contains all of the source code and dependent third-party code in the form of various .jar files. This folder also contains the following group of configuration files:

• index-name-mapping.txt
• log4j.properties
• meta-tag-mapping.txt
• regex-urlfilter.txt
• regex-url-to-category-mapping.txt

A batch file named sample_run.bat is provided, which is a batch script to invoke the Crawler program.

Edit the sample_run.bat file to match your deployment environment and preferences. For example:

The main Crawler program is contained within magnify_crawler.jar (com.ibi.applications.search.magnify.crawler.MagnifyCrawler), which takes two required arguments and one optional argument:

• The first argument (required) is the Starting URL, which is the entry point to the website to be crawled.
• The second argument (required) is the Magnify server URL, which can be either local or remote.
• The third argument (optional) represents the crawler Thread pool size. If this argument is missing, then a default value of 1 is applied, which indicates that the Crawler will process web pages in a linear process (one at a time). In most cases, you would want to specify a value that allows the Crawler to run more efficiently using multiple threads. Keep in mind that available system memory and CPU speed impacts the performance as well.