Overview

Magnify Search Crawler (referred to as the Crawler in this document) is a useful addition to Magnify Search. This feature enables you to gather content from internal or external websites that you identify and allow this content to be searchable through Magnify. The Crawler is:

• A lightweight, standalone Java program that can be deployed on any platform and can be included in batch scripts for automation or be run as a service in the background.
• Highly efficient with options for running multiple threads in parallel.
• Highly flexible with configuration options in the following areas:
  • Logging levels.
  • URL filtering.
  • Meta tag injection.
  • Index name customization based on URL patterns.
  • Category injection based on URL patterns.