Magnify Search > Magnify: Building Indexes With WebFOCUS Reports > Magnify Distributed Indexes

Magnify Distributed Indexes

Topics:

How to:

Magnify indexes can be distributed across load balanced or indexed environments, multiple WebFOCUS installs, and mounted drives or across other similar networked devices. This means an index is virtually indexed together by actual indexes residing in multiple locations, with each location being an index that makes up the greater virtual index.

To do so, a shared directory is first setup as central location to receive incoming feed documents, which are then indexed with one of the available distributed indexes. Then, when searches are performed, all distributed indexes are queried at once, and all results are merged together into a single result set for the end user.

Each index can be independently searched, but this represents only a portion of all the information that is expected to be made searchable and is only accomplished when searching all indexes in parallel. Due to this, the collection file should be configured so that an index not only references the local index, but has bidirectional pointers to all other indexes in other locations. Each of these locations is represented by their own collection file and each must be processed in their own application server space.

Requirements:

  1. In order to feed data to a shared directory, the feed to disk method must be employed. In order to index fed data to distributed indexes, each index must be run in its own application server instance.
  2. This is only used with the Magnify feedtype of FULL where unique identification is not used and duplicate records can exist.

The following procedures will walk you through how to enable and configure Magnify to concurrently index incoming feed documents across indexes.

Procedure: How to Set Up Your Network Connected Environment

  1. Confirm each machine can connect with one another.
  2. Designate one directory to store incoming feed documents. This is shared to feeding machines and those machines part of the index with read and write permissions. It is recommended to use the default WebFOCUS feedcache directory.
  3. In the WebFOCUS administration console for each WebFOCUS installation that is included, set the Magnify feed cache directory to the above shared path. On the local machine, the DOS path can be used.
  4. Create a directory to store the index in each WebFOCUS installation that is included. You must confirm that the local machine has write access, but also has shared access with read permissions to all other machines. By default, it is recommended to use the default Magnify index directory. In addition, it is recommended that a subdirectory be created within this directory to help future-organize indexes, such as the automatic index directory creation and dynamic partitioning.

    For example, the directory can be set as: magnify\lucene_index\mag. Then, the index can be created under the following directory: mag\index.

Procedure: How to Map Indexes

  1. Create a collections file for each index to be represented. Make sure to perform the following steps on each file.
  2. Edit the collections.xml file.
  3. Configure any pointers to the index locations. Doing so defines where to load indexes with data from feeds that are picked up from the shared feedcache directory. To do so:
    1. Add an index element for each index participating in the greater index.
    2. Use a unique name and make sure that the directory points to the location of the index.
      • For local locations, this is the DOS path.
      • For remote locations, this is the UNC path.
    3. Include the feed element for the datasource.
      Note:
      • There is only one feed element defined in the set of indexes.
      • The one feed element that is defined is placed within the index element for the index being represented.
  4. Group the indexes into a greater index in order to perform a search.
    Tips:
    • Create each index as a group component.
    • The group component can then be referenced like any other index through the collections drop-down menu.
  5. Create a version for each index. You can do this by either directly copying the index to any additional WebFOCUS Magnify installations, or renaming them accordingly.

Procedure: How to Configure Application Server Settings

The following settings are enabled through your WebFOCUS Client application server that is configured with your WebFOCUS environment. An example of the application server is Apache Tomcat.

  1. Force the application server to lock files from being distributed to other indexes once they are picked up for processing. To do this, add the following Java setting to your application server:
    -DIBI_Magnify_FeedCache_Locking_Enable=true
  2. Set the length of time (in minutes) that the application server waits to unlock a file to be distributed. To do this, add the following Java setting to your application server:
    -DIBI_Magnify_FeedCache_Lock_File_Lifetime_Minutes=5
  3. Set the collection file representing the index.
    This is used when using one WebFOCUS Magnify install across multiple application server instances. To do this, add the following Java setting to your application server:
    -DIBI_Magnify_Collections_File_Name=collections[N].xml
Notes:
  • When performing these steps, take the usual WAR file considerations.
  • The first search performed will be slow, as memory is being loaded.
  • This technique works across all supported operating systems.

Enable Distributed Indexes for Load Balancing

One limitation to using this technique is that WebFOCUS Magnify can still only be configured to a single collection file, meaning that the only way to get to Magnify is through one collection file. This results in mapping pointers to other collection files in order to virtually represent the large single index when distributing indexes.

This is a challenge in a load balanced environment where entry to Magnify should occur from any index or in the collection file.

To enable and configure Magnify for multi-collection entry point for bi-directional collection configuration, reference in load-balanced environment.

WebFOCUS

Feedback