Fast Search Plugin - Faster Searching for Movable Type

One of the great benefits of the Movable Type developer community is the commitment its own ProNet members have to improving Movable Type not just for themselves, but for all MT users. Their contributions are what make Movable Type so great. No better example exists then that of Mark Carey, who recently built a plugin that improves the performance of searching against a MySQL database in Movable Type. We asked if he wouldn't mind writing an article for us that explain how this plugin works. What follows is a description of Fast Search in Mark's own words.

Fast Search is a PHP plugin for Movable Type that provides fast searching of Movable Type entries with low CPU load. The plugin is based on a solution authored by Iñaki, that was posted on the Movable Type support forums. With Iñaki's permission, I modified the solution, added some features, and packaged it up as plugin.

How does Fast Search work?

At a high level Fast Search uses the PHP-based dynamic publishing system of Movable Type to search for entries using a MySQL fulltext index. There are two basic components to this approach:

• Database queries against a MySQL fulltext index
• Using MT templates and PHP dynamic publishing to display the search results

MySQL Fulltext Queries

While I am not a database or MySQL expert, I have learned that MySQL has a feature call fulltext indexing. Essentially, MySQL will create a search index of all the text in a database table. It does so in such a way that makes it much faster to search the index than to search through every row of the database. Fulltext searching include relevancy algorithms, boolean searching, and excludes very short words and words on a defined 'stopwords' list. Fulltext indices provide fast searching with some intelligence and optimization built-in. Indices are updated and managed completely by MySQL, so web applications don't need to worry about maintaining the index.

To implement fulltext searching for Movable Type, the first step is to create a fulltext index on the database. Fast Search creates an fulltext index on the mt_entry table, indexing only the entry_title, entry_text (body), entry_text_more (extended entry), and entry_keywords fields. The MySQL command to create the index is:

ALTER TABLE 'mt_entry' ADD FULLTEXT 'fastsearch' (
   'entry_title' ,
   'entry_text' ,
   'entry_text_more',
   'entry_keywords'
 )

In an effort to make Fast Search easy to install, the plugin will automatically add the index, so you don't have issue the command via the command line or MySQL management tool. If you are running Movable Type 3.3, the index is added as part of an upgrade task, which hooks into the MT upgrade process (mt-upgrade.cgi) -- taking advanatge of a new feature in MT 3.3 that enables plugins to automatically trigger upgrade tasks. If you're a running MT 3.2, creating the index is done by a single click from the Plugin Settings area for Fast Search.

Adding a fulltext index does add to the overall size of your database, although I did not notice a significant increase in size after adding the fulltext index. Creating the index takes a bit of time, but it took less than a minute for my database, which has many entries.

Once the index has been created, it can be queried using a MATCH....AGAINST syntax. A simple example is:

SELECT * FROM mt_entry WHERE entry_blog_id='2' AND
     MATCH (entry_title, entry_text, entry_text_more, entry_keywords )
     AGAINST ('movable type')

Using variations on this basic syntax, Fast Search builds MySQL fulltext queries based on the search string and (optional) arguments. The default is to order the results by relevance score, but results can also orderd by date (most recent result first). Search results can also be restricted to a single category. All Fast Search queries include a LIMIT clause which limits the search results displayed (per page) to a defined maximum. This limit is defined by the user, but it should be kept low for performance reasons -- Fast Search may not seem so fast if it has to display 1,000 results on a page. In conjunction with this limit, Fast Search can also display additonal pages of search results -- in such cases, the MySQL query will only fetch the results for that page.

Using PHP Dynamic Publishing to Display Search Results

The MySQL fulltext queries discussed above are built and executed within the framework of Movable Type's PHP-based dynamic publishing system. There are two basic things that happen here: interfacing with the database, and displaying database content via MT templates and tags. The bulk of the PHP functionality of Fast Search is contained in the block. MTSearchResults.php file, with controls the output of the content between <MTSearchResults> and </MTSearchResults> tags (more on template tags later).

The first step of the PHP code is to determine the search query and any optional arguments that need to be considered. Fast Search looks for these in 3 places: arguments specificed as part of the

$entries = $ctx->mt->db->get_results( $sql, ARRAY_A );

Now that the plugin has fetched the results of the fulltext query, the next task is to display those using MT templates tags. By design, I tried to use the same tag names that are used with MT's built-in (CGI-based) search results. These tag names don't conflict with any existing tags because they are not used within MT's dynamically published pages (until now). By using the same template tag names, this makes it easier to convert existing search results templates for use with Fast Search. The results of this single fulltext query are used to determine the values of all of the template tags provided by the plugin, as well as the dynamic publishing, since that has been covered previously.

The Fast Search Template

The Fast Search template is an index template that must be be built dynamically. This template can be created manually, or you can install the 'default' template with a single click from the Plugin Settings area. By default, the search results URL is http://www.yourdomain.com/blogpath/fastsearch, but you can change this to anything (as long as you point your search forms to the correct location). The default template looks a lot like the default search template from MT 3.2, with some minor changes including additonal tags for the pagination features of Fast Search.

Logging Search Results

Similar to the built in search tool in MT, Fast Search will (optionally) log each query to the Activity Log. From an implementation perspective, this is the only area in which I needed to differentiate the code for MT 3.2 versus 3.3. Since the Log table has been changed in MT 3.3, different MySQL commands are required to insert an entry into the Log. The plugin checks the version of MT and builds the MySQL command appropriately. Logging of search queries can be enabled and disabled on a per-blog basis via the Plugin Settings.

The Fast Search Form

To complete the implementation of Fast Search, you need to point your search forms to the new Fast Search dynamic index template. The plugin has been designed such that simply changing the 'action' parameter of existing search forms is sufficient. For this reason the plugin will accept search queries with fields named "search" or "query". Of course, search forms can be be modified to include Fast Search optional arguments such as category, order and lastn (note that 'MaxResults' is recognized as synonymous with 'lastn', for compatibility with older search forms that may use this field).

Summary

Fast Search is designed to be a high-speed, high-performance search alternative for Movable Type 3.2 and 3.3. One of my goals was to provide most the core features of the exisiting CGI-based search tool, in a way that would provide a perfomance improvement for many sites, especially those with a large number of entries. Apart from the efficiencies and benefits of using a MySQL fulltext index, I made an effort to limit the number of database queries. With logging disabled, Fast Search makes only a single database query to execute a search. Combined with limiting the number of search results per page to a reasonable number, I hope this helps keep searches fast with minimal CPU load.

Fast Search is currently available from MT Hacks.