For those who aren't using the dynamic publishing feature of Movable Type (see below), experiencing slow rebuild times is not an infrequent complaint especially as the size or complexity of the blog and its templates grows. While there are very good reasons to rebuild, there are no good reasons for it to be painfully slow.
In this essay, we will detail a number of reasons why your rebuild times might be slow and steps you can take to drastically reduce or completely eliminate the time you spend waiting for Movable Type to finish its work.
Upgrade to Movable Type 3
Has anything significant happened to you in the 16 months? Probably so. Well, the same can be said for Movable Type. A lot of work has gone into making the static rendering engine in Movable Type much, much faster than what those of you using v2.x are used to.
That's not to mention all of the other goodies that MT 3 includes such as comment moderation, much more powerful plugins, a new and speedy interface, commenter authentication and a smarter, more pleasant and more secure application.
If you haven't upgraded, here's reason #2,819.
Use a SQL database
While BerkeleyDB is built to be a fast and efficient database for reads, it's not so good on writes and many older versions installed on hosting services are prone to corruption. To see an immediate speed boost (and gain a number of new features to boot), convert your database to one of the SQL options supported by Movable Type: MySQL, SQLite or PostgreSQL.
Although this is listed second in importance in terms of speed, it is always better to convert your database to a SQL DB first and then upgrade to the latest version. To convert your database, simple add (in addition to DataSource) your SQL database information in mt-config.cgi (before MT 3.2:
mt-db-pass.cgi) and run the
Movable Type 3 also includes a feature called "background tasks" which allows certain tasks to go on in the background on your server returning control to you immediately after they are started.
One of the biggest differences you'll notice with background tasks turned on is in saving an entry or rebuilding entries. Once you click the save button on a new or edited entry, the entry is saved to the database, the rebuild is kicked off in the background and then the page reloads. There's no need to wait as all of your templates and archives finish rebuilding. You get to go on with your day letting the server do the work it already knows how to do!
Before turning on background tasks, you should first test to see if your server supports them. There is a script in the main distribution called
mt-testbg.cgi. Run that script through your web browser. If you see two numbers and they are different, background tasks are supported.
To enable background tasks, go into your mt-config.cgi and change:
Then go save an entry and see what I'm talking about. It's a dream...
Each plugin that you add to your system adds at least a little bit of overhead to system. In most cases, this doesn't cause a problem, and in fact, this site is published using a number of plugins. However, all it takes is one poorly-written plugin to make molasses look speedy.
If your rebuilding suddenly got slower, think about any plugins you may have just added and get rid of them. If you installed a few of them, do the following:
- Uninstall them all, rebuild and time the process
- Install one, reubuild and time it
- Repeat the process for each new plugin
By doing this, you may find your one inefficient plugin. If you haven't added any plugins lately, then move onto the other steps.
If you are regenerating the same
MTArchiveList data in multiple templates, it will compound the amount of time it takes to publish a single entry. It's better to generate your archive list once in an index template and then include it via PHP or SSI in all the places you need to.
Unnecessary rebuilding of index templates
While most of your index template need to be rebuilt to reflect the latest content you post, there are some that do not. For example, those which don't have any MT template tags in them (e.g. a CSS stylesheet or static files that don't change with your MT data) should be configured to not rebuild with index templates. Then, when you make a change to any of those templates, you simply rebuild it after saving.
Unused template maps
Template maps are mappings between groupings of entries and particular blog templates. Look under Blog config » Archive Files and scroll to the bottom. There you will see all of your template maps. They are a little understood and highly powerful feature, so let's take just a second to detail this for the curious.
Movable Type comes with one template map for Indivdual and another for Monthly archives enabled. What this means is that when Movable Type goes to rebuild, it does so once for each month's worth of entries you have using the "Date-based Archive" template creating a single file for each month. In addition, it does so once for each entry (a grouping of one) using the "Individual archive" template creating a single file for each entry.
Some people enable category archiving which, again, uses the "Category Archive" template to create a single file for each category containing the entries in that category. Usually, this should be no problem. (A word on that in a second...)
While most people just fiddle with the checkboxes on the left, enabling and disabling different archive types, there is something far more cool and more powerful that you can do with Movable Type: Output the same content in many different formats.
For example, you could have two Individual archive mappings, one using the regular Individual archive template and another using an individual news feed template including updates for comments and TrackBacks. Other cool uses include publishing individual entries in PDF format (for essays?), ICS format (for an event blog?). Uses are limited only by the imagination.
It's sad really that one of the most powerful features of Movable Type is so poorly understood and presented. We intend to fix this problem in the future.
Anyhow, the thing to do here is to make sure that you only enable the template maps you care about. You can tell which ones are enabled by looking at the checkboxes on the right. Uncheck any you don't want and click Save.
Wasteful use of template maps
While template maps are ridiculously powerful, each one taxes the system during a rebuild. If you are using multiple template maps to output extremely similar content (or worse the exact same data) in multiple formats, you might benefit from some streamlining.
The worst case we've seen was a user who had hundreds of categories, tens of thousands of entries and was generating category archives in HTML, RSS 1.0, RSS 2.0 and Atom format. Ouch! That's nearly the same exact information being compiled and written out over and over and over again.
If you're doing this, you may want to consider having Movable Type output the information you need once and then include it, via PHP or SSI, into each format. Conversely, you could just skip to the next step.
Yes, Virginia, Movable Type 3.1x does have dynamic publishing. And it rocks! By switching some or all of your templates to dynamic publishing, you can drastically cut down or eliminate the time it takes to get an entry published or to rebuild your whole blog.
When thinking about moving to dynamic templates you must consider a few points:
Your blog's traffic - Dynamic generation of pages isn't a panacea no matter what the buzzmeisters might say. There's a reason that Yahoo has historically written all of their directory pages out to the filesystem every night (this may not be true anymore). In short, each hit to a dynamic page taxes your system just a bit and takes some CPU time. With dynamic publishing, you offload the task of building a single file to each and every person who views that file. On a small scale, this is not a big deal, but if you've got a huge amount of traffic and aren't set up for it, you will bring your server to its knees. What's more, static pages don't require that your database be up and working for your visitors to read your words.
Plugins - The Movable Type publishing system prior to MT 3.1 uses Perl to publish your blog. MT 3.1 introduced a parallel and identical PHP subsystem to publish dynamic pages. Similarly, pre-3.1x plugins were written in Perl whereas and are not called by the PHP subsystem.
Both of these issues can be resolved in most non-extreme cases.
First, Movable Type provides a caching functionality which saves all of the work of compiling the page (after the first hit) and only costs a single database connection upon each page view in order to grab the appropriate cache file. That will take care of all but the sites with the highest traffic.
Secondly, there's no reason to go whole hog. There is a balance to strike and, luckily, Movable Type gives you the ultimate flexibility to pick and choose what you want to be dynamic and what should be static.
For example, enable dynamic publishing for the archive type on your site with the least traffic and leave the rest static. Each file that doesn't have to be rebuilt is time saved. As you get more comfortable with dynamic publishing, you can move more of your templates over.
As far as plugins, nearly all of the most popular MT plugins have been updated to work with both or have analogs in PHP. Still, there is a chance that that one long-time favorite plugin of yours does not. Check first and be aware. And if you happen to write a PHP analog for it, let us know!
A few dynamic tips
There are a good number of lessons that many of us have learned in switching over to dynamic publishing (other than "My Lord, that's fast.) Below we will detail a number of these tips and time-savers.
Duplicate your template
Switching to dynamic publishing can be onerous for those of us with years of customizations built into our templates because the dynamic publishing subsystem does things slightly differently than the static rendering system. If you go full-bore into editing a particular template, you may find that you've broken that set of archives and your vistors will be staring at a PHP error instead of reading your words.
The way to avoid this problem is to make a backup of your template and work on it, while leaving your statically rendered template and archvies in place.
If you're making an index template dynamic, the way to do that is obvious, but if you are doing an entire archive type, it may not be.
Remember template maps? There was a reason for detailing all of that above.
First of all, let's assume you're moving your individual archives to dynamic publishing. In this case, you want to create another archive template. Call it "Individual Archive template - dynamic" or something like that.
Then go to Blog config » Archive Files and create a new template map (Template/Archive type association) between Individual Archive Type and your new template.
After that, customize the Archive File Template in the bottom section to be:
<$MTEntryDate format="%Y/%m"$>/<$MTEntryBasename dirify="1">-2.php
Make sure to enable the template map by checking the checkbox to the left of listing. Keep the "Preferred" radio button of the original archive selected for now and save changes.
What this will do is create a parallel page for each of your static individual entries. You may then have, for example:
- /2005/05/09/my_entry.php (static)
- /2005/05/09/my_entry-2.php (dynamic)
Now you can work on the dynamic pages without affecting your static ones!
Your last rebuild
Once you switch to dynamic publishing, you need to do one last rebuild of that template or archive type in order for the dynamic publishing to take effect. This last rebuild moves all of the static files previously built on your system to "FILENAME.static". Once those static files are gone, the special htaccess file that Movable Type creates in the process notices and pulls the content directly from the database.
If some of your pages don't seem to be dynamic, make sure that MT was able to remove the corresponding static file.
MT Tags in PHP blocks
In the past, you could do all types of interesting things by including PHP blocks in your template along with MT template tags to feed in relevant data. In Movable Type's dynamic publishing, this is no longer possible due to the use of the Smarty templating system.
However, you can still include the data from MT template tags in PHP blocks without using the template tag themselves using MT PHP functions or even Smarty itself.
For a simple example, instead of doing this:
<?php $weblog_name = '<$MTBlogName encode_php="q"$>'; ?>
You do the following:
<?php $weblog_name = $this->tag('MTBlogName'); ?>
There's a whole lot more where this came from. See the following for everything you always wanted to know about MT's dynamic publishing feature but didn't know to ask:
- Movable Type User Manual: Dynamic Publishing
- Dynamic Publishing: PHP Architecture Overview
- PHP Dynamic Publishing: Developing Plugins
Finally, thanks to use of the Smarty templating system by Movable Type, you can use every single Smarty plugin and any Smarty template construct inside of your templates as well. It can even replace MT plugins without you ever having to learn a lick of Perl or PHP!
That is raw power, folks.
BIG pages and pagination
One thing you may notice with dynamic individual archives is that if there are a large number of comments on a particular entry (greater than, say, 200-300), the entry page may time out with an error as the server reaps PHP processes which take longer than 30 seconds to execute. The same thing could happen on a category archive page where you have hundreds of entries. In both cases, you need to use pagination.
Luckily, MT developer Chad Everett boiled down Smarty pagination really, really well in these two entries:
With Smarty pagination, you can have virtually an unlimited number of comments or TrackBacks to an entry without a hitch.
Movable Type doesn't have to take long to rebuild it's pages or even any time at all. If you've followed the steps above, you should find that you are waiting around watching that little popup a hell of a lot less, giving you more time to reload Blogdex or find interesting things on Google Maps. Or maybe even blogging about it.