Maintenance
02.07.2006
TypePad Maintenance Update
We recently completed two scheduled events and we'd like to update you on the results.
Saturday's scheduled maintenance went well. This database server upgrade will help us to scale our operations and deliver more reliable performance.
Secondly, we finished our data center migration today, retiring a previous permanent IP address and a temporary IP address for TypePad's published weblogs.
Should you find that your weblog is unavailable at your custom domain, but is available at your TypePad URL, it's possible that your blog is one of the few (<100) affected by the migration.
You can find additional information on how to correct the issue in our Knowledge Base article: Domain Mapping Affected by Upcoming IP Address Changes. You can also enter a Help Ticket by going to your Control Panel > Help > New Ticket and our Support Staff will be happy to provide assistance.
Posted by Technical Support at 09:33 PM | Permalink
12.19.2005
Recap of Friday's TypePad Outage
Now that we've restored user's photos and files, and the application has been reliably back online for three days, we want to provide more details about our outage on Friday and the steps we took to recover from it. We will post more information in the coming days about our plans to avoid future outages and compensate customers for Friday's downtime.
Before we get into the details, we want to reiterate just how sorry we are for the inconvenience this has caused. We know our customers rely on us to provide superior service and performance, and that on Friday we let you down. The fact that Friday's outage came on the heels of our performance issues in October is obviously frustrating, both for you and for us.
What happened
As you may know from our previous posts to Everything TypePad, we have been migrating our operations to a new state-of-the-art data center. Over the past two months we've nearly completed that move, and as part of it made upgrades to our networking equipment, our bandwidth capacity, our application hardware and our storage systems. The maintenance window we took on Thursday night was designed to accomplish two things: configuring and testing high-performance and redundant network traffic load balancers, and configuring a recently added redundant "head" in our new, high capacity disk storage system.
The first task on Thursday night went off without a hitch. The second task, however, was where the problem started. As part of the configuration of the new storage system, we needed to reboot the storage device. When we attempted to bring the device back online at 10:50 pm PST, a hardware failure occurred and damaged the index of the file system. Essentially, the system couldn't mount the drive, even though the data was still there.
This disk failure led to both published blogs and the TypePad application being down. After diagnosing the problem in order to understand the severity of the issue, we decided to serve published blogs from a snapshot that was between 2-6 days old. This was why some TypePad blogs were out of date.
Recovering TypePad and restoring your data
Through the night and into Friday, our operations engineers worked to diagnose the problem with our storage vendor, and to bring the application back online.
We knew that when we brought TypePad back up user's posts, comments and TrackBacks, TypeLists and photo albums would be current in the application's database, but out of sync with their published blog. When we restored the application at approximately 3:00 pm on Friday, we encouraged users who were logging in to republish their weblogs to bring them up to date, and at the same time began a process to proactively republish user's content on their site. We advised users that republishing would resolve some issues, but not those related to photos or files that they had uploaded to TypePad in the past several days.
On Saturday morning we completed our process of republishing user's weblogs. Also on Saturday morning, we worked with our storage vendor to bring the data with the damaged file index back online in a new unit that they rush delivered to our data center. Once we were able to bring that machine up, we began the process of restoring user's missing photos and files to their weblogs and photo albums. Additionally, over the course of Saturday night and Sunday, we did more republishing of weblogs, photo albums and TypeLists to restore data and to address particular customer issues.
As of Sunday night at approximately 10:30 pm, we completed the data restoration process. We encourage any of our customers who are having any unresolved or related issues to file a help ticket with our support team, and they'll work hard to get your problems resolved. You can file a help ticket by visiting the Help tab under the Control Panel inside TypePad.
We hope that this post provides a bit more detail about what happened on Friday and over the weekend. Again, we are very sorry for the inconvenience, and appreciate your continued patience. We will post more information in the coming days about our plans to avoid future outages and compensate customers for Friday's downtime.
Posted by Michael Sippey at 04:38 PM | Permalink
04.08.2005
Maintenance this Weekend
It's time for some upgrades!
TypePad will be unavailable due to scheduled maintenance for four hours starting Saturday, April 9 at 10:00 pm Pacific Daylight Time. During that time, weblogs will be available for reading and viewing, but you will not be able to log in to the TypePad application, and visitors will not be able to comment on your weblogs.
We'll be upgrading our database servers as part of our ongoing efforts to improve TypePad's performance; we apologize for any inconvenience.
Update: We'll be posting status updates about the maintenance activities to status.sixapart.com.
Update: We've successfully completed our maintenance activities, and TypePad came back online on Sunday at 1:00 a.m. PDT. Thanks for your patience!
Posted by Michael Sippey at 03:09 PM | Permalink
01.20.2005
Bursting at the Seams!
This coming weekend we will be conducting the final stages of a much needed cage move in our data center, as we have completely outgrown our space. We are not moving our machines far, but we are moving them. Thus, while the move is taking place (on January 22, 2005 at 11:00pm PST) the TypePad application may be unavailable, and access to published weblog content will be intermittent. We have scheduled this service outage during a time in which traffic is at its lowest to minimize the impact to our users and their readers, but we would still like to apologize for any inconvenience this may cause.
Update: the move is complete. It took a little longer than we expected, but the system was brought back up at around 2:30am and is now fully located in the new cage. Thanks for your patience!
Posted by Byrne Reese at 04:53 PM | Permalink
01.13.2005
Temporary Service Degradation - January 13, 2005
This morning between the hours of 12:00 am and 8:30 am Pacific time on January 13, 2005, the TypePad application suffered from a temporary service impairment. The problem occurred during a scheduled maintenance window while performing a system upgrade. During this time access to the TypePad application was severely impaired due to load issues associated with our database. While the ability for people to post comments to TypePad weblogs was hindered, published weblogs were not affected in any other way. The problem has since been recovered from, and steps are being taken to assure that the problem does not happen again.
The TypePad team, and all of us here at Six Apart would like to apologize for any inconvenience this may have caused anyone.
Posted by Byrne Reese at 12:45 PM | Permalink
09.24.2004
Scheduled Downtime September 25, 2004
TypePad will be unavailable due to scheduled maintenance for 4 hours starting Saturday, September 25, 2004 at 11:00pm Pacific time. During this time, your weblog will still be available for reading and viewing, but you won't be able to log in to TypePad to post, and visitors will not be able to comment on weblogs.
Posted by Mena Trott at 09:23 AM | Permalink
09.03.2004
Stats Maintenance
During the course of some routine maintenance of our stats database last night (because stats are constantly being tracked on our servers, the stats database requires routine maintenance to keep it running quickly), we discovered some issues with one of our database tables that needed to be repaired. This caused about 8 hours of unavailability for viewing stats within the application; during this period, TypePad itself was fully available for posting, commenting, etc., but statistics (in Control Panel > Stats) were disabled.
We were able to recover all of the historical data older than 2 days, and all of the data received within that 9-hour period, which means that the statistics view within the application only shows statistics back for about the last 9 hours (as of about 9:30am Pacific time). We're sorry for the inconvenience, and thank you for your patience.
Posted by Mena Trott at 09:50 AM | Permalink
