Recently in Movable Type Category

Just upgraded to the latest Movable Type. No pain, no muss, no fuss.

I've changed the HTML of my HTML header and archive pages so that the monthly archives and category archives do not display expanded. Instead, they are presented contracted and can be expanded with a click on the respective headers. It bugged me that the pages for these were so long when the archives are not necessarily used.

Update 2009-03-27: I did the same thing for Recent Comments and Recent Entries. Here are the required edits:

1. Widgets
   • Recent Comments widget
   • Recent Entries widget
   • Monthly Archives widget
   • Category Archives widget

   Changes in bold, metanames in italics.

       <a href="javascript:animatedcollapse.toggle('WidgetId')">
       <h3 class="widget-header">Monthly Archives</h3>
       </a>
       <div id="WidgetId" class="widget-content">
   

2. HTML Head template

   Add javascript to the HEAD section of the page. 'divs' matches the names given to each of the div elements in the previous step.

   <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
   <script type="text/javascript" src="/mt-static/js/animatedcollapse.js">
   /***********************************************
   * Animated Collapsible DIV v2.2- (c) Dynamic Drive DHTML code library (www.dynamicdrive.com)
   * This notice MUST stay intact for legal use
   * Visit Dynamic Drive at http://www.dynamicdrive.com/ for this script and 100s more
   ***********************************************/
   </script>
   
   <script type="text/javascript">
   divs = [ 'Categories', 'MonthlyArchives', 'RecentEntries', 'RecentComments' ]
   for (x in divs) { animatedcollapse.addDiv(divs[x]) }
   animatedcollapse.init()
   
   $(document).ready(function() {
       for (x in divs) { animatedcollapse.toggle(divs[x]) }
   });
   </script>
   

3. animatedcollapse.js

   Copy this file from dynamicdrive.com to /mt-static/js/animatedcollapse.js

So if you are reading this, then you have been successfully redirected to my new website on hostmysite.com. Congratulations!

The whole adventure was not without some complications and extra steps. Partly, it was the change of internet host. In addition, I was changing from Movable Type 4.21 to MT 4.23, from Microsoft IIS to Apache, and from Windows to Linux. All of these factors contributed.

• mt-config.cgi

  A lot of the problems flowed from the new mt-config.cgi file. Every Movable Type site has a mt-config.cgi file which describes where the CGI/Perl files can be found, where the generated static HTML files should go, and what the IP/URL of the website is. I was previously using MT under Microsoft IIS, and it allowed certain configuration errors that will haunt me.

  MT is designed to allow multiple blogs to share a single domain. Typically, your blog is rooted at http://www.domain.com/blogname/index.html. In my case, I had mine rooted at http://www.domain.com/index.html because I put the MT code and generated files into a directory hierarchy that omitted the blog name. Unfortunately, that is really not possible in a typical Linux installation, as far as I can tell.

  Linux sites set up (interpreted, executable) CGI files under /cgi-bin/ and (static, uninterpreted) HTML files under /htdocs/. Thus, you end up with your mt-static directory moved under htdocs and everything else put under cgi-bin.

  Everything would still be cool if either:

  1. I omitted the directory with the blog name (putting everything in the htdocs root) or
  2. I could find a way to configure mt-config.cgi to redirect to index.html without the blog name.
  Neither seemed desirable, so I now have iwebthereforeiam stuck between my domain and all pages.

• Hard-coded iwebthereforeiam.com / missing blog name

  Because of the change above, I could not have internal links that omitted the blog name. At a minimum, I had to change all these internal links. All of them used my domain name, so for simplicity, I just replaced the domain name with ../... That will work even if I someday discover a simple way to omit the blog name from URLs.

• Move static HTML/ASP files

  Because of laziness, I had placed many files into the root of my configuration. In most cases, it was easier to just put them into a new directory /htdocs/files/. In like fashion, I ended up adding some other directories to the root so that I did not go insane patching my entire site:

  • /files/
  • /images/
  • /projects/
  • /sounds/
  • /styles/
• Fix Movable Type templates

  I have a bunch of CSS and JavaScript packages on my site that have to be added to parts of the MT templates in order to flow through directly:

  • google-analytics
  • code coloring with prettify.js and prettify.css
  • snapshots from snap.com
  • blockquote.quote

  I would have followed the instructions I posted some months ago for this, but MT 4.23 has removed some templates and added others, so I ended up figuring it out from scratch again. The gist of it is that anywhere that MT generates a head section, I needed to include the CSS and JS files, and anywhere that MT generates a body section, I needed to add an onload setting. The color coding involved adding the CSS and JavaScript inclusions to HTML Head:

  <link rel="stylesheet" href="/mt-static/css/prettify.css" type="text/css" />
  <script type="text/javascript" src="/mt-static/js/prettify.js"> </script>
  

  and then adding onload="prettyPrint();" to the body element of these templates:

  • Search results
  • Comment Preview
  • Category Entry Listing
  • Entry
  • Monthly Entry Listing
  • Page

  Setting up google analytics and snap.com involved making modifications to the bottom of the template Banner Footer. I still have not got blockquote.quote in extra.css working.

• Add banned sites using mt_ipbanlist

  I have a huge collection of banned spammers from my previous site. I was not going to let them start spamming my trackbacks again, so I have to insert all these IP addresses into the MySQL table mt_ipbanlist. This will be a bit of a pain. I'll probably write a python script that does the insertions and upload the script to MySQL and let it chew on that for awhile. I imagine that must be how I did it before.

• Rewrite ASP pages

  SInce I was under Microsoft IIS for so long, I have old ASP pages that will have to be rewritten, probably in PHP. I'll have to evaluate why they had to be ASP (i.e. interpreted/dynamic ) in the first place.

• Redirect name servers

  This was the final step in throwing the switch. Once I did this, web surfers could find my new site with the old domain name. I had to add in an HTML file with a redirect to deal with the absence of the blog name:

  <html>
  <head>
  <meta HTTP-EQUIV="REFRESH" 
  content="3; url=http://www.iwebthereforeiam.com/iwebthereforeiam/index.html">
  </head>
  <body><p>Redirecting to <a href="http://www.iwebthereforeiam.com/iwebthereforeiam/index.html">new blog location</a> in 3 seconds.</p></body>
  </html>
  

• Alert the search engines

  Because of the addition of the blog name to the middle of the URL, most all URLs in the search engines are now bad. Sigh. So I have to resubmit my site for spidering.

Update 2009-01-09: Unbelievable. The whole problem with the blockquote.quote style in extra.css was that the file began with a C++-style comment and the entire file was interpreted as if it was commented out. Some sort of Unix versus Windows/LF versus CR-LF issue. Totally resolved by taking out the comment.

This is not my idea of good performance. This is 1400 entries. It used to take about 10-20 minutes. What's going on?

Update 2008-10-30: Here is just the entries. Yes, 144 minutes for the 1400 entries in this weblog. That's 10 per minute. I thought Movable Type 4.21 was supposed to be faster?

Update 2008-10-31: In an effort to assist my website host, webhosting4life.com, in isolating the problem< I republished my website this afternoon. It took a really long time: 251 minutes.

I've provided a number of ideas for them to explore:

It appears that the queries are all causing table scans. I've looked at the phpmyadmin system variables for MySQL and they are consistent with large numbers of table scans due to missing keys in searches.

If no activity was found on my database, then perhaps the MySQL activity is from another user. These MySQL settings are both database and global settings:

• Slow_queries
• Handler_read_rnd
• Handler_read_rnd_next
• Select_full_join
• Select_range_check

All of these values are exceptional and all suggest queries that are not properly hitting indexes, causing table scans. Again, I recommend looking at the slow queries log.

Right now, 10% of the queries to the database my website uses have taken more than long_query_time seconds. That seems a really high ratio.

Update 2008-10-31: Another personal best: 19 hours, 47 minutes.

I upgraded to 4.21 from 4.1 in the past day. I have tried to republish my weblog with a new style, but the republish times are horrific (1400 entries across 4 years):

• ??? seconds for entry (ran for about 80 minutes before stalling on the completion screen)
• 2353 seconds for author monthly
• 1335 seconds for category monthly
• 843 seconds for monthly
• 161 seconds for category archives
• 73 seconds for index templates
• 16 seconds for page

That's 80 minutes without entries. For comparison, it used to take only 15 to 20 minutes to republish the entire site before.

I've looked at the phpmyadmin system variables for MySQL and they are consistent with large numbers of slow queries, probably caused by table scans due to missing indexes in searches:

• Slow_queries has crept up from 526k to 574k slow queries.
• Handler_read_rnd from 4641k to 5341k
• Handler_read_rnd_next from 955M to 1116M
• Select_full_join from 9373 to 10k

However, my weblog is hosted (on Windows under IIS6, no less) and I don't have access to any raw MySQL logs. I can't run MySQL from a command prompt. I have no way of telling which queries are causing the slow down. Also, I cannot upgrade the perl version.

Ideas for discovering the problem:

• instrument the perl code to record SQL queries & time to execute
  ...except I don't know what perl code executes the SQL queries. Plus, the code goes directly to the DBI/DBD perl module to execute SQL. It would mean patching the code in a zillion places.
• add indexes to MT tables in MySQL
  But indexes on which columns of which tables?
• add foreign keys to MT tables in MySQL
  Again: where?

Anyone with an idea is welcome to leave a comment.

Probably a big mistake. I can't see any published pages anymore. If you are reading this, though, then I've solved the problem.

I ended up installing a lot of Perl packages to get things working:

• version
• Text::Balance
• IO::Compress::Zlib
• IO::Compress::Base
• Compress::Raw::Zlib

Wow. Now publishing my website (say, after a style change) takes hours.

I am getting really tired of the trackback spam. I was looking around in my MySQL administrator and found the tables that Movable Type uses to track tentative trackbacks (mt_tbping) and banned IP addresses (mt_ipbanlist). I wrote up a little SQL to copy IP addresses from unapproved trackback locations to the banned list:

INSERT INTO mt_ipbanlist
(ipbanlist_blog_id, ipbanlist_ip)
SELECT MY_BLOG_ID, tbping_ip
FROM mt_tbping
WHERE tbping_ip <> 'MY_HOST_IP_ADDRESS'
AND tbping_ip NOT IN (
	SELECT ipbanlist_ip 
	FROM mt_ipbanlist
	WHERE ipbanlist_blog_id = MY_BLOG_ID)

Of course, I'll have to check that all unapproved trackbacks are spam first, but I am hoping that the volume drops in time.

2008-08-13: added my list of banned IP addresses, trackback spammers one and all.

Thanks to several google-translated Japanese Movable Type blog entries [makotokw, kazuno, and tadateto], I've finally figured out how to hack MT to get python code pretty-printed. (To read these Japanese articles, I recommend searching using google and then pressing Translate this page.) The code is downloaded from google and the original idea was from Thomas Guest.

To get this to work, you have to do four things:

1. Add google's prettify.css stylesheet to your header page template
   

   <link rel="stylesheet" href="/mt-static/css/prettify.css" type="text/css" />

2. Add google's prettify.js JavaScript to you header page template
   

   <script type="text/javascript" src="/mt-static/js/prettify.js"> </script>

3. Get your body.onload to call prettyPrint()
   

   <body class="mt-main-index layout-wtt" onload="prettyPrint();">

4. Mark the sections that you want pretty-printed
   

   <pre class="prettyprint">

The third step is the most involved. You have to change three templates:

1. Page
   

   <MTSetVar name="body_onload" value="prettyPrint();">
   <MTIfCommentsAccepted>
       <$MTSetVar name="body_onload" value="prettyPrint();individualArchivesOnLoad(commenter_name)"$>
   </MTIfCommentsAccepted>

2. Entry
   

   <MTSetVar name="body_onload" value="prettyPrint();">
   <MTIfCommentsAccepted><
       $MTSetVar name="body_onload" value="prettyPrint();individualArchivesOnLoad(commenter_name)"$>
   </MTIfCommentsAccepted>
   

3. Comment Preview
   

   <MTSetVar name="body_onload" value="prettyPrint();individualArchivesOnLoad(commenter_name)">
   

2008-06-27: Did I say three templates? I meant five. Here are two more templates to change:

1. Search results
   

   <MTSetVar name="body_onload" value="prettyPrint();">
   

2. Archive index
   

   <MTSetVar name="body_onload" value="prettyPrint();">
   

If you are using Movable Type and using WebHosting4Life.com as your ISP, you will not be able to get comments and trackbacks to work until you turn off the WebHosting4Life setting, Security Guard. If, whenever you post a comment on your blog, you get this error message:

Webhost4life Application Firewall Alert

Your request triggered an alert! If you feel that you have received this page in error, please contact the administrator of this web site.

Then you need to do this:

1. Login to WebHosting4Life.com
2. Go to Security
3. Select Security Guard
4. Set to OFF

It took me three weeks of googling this before I put in the system request and got this fixed. I hope you find this posting sooner.

• reCaptcha is listed as a plugin
• reCaptcha is selected as my CAPTCHA provider
• Comments are set to Immediate approve comments from: Anyone
• Registration is set to Anonymous comments with Require E-mail Address for Anonymous Comments set
• Web Services setting has a Type Key value obtained from SixApart
• Entire website has been rebuilt
• I've followed all the instructions on Adding reCAPTCHA to Movable Type