Over at illuminea, we’ve been migrating WordPress sites for years. So we thought we’d seen it all when it came to WordPress migration. And yet, recently we encountered an issue that we had never dealt with before, that puts a snag in the ease with which you can move WordPress sites: data serialization.

One of the reasons we love WordPress is because it’s so future-proof. What I mean by that is that once you have created a WordPress site, you aren’t stuck where you are for the following reasons:

• Content is separate from form so at any point you can completely redesign your site while retaining content and link structure. This is important to ensure you don’t lose traffic from search engines and inbound links.
• As long as the WordPress software continues to be developed, it will continue to support modern web technologies so users are not left with an archaic system that, for example, adores IE7 but doesn’t know what HTML5 is.
• Your site is portable. You need to move a site to another server, or even another domain name? No problem. A few steps related to find-and-replace, exporting and importing the database, and setting up global 301 redirects (if you’ve changed domains) and you’re there.

All the above is true, except the data portability point. It’s not quite as true as I thought it was.

Data serialization and its impact on data portability

(I am a big newbie when it comes to data serialization, so the following is based on what I’ve managed to learn over the while and could very well be inaccurate/wrong.)

It seems that developers use data serialization in order to improve performance or efficiency in a database. Instead of creating a new table for data, or storing data in a table which could end up having wasteful empty fields, you can keep all the data in an array in one field. This leads to saved space and simplified SQL statements. Serialized data is stored and recovered using PHP’s serialize() and unserialize() functions. Terri Swallow explains how using serialized data for her multi-user plugin helped save her from creating hundreds of new options for every new user created.

Here’s an example of serialized data from Andrew Nacin:

But there are serious drawbacks to using serialized data. One problem is that you can’t run a MySQL query on serialized data. The data would have to be pulled out of the table via PHP, unserialized, and then searched using PHP instead of MySQL (thanks to Catch My Fame for explaining this). So let’s say you are storing user data in serialized format, but you want to find out how many of your users are from Australia, you can’t get that info easily.

A serious problem, in my opinion, is the impact serialization has on data portability. It makes it near-impossible to port a WordPress site from one domain to another without losing data. Another possible, yet unlikely issue, is if you need to port data from WordPress into another programming language – admittedly a rare occurrence, but a possible one that needs to be considered.

There are developers out there who are very frustrated by the use of serialized data:

An anonymous comment left on the PHP manual entry for serialize():

Please! please! please! DO NOT serialize data and place it into your database. Serialize can be used that way, but that’s missing the point of a relational database and the datatypes inherent in your database engine. Doing this makes data in your database non-portable, difficult to read, and can complicate queries. If you want your application to be portable to other languages, like let’s say you find that you want to use Java for some portion of your app that it makes sense to use Java in, serialization will become a pain in the buttocks. You should always be able to query and modify data in the database without using a third party intermediary tool to manipulate data to be inserted…That’s not to say serialize() is useless. It’s not…Just don’t abuse serialize because the next guy who comes along will have a maintenance or migration nightmare.

Alexander Scammon: “WordPress does a silly thing.  It has serialized data inside its database.”

Serialized data usage that can have a serious impact on WordPress portability:

• Some widget data (though for some reason we’ve never had issues retaining widgets via a database dump)
• Some theme settings
• Plugin data

It was related to serialized plugin data that we had our first encounter with data serialization. We had installed a plugin to help a client manage a technical issue on their site without having to get too technical. They entered hundreds of entries in the plugin, but upon moving the site from development to production, the data disappeared. By that point, there was no way to recover it unfortunately, but we had definitely learned a painful, important lesson.

So what do we do?

Some suggestions:

1. Theme and plugin developers: please seriously consider whether the benefits of serializing your data outweigh the potential future problems it could cause.
2. Theme and plugin developers: if you must use serialized data, please try to make sure there’s some way to export and import the data. Even a simple .csv export is better than nothing.
3. WordPress users: try to ascertain before using a theme or plugin whether the data is stored in a serialized format.

But let’s say you find yourself with a database that includes serialized data. Here are some tools created by generous souls that might be able to help you port your data without losing it:

Peach – an online utility that aims to deal with the fact that WordPress stores URLs in serialized format, which makes it hard to move a site from one domain to another, or from staging to production. To use this tool (which works best in Chrome) ou drag-and-drop an .sql file into the square in the middle of the page, and set a new domain.

WordPress Search and Replace Tool – this script can also handle multiply nested serializations and multi-byte Unicode changes, which is important now that internationalized domain names are allowed. The code will work for any platform that stores PHP serialized arrays in a MySQL database (Drupal, Joomla, etc.). You must delete the script from your server once you’re finished with it.

fix_serialization.php – Not much description provided with this tool, but Alex Scammon provides instructions in his comment below.

WP Migrate DB – this is a plugin that exports your database as a MySQL data dump, does a find and replace on URLs and file paths, then allows you to save it to your computer.  It even takes into account serialized data and updates the string length values. (thanks Matt Wiebe)

After trying these tools, we wrote an update on data serialization.

This post was originally published at Data portability and data serialization in WordPress on WP Garage - WP Garage - wordpress tricks, hacks, and tips.

I recently had to move all my WordPress sites and blogs to a new server because Bluehost sucks. My sites were going down on a near-daily basis for way too long, and they couldn’t care less. So I did what I really did not want to do, and moved six WordPress sites to a new hosting provider, which has proven itself with my clients’ sites: Hostgator. And along the way I’ve learned quite a few things about what to pay attention to when moving WordPress sites from one place to another, which I’d like to share.

Lesson 1: WordPress’ native XML export is not good enough

In WordPress, you can easily export a site by going to Manage > Export, which will create a unique XML file that you can import in any new site. This is fine for a very simple, single-user blog without heavy customizations. However, if your blog has multiple users, and if you have added info to their profiles, that info will be lost in this export. In addition, all users will be given a permission level of Subscriber, which is annoying if you need to go and change everyone back to their original permission level.

Also, if you have installed plugins that created new tables in your database, these will not be exported.

And, if you had any links saved, this will not save your links. See Lesson 6 below for more information on this, but the point is that the only way to save your links AND their categories is to export your entire database.

Therefore, in order to retain all settings and information that your site needs, the only way to do so is to export your entire database, and then import it to the new WordPress installation.

Exporting and Importing a WordPress blog

Here’s how you export and import your WordPress database. WARNING: Messing with databases in PHPMyAdmin can be hazardous to your health and to the health of your site. One wrong move and everything will be but a distant memory. So make sure to backup like crazy; and I take no responsibility for any harm you may come to by following my instructions. Now, let us begin:

1. Log in to PHPMyAdmin via your control panel or however else you log in.
2. If you have multiple WordPress blogs on one server, you may see a list of databases in the left-hand sidebar. Make sure you are dealing with the right database. The way that I check which database is the right one is by browsing the database and viewing the information there, whether it’s posts or even meta data like the blog name. To browse, click on the browse icon next to a table’s name (Note: your PHPMyAdmin interface may look a bit different, depending on the version you are using):
   
3. Once you are sure you are in the right database, click on Export at the top of the PHPMyAdmin toolbar:
   
4. On the Export screen, I usually leave all the settings as is, but I do make sure to to 2 exports: a compressed (gzip) export, and a regular export, just in case. So export one while selecting None, as shown below, and one while selecting gzipped:
   
5. Now on to importing: install WordPress on your new server.
6. Go in to PHPMyAdmin for your new server, and delete all your tables for your new WordPress installation (Yikes!). In PHPMyAdmin, to delete is called to “drop.” So on your main database page, select all your tables and click on Drop, or on the red Xs next to the table names:
   
7. Now, import your exported WordPress database into your new installation. To do that, stay in PHPMyAdmin for your new site, and click on the Import tab that is to the right of the Export tab on the toolbar (see image in step 3 above).
8. Browse to find your exported database file. Leave the rest of the settings, and click Go.
9. Now your old WordPress database is in your new installation. It may work at this point, so go check. If it does, great. If not, continue reading.
10. You need to make sure that your database table names work with the new installation of WordPress. This means the following: if you installed your old database with Fantastico, all your database tables have a prefix of wp_. If you didn’t install it with Fantastico, the database tables probably don’t have a prefix. Your new database tables may have a different naming convention depending on how you installed it. So, take a look at your database table names, and note if they start with wp_ or not. Now it is time to open wp-config.php.
11. Once you have opened wp-config.php, look at line 16, which says something like this:
    $table_prefix = ‘wp_’;
    If your database tables start with the wp_ prefix, leave the line as above. If they don’t have a prefix, change it to:

$table_prefix = ”;

Now your blog should mostly work (I hope)! But there are a few more things that we need to take care of:

Lesson 2: Widgets are not preserved with an XML backup, even with a full database backup

If you are using widgets somewhere in your blog, you will have to redo them one by one on your new blog if you use the XML backup. So take out a pencil and paper, and write down exactly what widgets you are using, what order they’re in, and any modifications or customizations that you made to them. Ok, you can use Word or Notepad if you want, but that’s about as technologically friendly as you can get.

Lesson 3: In addition to backing up your theme and plugins folder, remember your Uploads folder!

In your blog under Settings > Miscellaneous, you have set where any uploaded files or images will be stored. Make sure to backup that folder, in addition to your theme and plugins folder, so that you will have all those files in your new blog. In your new blog, go to Settings > Miscellaneous and make sure it’s pointing to that re-uploaded Uploads folder.

Lesson 4: Make a note of your site’s permalink structure

You want to retain the same permalink structure for your new site so that you can continue to enjoy the same traffic you had before from search engines and incoming links. So make note of your permalink structure, and once your new blog is installed, make sure to go to Settings > Permalinks, and set the same structure there.

Lesson 5: Don’t forget your favicon

Your favicon can get lost in the shuffle. Make sure to back it up, and upload it to your new site.

Lesson 6: It is hard to preserve WordPress blogroll links and their categories

Here at WordPress Garage, I had a page until recently called “Links we Love.” I had painstakingly added, categorized and described many links that I thought were useful to my readers. Well, during the whole server move, the links got lost because we didn’t export them by going to wp-links-opml.php, and saving that OPML file as a text file for importing later. In any case, even if I had saved that and imported it here, I would have lost all my categories anyway. So just note that if you’re depending on the XML backup, you will lose all your links. If you back up your links with the OPML file, you will lose your categories.

——————-

To sum up: try not to ever have to move your WordPress blog from one server to another, and you will add years on to your life.

Happy blogging!

This post was originally published at 6 things to remember when moving WordPress blogs to a different host on WP Garage - WP Garage - wordpress tricks, hacks, and tips.

WordPress blogs and sites can be notoriously slow. But fear not – here are ways to make your WordPress blog super speedy and fun for your visitors to view with a few tweaks, hacks and plugins. Be sure to check back because I will be updating this post as I discover new and wonderful ways to optimize and speed up WordPress blogs.

[This post is in honor of the current Pessach (Passover) season, a Jewish holiday where we clean our houses frantically in the hope that not one crumb of leavened bread (i.e. regular bread) be found during the 7 day holiday. This post is the WordPress version of Pessach cleaning, where cluttered databases are equivalent to leavened bread, etc. Happy Pessach!]

Define goals and create benchmarks:

1. First, define a goal, such as reducing page load time from 8 seconds to 2 seconds.
2. Measure your initial state and the results of each modification so that you can quantify any improvement. Test your site’s speed with the Website Speed Test, but do multiple tests since the results can be inaccurate due to fluctuations in your internet connection and other factors.
3. Use Pingdom to get a detailed analysis of your blog’s loading time and performance.
4. See what your browser is doing with tools like Firebug’s network tool, Charles Proxy or Wireshark, and review the server logs.
5. YSlow – analyzes web pages and tells you why they’re slow based on the rules for high performance web sites. YSlow is a Firefox add-on integrated with Firebug. See this presentation from Yahoo! that covers their latest research results and performance breakthroughs. It covers their existing 14 rules, plus 20 new rules for faster web pages. They’ve categorized the optimizations into: server, content, cookie, JavaScript, CSS, images, and mobile.

    

   | View | Upload your own

Reduce the number of dynamic PHP and http calls:

1. “There is an inherent overhead in each HTTP request. It takes substantially less time to serve one 30K file than it does three 10K files.” So combine all files in a type into a library. Learn how here.
2. Use different host names to increase the number of active download threads.
3. Minimize PHP and database queries – Each time a page on your site loads, if your browser has to execute any PHP queries, it adds to the load time. If you replace the PHP queries with static HTML, every time a page loads, your browser just reads the HTML. An example from WP Candy:
   With PHP requests: <title><?php bloginfo(’name’); ?><?php bloginfo(’description’); ?></title>
   Without PHP requests: <title>WPCandy - The Best of WordPress</title>
Joost de Valk says that you can remove 11 queries to the database by doing the following in your header.php and footer.php files:

   • making your stylesheet URL’s static
   • making your pingback URL static
   • making your feed URLs static
   • removing the blog’s WordPress version
   • making your blog’s name and tagline / description staticSee more examples of how you can replace code in your WordPress template files with static HTML here and here.
4. Check if you have too many external calls to things like Amazon, eBay, MyBlogLog, etc. Try commenting them out one by one to see if it speeds things up.

Optimize your files: CSS, HTML, Javascript, images, video

1. Optimize your image files for the web.
2. Make sure that all images have height and width tags.
3. Consider hosting your images on an external site like flickr that has huge servers and can handle the load.
4. Use CSS sprites for static web images. CSS sprites are where the images are added to one larger image file, and laid out in a convenient way. Here’s a CSS Sprites generator.
5. Do not host videos on your server. Upload them to YouTube, Google Video, or any other video sharing site and let them handle the server load.
6. Compress your Javascript, using a tool or by removing formatting (and potentially by shortening function and variable names). This can reduce file size by 60%. Add gzip compression to that as well and you’re looking at a serious size reduction.
7. Compress HTML and CSS by removing HTML formatting, white space (where you divide code among separate lines for easier readability), trimming class names, omitting unambiguous quotes around attributes, etc.
8. Compress your CSS with the CSS Compress WordPress plugin – Automatically removes comments, new lines, tabs, and gzip compresses (GZIP) any CSS file called with “<?php bloginfo(‘stylesheet_url’); ?>” Just activating the plugin with the default Kubrick theme will reduce the CSS file from 8k to 1.7k.
9. Compress your CSS by using shorthand CSS. Here’s an example from WP Candy:
   Long: .test {margin-top: 7px; margin-right: 1px; margin-bottom: 5px; margin-left: 3px;}
Short: .test {margin: 7px 1px 5px 3px;}
10. Use external scripts – Instead of placing tons of code in your header.php file, use external scripts. This allows the browser to cache the script so it won’t have to read it for every other page.
11. Validate your code at W3C to make sure you don’t have any major errors slowing down your page.
12. Allow progressive rendering: Load CSS files at the top of the page, from within the head section; load JavaScript files at the bottom of the HTML. And/Or…
13. Stop slow loading scripts from breaking your blog with IFrameWidgets v1.0 WordPress plugin. Slow widgets or snippets of Javascript can either time-out, or prevent the items below them from loading. The plugin creates WordPress sidebar widgets that run in an IFrame. Since IFrames load in parallel to the rest of the page, slow loading JavaScript widgets won’t affect the rest of the page.

Plugins

1. Disable or delete unused plugins – some plugins have tons of script and code, and even create database tables in your WordPress database. Use only the plugins you really need, and delete the rest.
2. Sometimes plugins require that you add a snippet of code to your theme’s template files to call the plugin. Usually, it looks something like this:
   < ?php refer_plugin(); ?>
   However, if for some reason you disable that plugin, you will get an error. Joost de Valk recommends using PHP’s special function called function_exists to prevent the blog from breaking if plugins are disabled or removed. Using it will make the code look like this:
   < ?php if (function_exists(‘refer_thanks’)) { refer_thanks(); } ?>
3. Control when your WordPress plugins are loaded: WordPress processes all of the code for all active plugins, even if that plugin isn’t used on a particular page. If a particular resource heavy plugin isn’t used on certain pages, then you can tell WordPress not to load it on those pages by wrapping an if statement around the content of each function to check what page is being loaded. Learn more about how to do this here and here.

Database

1. Use phpMyAdmin to optimize your database: Log in to phpMyAdmin, select all the tables, and then “repair” and “optimize.”
2. Delete excess records in your WordPress database. All plugins use the wp_options table to store data, which is the same table used by WordPress to store all settings for your blog, and is accessed every time you open any page. When you deactivate a plugin, these records are left behind, bloating your database. To clean it up you can use the WordPress Clean Options Plugin, which finds orphaned options left after you have removed plugins and removes them from the wp_options table, or manually as follows: Back up your database, login to phpMyAdmin, open your blog’s database, and click on browser for the wp_options table. Go through this table record by record to identify any records left behind by old plugins. (from WordPress Web 2.0 Spot-Er).
3. Use the Optimize DB plugin to optimize the tables of your database.
4. Use WordPress Plugin: Fix Database to check all tables in your database and fix any errors.
5. Lester “GaMerZ” Chan’s WP-DBManager 2.11 plugin sorts your database backup files by date in descending order, can repair databases, and allows automatic scheduling of database backups and optimization.

Caching and protecting for server overload

1. WordPress has a built-in caching system. Learn how to enable the default WordPress object cache. This has apparently been disabled in version 2.5.
   
2. WP-Cache 2 – caches Worpress pages and stores them in a static file for serving future requests directly from the file rather than loading and compiling the whole PHP code and then building the page from the database.
3. WP Super Cache – This plugin is a fork of the WP-Cache 2 plugin, and generates html files which are served without ever invoking a single line of PHP.
4. PHP Speedy – PHP Speedy is a script that you can install on your web server to automatically speed up the download time of your web pages.
5. Use the Expires and cache-control max age headers for all pages; Make dynamic pages support the if-modified-since request header; Use far future expiry headers on static resources; Use the cacheability engine to test that you have caching and validation set up correctly. If you don’t know what all this means, don’t worry, neither do I, but you can find out more here.
6. Digg Protector plugin – The Digg Protector will determine if a visitor is from Digg, and if the visitor is indeed from Digg, the plugin will serve them a remotely-hosted version of the image. Otherwise, the plugin will serve the locally-hosted (on that server) image.
7. Some more caching possibilities: MySQL query cache, PHP Compiler Cache. Learn more here.
8. Configure Apache for maximum performance.

Happy Optimization!

Sources:

Speed up your website: Part One

How-to: Optimize your site for speed – really excellent article with practical tips

How to Enable the Default WordPress Object Cache – talks about wp-cache, wp-cache 2, and built in wordpress caching.

Digg Protector

4 Simple Ways To Speed Up WordPress

Speed up and clean up your WordPress!

5 Tips to Help Your Slow or Sluggish Blog or Web Site (WordPress Especially)

Deleting excess records in your WordPress database

WordPress on Speed: 17 Tweaks to Accelerate your WP

The 3 Easiest Ways to Speed Up WordPress

WordPress Theme Hacks

Diggproof & Speed up Your WordPress Blog

WordPress Optimisation: Control When Plugins Are Loaded

This post was originally published at 38 ways to optimize and speed up your WordPress blog on WP Garage - WP Garage - wordpress tricks, hacks, and tips.

I recently had to migrate an entire blog from one installation to another. WordPress does have an export function, and there are many database backup plugins and managers, but these really only work well with a basic installation that doesn’t use any plugins that created tables. So, in my case, I couldn’t find anything that enabled me to move the Simple Tagging tags from one blog to another – easily.

Why am I still using Simple Tagging, you may ask, if shiny new WordPress 2.3 has tags built-in?

The answer, my friends, is that most of my blogs are still running on 2.2.x, and that is because I try to install all my WordPress sites with Fantastico, and Fantastico has still not upgraded to 2.3. But once I’ve got my Simple Tagging tags in my new blog, they can be automatically integrated into 2.3′s tagging system once I upgrade.

I finally figured out how to move the whole blog, tags and all, using phpMyAdmin, so here are the directions in case anyone else may find they need them. These directions can be applied to migrating any plugin databases to an entirely new installation. Before you do any of this, backup ALL database files, and pray that nothing gets screwed up:

1. Log in to phpMyAdmin for the old site. You can do this through your Control Panel, or I guess through complicated server commands. I’m a Control Panel gal myself.
2. Click on the drop-down menu on the left and select the right database, or click on Databases in the main window, and then click on the database. You will now see a list of all the tables in this database.
3. Click on the Export tab at the top of the page.
4. There is an Export box at the top right-hand side of the main window. It lists all available tables in the database. Select the following by holding down Ctrl and clicking: wp_categories, wp_comments, wp_link2cat, wp_links, wp_post2cat, wp_post2tag (this is a Simple Tagging (ST) table), wp_postmeta, wp_posts, wp_stp_tags (ST table), wp_tags (ST table), wp_tag_synonyms (ST table), wp_usermet, wp_users. If you’re doing this for other plugin tables, just ignore the tables above that say (ST table) after them, and look for unique databases that are not in the list above (except wp_options – see below to find out why).
5. Leave all the other options as they are, click on Save as file, leave the File name template as is, Compression can be None or zipped or gzipped – if your databases are really big, you might want to choose zip or gzipped so that the download (and later the upload), won’t take so long.
6. Click Go.
7. Save this file somewhere where you can find it easily.
8. Now, go to the other database in phpMyAdmin. Depending on how your sites are set up, you may be able to access the database where you are right now, or you may need to login to phpMyAdmin somehwere else.
9. Once you have clicked on the correct database, delete all the tables that have the same name as the ones listed above in step 4. The reason for this is that you are going to be importing the tables from the old blog, and when phpMyAdmin sees a file with the same name, it won’t import it and will skip it. In short, phpMyAdmin does not synchronize tables. To delete the files quickly, select each table you want to delete by checking the box on the left of it, and then scroll to the bottom of the list of tables to a drop-down box that says “With selected:”. Click on that, and select Drop. That will delete all the selected tables.
10. Now, click Import. Under File to Import, browse to find the file you exported in Steps 4-6. Leave all the other settings, and press Go.

That’s it! Now hopefully your new blog works and has all the info from the old one. If it doesn’t, you’ll just restore all your backups and try again!

You may have noticed that we ignored a table in both blogs called wp_options. The reasons for this is that it seems that this file records the domain and URL for the blog. So if your old blog was at www.mydomain.com, and your new one is at www.yourdomain.com, the first domain will appear in the second blog. This can screw things up pretty royally in terms of folders and files for the blog. Believe me, I know. At one point I couldn’t even get into the Options page in the WordPress admin anymore.

I hope I didn’t miss anything in the steps, and I hope this helps!

This post was originally published at Transferring Simple Tagging tags into another WordPress blog on WP Garage - WP Garage - wordpress tricks, hacks, and tips.