semioticpixels scratch pad

I’ve been slow to adopt the automatic upgrade feature of Wordpress, but I reached my pain threshold with 20+ different plugins across 7 different Wordpress sites and used it for the 2.9.0 -> 2.9.1 upgrade. And it was great! I logged into each server, temporarily changed file ownership on all the Wordpress files to be owned by my Apache user, ran the upgrades and had a cup of coffee to celebrate.

Unfortunately, less than a week later one of those blog servers crashed under an intense load of runaway Apache processes. The server was running at 108% cpu capacity and the top 15 processes were apache. In the course of investigating that, I discovered this collation issue.

In the apache error log, the first error to confront me was this:
WordPress database error Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '=' for query SELECT comment_ID FROM wp_comments WHERE comment_post_ID = '2933' AND comment_approved != 'trash' AND .... etc

Collation conflict …. I looked at the database structure, sure enough, there’s a mix of utf8_general_ci and latin1_swedish_ci. Most of the default WordPress database tables are Latin1 and it looks like the newer tables and all of my plugins are UTF 8.

Looking at WordPress installs that I haven’t upgraded yet, all of them are set to UTF 8. But these are newer installs. The Wordpress installation that has this problem has been in place since version 1.* and Wordpress used to default to Latin 1 encoding. The UTF 8 tables are newer.

So, in other words, if you’re maintaining a WordPress install that’s been around for a while, you probably also have this issue in your database.

Now, to get this particular database fixed up …
One suggestion I came across was to export the schema and data separately then recreate the database with UTF 8 charset and reimport the data. However, that will likely break the existing content because the content has Latin 1 characters which will cause the newly minted UTF tables to choke a little.

… When converting the character sets, all TEXT (and similar) fields are converted to UTF-8, but that conversion will BREAK existing TEXT because the conversion expects the data to be in latin1, but WordPress may have stored unicode characters in a latin1 database, and as a result, data could end up as garbage after a conversion!
ref

The better way is to run an ALTER tables query. The steps are:

1. backup everything up
2. ALTER all TEXT and related fields to their binary counterparts using the SQL statements generated below
3. alter the character set
4. change the binary data type fields back to TEXT
5. Add DB_CHARSET and DB_COLLATE definitions to wp-config.php

Using MySQL’s information_schema to generate the actual ALTER tables statements needed, I wanted to pipe out the statements to a reusable text document so I could do a test run on a copy of the problem database first. Generating the ALTER tables statements using information_schema will then include whatever extra tables or rows have been added by plugins.

The following 4 queries are lifted directly from Haidong Ji via the Wordpress Codex. I added the pipe out to text file. Note that the path to mysql executable is on Ubuntu. Your path might be different, for example, another common location might be /usr/local/mysql/bin.

1. ALTER all TEXT and related string field types to binary field types counterparts using the SQL statements generated below. The list of conversions being made is as follows:

CHAR -> BINARY
VARCHAR -> VARBINARY
TINYTEXT -> TINYBLOB
TEXT -> BLOB
MEDIUMTEXT -> MEDIUMBLOB
LONGTEXT -> LONGBLOB

• echo "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'char', 'binary'),';') FROM columns WHERE table_schema = 'testblog' and data_type LIKE '%char%';" | /usr/bin/mysql -uuser -ppassword information_schema > /home/userdir/char2binary.txt
• echo "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'text', 'blob'),';') FROM columns WHERE table_schema = 'testblog' and data_type LIKE '%text%';" | /usr/bin/mysql -uuser -ppassword information_schema > /home/userdir/text2blob.txt
• echo "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', column_type, 'CHARACTER SET utf8;') FROM columns WHERE table_schema = 'testblog' and data_type LIKE '%char%';" | /usr/bin/mysql -uuser -ppassword information_schema > /home/userdir/char2utf.txt
• echo "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', column_type, 'CHARACTER SET utf8;') FROM columns WHERE table_schema = 'testblog' and data_type LIKE '%text%';" | /usr/bin/mysql -uuser -ppassword information_schema > /home/userdir/text2utf.txt

2. Change the default character set of the database from Latin 1 to UTF 8. This will ensure all new tables created are UTF 8, but doesn’t affect existing tables (which is fine since we’ve already changed those above, right?).
This can be done either using phpmyadmin (select database -> click the Operations tab -> select UTF 8 from dropdown menu at bottom) or in your shell ALTER DATABASE MyDb CHARACTER SET utf8;

3. change the binary data type fields back to TEXT
This can get sticky if your original structure had datatypes that … were supposed to be binary or blobs. I looked through my database before altering it and did not find any blobs or binaries. But this could vary depending upon plugins used, etc. Changing the binaries back to TEXT essentially involves running the above sql scripts in reverse.

• 
echo "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'binary', 'char'), ';') FROM columns WHERE table_schema = 'testblog' and data_type LIKE '%binary%';" | /usr/bin/mysql -uuser -ppassword information_schema > /home/userdir/rev-binary2char.txt

• echo "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'blob', 'text'), ';') FROM columns WHERE table_schema = 'testblog' and data_type LIKE '%blob%';" | /usr/bin/mysql -uuser -ppassword information_schema > /home/userdir/rev-blob2text.txt

4. Add DB_CHARSET and DB_COLLATE definitions to wp-config.php

references

wordpress article on changing collation
article by alex king

I have 2 old Plone 2.0 websites running on a single Zope instance that I need to:

1. migrate from it’s ancient host to a more modern server
2. upgrade to Plone 3

I’m migrating away from a 32 bit Red Hat 7(!) server to a 64 bit Ubuntu Hardy Virtual Machine. The reasons should be obvious from that statement alone, but in short: Red Hat 7 reached its end-of-life years ago. It’s stable (so far) but all system maintenance has to be done manually with no package or dependency management. There are many benefits to running within a virtualized environment, namely that I can clone it as many times as I like to develop and test other websites without having to set up a whole new server, and it simplifies our redundancy and backup process.

Anyway, this article is both a cheatsheet and an historical commentary for when I or someone else is trying to solve a problem and has a WTF moment regarding how and why things are set up the way they are.

The server environment I’m migrating away from is:
- Server: Zope/(Zope 2.7.3-0, python 2.3.3)
- ZServer/1.1
- Plone/2.0.3

I’ll also be upgrading from Apache 1.3 to Apache 2 and switching from Squid to Varnish for caching

Getting started: Migrating from Red Hat 7.3 to a Ubuntu 8 VM

Zope is an “in-place” installation, which means it runs from the directory that you put it in. This is nice and simple, makes it easier to run different versions of Zope on the same server with associated different versions of Python and Plone and, theoretically it should be easy to move to another system altogether.

So, my first pass, I rsynced the Zope directory over from the old server to the new one and tried to compile Python and Zope to see what would happen. Unfortunately, when I compiled Python, I got a bunch of errors that indicated my version of Python was optimized for a 32 bit system and … the new VM is 64 bit. E.g.

`Parser/tokenizer_pgen.o' is incompatible with i386:x86-64 output

Python 2.3.7 was the oldest version that I could get running on the new VM that didn’t throw 32 bit errors. I briefly considered re-compiling Python 2.3.3 with -m32 option to GCC and I could have spent quite a lot of time researching and very likely could have gotten 2.3.3 running on the VM but 2.3.7 was a bugfix release and the most stable of the 2.3 releases and should work fine so I chose to save time by going with it. We’ll see if I regret that later. I was also reminded in the course of this process to clean up my mess with

$ sudo make clean

When I compiled Python, the following error occurred: Can't locate Tcl/Tk libs and/or headers, so I had to install the tcl/tk libraries (added to setup below)

Plone/2.0.3 is no longer available for download, although I was able to locate a copy of Plone/2.0.5. I don’t have a development server to upgrade the website to 2.0.5 so if I have to use it, I may have to keep my fingers crossed. Since Plone is installed as a Zope Product and doesn’t require compilation, I will first try to copy over the Products directory of the website and see if it’ll just work.

I will also need to install PIL and PyXML

When I compiled Zope the first time, I got error messages that zlib was missing. It turns out that Ubuntu ships with zlib, but not zlib-dev installed. Also, when researching that error, I came across a comment that I would need to install system support (libjpeg62) for PIL so those 2 lines have been added to the VM setup below.

Initial VM setup

update and install basics


$ sudo apt-get install update
$sudo apt-get upgrade
make sure gcc is up to date
$sudo apt-get install build-essential
$sudo apt-get install zlib1g zlib1g-dev libjpeg62 libjpeg62-dev tcl tcl-dev tk tk-dev vim-full lynx


update the locate db so you can find stuff

$sudo updatedb

change root password
$sudo passwd

create a user with sudo privileges:
$ useradd -m -c "real name" -s /bin/bash chris

The flags create a home directory with skel profile defaults and this account is linked to the defaults for bash shell, which is important if you like syntax coloring and tab completion.
$passwd chris
$sudo visudo


Find the line that says #User Privilege specification and add

chris ALL=(ALL) ALL
shift ZZ to save and exit

create zope user
$useradd -m -c “zope” -s /bin/bash zope
$passwd zope

Do not add zope user to the sudoers list.

Disable root login
$sudo vi /etc/ssh/sshd_config

Change “PermitRootLogin yes” to “PermitRootLogin no”
Reload ssh
$sudo /etc/init.d/ssh reload

create directory structure

The Library directory in the below structure may look a little over-organized at first pass. However, my next project after completing this migration will be to upgrade from Plone 2.0.3 to Plone 3 which will require several interim upgrades. This version of the website is running on Zope 2.7.3. Each interim upgrade will be installed to it’s respective directory within Library/Software keeping everything well organized and easy to roll back to if necessary.

in /usr/local

/usr/local/Zope
…./Downloads: where downloads will be stored
…./src: where software will be unarchived
…./Library: where compiled software will be compiled to
………./Software
……………/Zope273
………………../Python
………………./Zope
…/Sites: where each website to be hosted lives
………/instancename1
……../instancename2


# cd /usr/local/Zope/Downloads
# wget http://www.python.org/ftp/python/2.3.7/Python-2.3.7.tgz
# wget http://www.zope.org/Products/Zope/2.7.0/Zope-2.7.0.tgz
# wget http://internap.dl.sourceforge.net/sourceforge/plone/Plone-2.0.5.tar.gz
# wget http://sourceforge.net/projects/pyxml/files/pyxml/0.8.4/PyXML-0.8.4.tar.gz
# wget http://effbot.org/downloads/Imaging-1.1.6.tar.gz

#cp * /usr/local/Zope/src/


Install Python 2.3.7

Build Python 2.3 from source with the following steps:

cd /usr/local/Zope/src/
tar -xvzf Python-2.3.7.tgz
cd Python-2.3.7
./configure --prefix=/usr/local/Zope/Library/Software/Zope273/Python
make
make install


Install PIL

More info

At the time of this writing, PIL 1.1.6 is compatible with Python 1.5.2 and up and fixes some 64 bit compatibility errors in Python 2.5 (which we’re not using yet for this Zope installation)

On Ubuntu Hardy, you have to edit setup.py before installing PIL
#sudo vi /usr/local/Zope/src/Imaging-1.1.6/setup.py

find the line
#TCL_ROOT = None

and replace it with
TCL_ROOT = "/usr/include/tk"

#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py build
#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py install


Install PyXML

cd /usr/local/Zope/src/PyXML-0.8.4/

#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py build
#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py install


Other recommended packages

elementree has been recommended, but it’s not installed on the old system so I’m not going to worry about it atm.

Also, the http://plone.org/documentation/kb/setup-from-source plone article recommends installing DocFinder as an invaluable development tool, but again, I don’t need this to get the website running so I’m going to come back to it.

install zope 2.7.3

Build Zope 2.7 from source with the following steps:

cd /usr/local/Zope/src
tar -xvzf Zope-2.7.3.tgz
cd Zope-2.7.3
./configure --with-python=/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 --prefix=/usr/local/Zope/Library/Software/Zope273/Zope
make
make install


create zope instance(s)

The zope user must have write access to create the directory. After the instance is created, edit #”effective-user zope” into the etc/zope.conf file, so if you start it as root later it should #su itself to the non-root user. Again: make install should be run as root, #mkzopeinstance.py should not.

Initially, when I tried to create an instance, it failed with the error:
# /usr/local/Zope/Library/Software/Zope273/Zope/bin/mkzopeinstance.py: /usr/local/Zope/bin/python: bad interpreter: No such file or directory


I think this was a result of not cleaning up a bad compilation. To fix this edit the first line in mkzopeinstance.py from
#!/usr/local/Zope/bin/python
To
#!/usr/local/Zope/LIbrary/Software/Zope273/Zope/bin/python

When Zope is compiled, one of the last things it does is create a symbolic link from where you told Zope the python interpreter you wanted it to use lives to Zope/bin/python. mkzopeinstance.py is looking for that symbolic link, and not looking for python itself. This provides further separation so that we could theoretically upgrade python without touching Zope and then that symbolic link would only need to be changed.

The instance needs to be created with the zope user, NOT root

# su zope
# /usr/local/Zope/Library/Software/Zope273/Zope/bin/mkzopeinstance.py
# when prompted for the path to your instance, use:
# /usr/local/Zope/Sites/instance1 .... and so forth for each site.


test Zope

For the first instance, you can test by running /usr/local/Zope/Sites/instance1/bin/runzope. OR running /usr/local/Zope/Sites/instance1/bin/zopectl start

Once loaded, this will make Zope accessible on http://localhost:8080 (unless you changed the port), with the Zope Management Interface available on http://localhost:8080/manage (obviously if you’re accessing a remote server, then localhost might not work and you need to use the IP address)

shut down zone by either hitting ctrl+c if runzope was used or by running /usr/local/Zope/Sites/instance1/bin/zopectl stop if zopectl was used to start zone.

Install the website


cp originalSite/Products/* /usr/local/Zope/Sites/instance1/Products/
cp originalSite/var/Data.fs /usr/local/Zope/Sites/instance1/var/

/usr/local/Zope/Sites/instance1/bin/zopectl start

navigate to the ZMI localhost:8080/manage
(Note that the admin login credentials will match the credentials of the site migrated, not the credentials you compiled Zope with)
The website I migrated over, even though I physically placed the files in the instance1 directory within /usr/local/Zope/Sites/instance1 are in the ZMI under “Plone”

click the Root Folder of the ZMI
check the box beside “Plone”
click the “rename” button and rename it to your instance1 site
(at this point I discovered a dependency on lynx which I’ve added to the apt-get list above)

navigate to localhost:8080/instance1 to view the website

On first pass, everything worked!

users

make sure that all the Zope files belong to the zope user
cd /usr/local
chown zope -R Zope
chgrp zope -R Zope
//////

start Zope

Start Zope with the following command:
su zope
/usr/local/Zope/Sites/dev/bin/zopectl start

check it out

Go to the http://mydomain.com:8080/manage, log in, create a new Plone site object.

resources/references

• setup from source
• plone installation

resolving tcl/tk error when installing PIL on Ubuntu Hardy

Next up:

• VirtualHostMonster and Apache with mod_rewrite & mod_proxy
• caching: Cachefu, Pound, Squid, Varnish

I’ve been slowly migrating from my desktop to my laptop so I’ve been setting it up piecemeal.

Bash (Terminal) File and Directory Colors

I do a lot of work within Terminal and not having directory and file coloring drives me nuts after a while. To add colors

cd ~/
vi .bash_profile

add

export CLICOLOR=1
export LSCOLORS=ExFxCxDxBxegedabagacad


reload it by typing (in Terminal)
source ~/.bash_profile

close out Terminal then start a new Terminal session. Do an ls and directories and files should now have color. A more in-depth explanation is listed below in “references”

References

adding file and directory colors to Terminal

Managing Paths

The problem with setting up an environment piecemeal is when you sit down to do something, you have to figure out what rabbit hole you’d gone down when you left off. In this case, I remembered installing mysql, but usually the last thing I do when I install mysql is install phpmyadmin because it’s such a handy productivity tool. Well, there was no phpmyadmin on my system, but that could be because Snow Leopard came with PHP 5.3 out of the box which is incompatible with most of the popular PHP web applications out there.

Anyhoo … so I started by checking my $PATH variable and was surprised by the output.

echo $PATH gave me duplicates of /usr/local and /opt/local et. al. and had stuff in it that wasn’t in my ~/.profile

my ~/.profile had the following in it:

export PATH=/usr/local/sbin:/usr/local/mysql/bin:$PATH
##
# Your previous /Users/username/.profile file was backed up as /Users/username/.profile.macports-saved_2009-12-29_at_11:21:58
##
# MacPorts Installer addition on 2009-12-29_at_11:21:58: adding an appropriate PATH variable for use with MacPorts.
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
# Finished adapting your PATH environment variable for use with MacPorts.


and the second entry made by MacPorts was overriding the first entry.

Pretty funky.

TheTao of Path Variables

1. /etc/profile is the default startup script for Bash, which is what I’m using. (If you’re using a different shell, then you may have a different startup script). /etc/profile calls /usr/libexec/path_helper
2. path_helper first calls /etc/path and /etc/manpath which contain the initial path environment variables. /etc/path contains system-wide defaults:
   /usr/bin
/bin
/usr/sbin
/sbin
/usr/local/bin


3. path_helper then looks for files in the directories: /etc/paths.d and /etc/manpaths.d and appends the paths found there. On my system, /etc/paths.d and /etc/manpaths.d contain a file named X11 which simply contains the paths for X11.
4. After /etc/profile has called path_helper, it then looks for /etc/.bashrc. I have a bashrc (no “.”), but my bashrc only has stuff in it specifying the bash shell prompt (name-of-my-computer:directory username$)
5. Bash next looks for ~/.bash_profile. This is the file where you’ll set file and directory colors and could also be where you place your Path environment variables. Obviously, ~/ represents your user directory, so your settings will only be valid for your user.
6. Next, bash looks for ~/.bash_login. I don’t have this on my system so it’s ignored in my case
7. next, bash looks for ~/.profile which I did have on my system and is the file that XCode wrote to.

Another file that can contain path variables is ~/.MacOSX/environment.plist. This sets environment variables, including paths, for gui applications. I’m not using it on my system so don’t have anything to say about it.

Recap

So, what that all means is that instead of exporting PATH environment variables to a .profile or .bash_profile in a user account directory, you (or your application) can, instead, make PATHs global by adding text files to the /etc/paths.d and /etc/manpaths.d directories.

If you need to control the order of a path, then try this:
Add a line PATH=”" before the call to path_helper like this in /etc/profile:

if [ -x /usr/libexec/path_helper ]; then
PATH=""
eval `/usr/libexec/path_helper -s`
fi


All that said and done … I’ll continue using ~/.bash_profile because it’s got that warm fuzzy familiarity. Personal preference, as always.

References

man page for path_helper
making use of paths.d
mastering the path_helper

Rysnc from 1 linux box to another and keep the file structure identical

Delete files from the target directory that have been deleted in the source directory
rsync -avz -e ssh --delete remoteuser@remotehost:/remote/dir/ /this/dir/

Note: the ending {/} is important if you don’t want to hose a directory.

Backup a website on a linux box to your windows machine

rsync reads “:” in a filepath as a remote directory, so if you’re trying to rsync to “c://” rsync will be confused. Instead of using windows syntax for filepaths, use the cygwin directory structure.

1. Install cygwin to windows
2. test rsync and ssh are installed by typing from within the cygwin terminal:
   rsync --version
ssh -l username somedomain
3. create a directory in c://cygwin named backups or whatever you want to call it.
4. rsync -avz -e ssh --delete remoteuser@remotehost:/remote/dir/ /cygdrive/c/directoryname Change the directory drive to whatever it should be

references

rsync over ssh

I inherited a number of web servers that had been installed and configured by different contractors and am now in the process of evaluating what needs to happen to bring them all up to snuff as well as figuring out ways to streamline regular updates. Some of those web servers are still on Red 7.3 (!), the last free open source version of Red Hat. They’re quite old in server years. Additionally I have a few other servers I set up a year ago that are more up-to-date running Ubuntu 8 LTS, but the Apache version is 2.2.11 and 2.2.8. Our security scanning service notifies me regularly that SSL v 2 must be upgraded to SSL v3, which requires an upgrade to Apache 2.2.13

Environment:

Apache 1.3.29

References:

How to Disable SSL v 2 support in Apache

On Aug 10, 2009, Apache released an upgrade that addresses a DOS vulnerability.

server:

/usr/local/apache2/bin/httpd -v
# Server version: Apache/2.2.11 (Unix)
# Server built:   Jan 15 2009 13:39:20

dev server:

/usr/sbin/apache2 -v

Issues

1. caused 401 error for all http requests (worked correctly for https connections) source
2. Seem to be some issues with mac version – It’s unclear whether this is an issue with a pre-compiled mac version or a generic self-compiled (which should be identical to the linux version) source

Dev Server Tests

Ran apt-get update and apt-get upgrade on dev server

ran apt-get upgrade apache2 and message returned says that 2.2.8 is the current version. Which means my dev and my production servers are out of sync. I was surprised to learn that the last contractor from whom I took over server management had installed from source, removing Apache from package management. Not a big deal really, but it was undocumented.

In any case, there doesn’t appear to be a package release for Apache 2.2.13 yet in Ubuntu. Only one of the 4 bugfixes has a security bulletin attached, so I’ve decided to wait a few weeks to see if anything new transpires in the security bulletins. In general, I prefer to wait on updating production servers until a new release has been out long enough for bugfixes to be released.

1. Add rotation script to logrotate

sudo vi /etc/logrotate.conf

add

# system-specific logs may be also be configured here.
/usr/local/Zope/Sites/SiteName/log/Z2.log {
rotate 5
weekly
compress
size=100k
sharedscripts
postrotate
#close and re-open all Zope log files (z2.log, event.log) The common idiom after rotating Zope log files
/bin/kill -s SIGUSR2 `cat /usr/local/Zope/Sites/SiteName/var/Z2.pid`
endscript
}

/usr/local/Zope/Sites/SiteName/log/event.log {
rotate 5
weekly
compress
size=100k

}

2. Test log rotation

Do a test run of the rotation without actually rotating anything:

/usr/sbin/logrotate -d /etc/logrotate.conf

if the test run completes without any erros, force a rotation:
/usr/sbin/logrotate -f /etc/logrotate.conf

3. Automate with crontab

On RedHat, crontab may be set up with runparts e.g.

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly


In this case, place a script in the directory where it should run regularly. I’m going to rotate weekly for now, so I’m placing a script in /etc/cron.weekly and naming it zope.cron (you can name it whatever you want. Any script in this directory will run weekely


sudo vi /etc/cron.weekly/zope.cron

Setting up the Slicehost Account

Update everything
sudo apt-get update
sudo apt-get upgrade

install vim full so you can edit files
apt-get install vim-full

update the “locate” db
sudo updatedb

Change the root password
sudo passwd

Set up Users
useradd zope (I like useradd because there’s no reason to have a home directory. If you want your user to have a home directory then use adduser or useradd -D zope to create ~/home/zope)
passwd zope
useradd -m -c "real name" -s /bin/bash auserwithsudoers The flags create a home directory with skel profile defaults and this account is linked to the defaults for bash shell, which is important if you like syntax coloring and tab completion.
passwd auserwithsudoers
sudo visudo
Find the line that says #User Privilege specification and add
auserwithsudoers ALL=(ALL) ALL
shift ZZ to save and exit

Now log out as root and login as your new admin user and test sudo
su auserwithsudoers
sudo bash

Disable root login
su root
sudo vim /etc/ssh/sshd_config
Change “PermitRootLogin yes” to “PermitRootLogin no”

Reload the ssh config
sudo /etc/init.d/ssh reload

Ok! Now we’re ready to think about plone

1. Follow the plone install instructions here
2. install some products (see plone documentation on using buildout). Here’s what I did to install a Press Release product
   1. download the product. If there’s a choice of files, choose the one with the naming convention Products.package.tar.gz. This is a python egg.
   2. su zope
vi /usr/local/Plone/zeocluster/buildout.cfg
      add the following:
      [buildout]
...
eggs =
Products.PressRelease

      and save (ctrl + ZZ)
   3. sudo /usr/local/Plone/zeocluster/bin/buildout
   4. sudo /usr/local/Plone/zeocluster/bin/plonectl restart
   5. In each Plone site where you want to install the Product, go Site Setup>Add/Remove Products and install the Product.

I know there are plenty of articles and posts for setting up a development server. Yet I still find the need to write up the exact commands and process for my later self to refer to, especially since I haven’t found a clone function in VMware Fusion like the linux version has. A pet “when-I-have-time” project would be to turn these steps into a bash script.

Again, the environment is Ubuntu Gutsy Gibbon

Setting up a new LAMP server

File Permissions

If you’re going to access your web directory from another computer, you’ll need to change the file permissions on the web directory so you can put files. For example, my development server is a Ubuntu virtual machine but I do some development in a Windows vm and some development on my mac. In both cases, I connect with my development server over an sftp client that’s integrated into my editor environment.

sudo chown yourusername /var/www
alternatively, you might create a new user like “web” or “www”. If you’ll occasionally make your server public, you might want to create a user that does not have sudo access for an extra layer of security.¹

install Apache 2

sudo apt-get install apache2
sudo /etc/init.d/apache2 start
test apache by navigating in browser to localhost.
place an index.html page into /var/www to test that’s all set up.

I like to stop apache while installing other things, but you can always restart it so it doesn’t really matter
sudo /etc/init.d/apache2 stop (or restart)

Install php 5

sudo apt-get install php5 libapache2-mod-php5
restart apache so it registers with php
sudo /etc/init.d/apache2 restart
test that it worked by placing a phpinfo file into /var/www
create the file
vi /var/www/phpinfo.php
then type
<?php phpinfo(); >

Navigate in browser to localhost/phpinfo.php to see that it’s working

install mysql server

sudo apt-get install mysql-server
sudo apt-get install libapache2-mod-auth-mysql php5-mysql mysql-client

Before you start configuring files, you need to install the full version of vi. By default, ubuntu only comes with vim-tiny. You’ll know it’s vim-tiny when you try to edit a file with vi and the console reads out characters like ^B. So,
sudo apt-get install vim-full

Now, tell php about mysql
sudo vi /etc/php5/apache2/php.ini

It doesn’t matter where you put it, but I like to put it under the line that says “Directory in which the loadable extensions (modules) reside”.
extension=mysql.so
restart apache again.
sudo /etc/init.d/apache2 restart

Install phpmyadmin

This can be done the Ubuntu way or it can be done manually by installing phpmyadmin to /var/www like any other web application.

The Ubuntu way

the benefit to doing it this way is that phpmyadmin is installed as a managed package which means it will be updated by our lovely package manager, apt-get which saves us a little time. Since I use phpmyadmin on multiple virtual machines, this makes a cumulative difference.

sudo apt-get install phpmyadmin

tell apache where phpmyadmin actually lives (which is in /etc/phpmyadmin)
sudo vi /etc/apache2/apache2.conf
You can place the following directive anywhere, but I prefer to place it under the line:
# Include all the user configurations
add the line
Include /etc/phpmyadmin/apache.conf

If you didn’t want to edit your apache.conf, you could always use a symlink instead.

navigate to localhost/phpmyadmin to check that it’s working

Some cleanup/maintenance/utility stuff

If this is a fresh install, you probably need to run the locate database updater so you can use it.
sudo updatedb

Done with the LAMP setup! Next up … subversion and trac

Footnotes

1. for more on managing user accounts see this O’Reilly article or for an in-depth treatise that addresses best practices – check out my favorite reference book “Essential System Administration”. [↩]

When I was in college, my fellow classmates and I whined to each other about the gyrations it took to just get our development environments configured so we could do our homework … building a development environment was not the assignment.

Working as a web developer or web designer isn’t much different. Setting up your infrastructure and development environment is not billable time but sometimes it seems we spend an inordinate amount of effort creating an infrastructure that supports us in getting our work done with less distraction.

The past couple years, infrastructure has gotten easier and more cost effective using virtual machine environments combined with free Linux distributions such as Ubuntu that strive to be more user friendly. So now seemed like a good time re-evaluate how we were doing things and how we might do things differently to both increase productivity and reduce the environmental impact of our office.

Our General Infrastructure Requirements:

1. A Linux development and test server for programming projects.
2. A Windows webserver for .NET development
3. Windows workstations for IE browser testing ¹
4. A design station: Apple computers have superior system-level image rendering and have long been the standard for design stations.
5. A workhorse: this can double as your design station. If you need to run the many Windows-only business applications, this might be a Windows PC.
6. A server for internal applications – this should be more stable than the development/test server.
7. A router, preferably that combines wireless and print networking.
8. And last, but certainly not least, we want to become more productive while reducing environmental impact of our office … which is a whole different topic I’ll cover another time.

Our starting point was

1. A year ago, my computer was desperately old but I was too busy with work to really think through what our ideal infrastructure should be. So I quickly built 2 computers from scavenged parts with the intent of using them for a year. Well, that year is up!
2. 1 Ubuntu machine which was my workhorse, running VMware with multiple installs of Windows and Linux for testing. I had built this a year ago reusing a cpu from a Dell
3. 1 Windows server for .NET development. I had built this machine a year ago reusing another old cpu from a Dell.
4. 1 linux server (a 5 year old Dell sc600) chugging noisily away in the closet. This was our development server
5. 1 power pc used as a design station.

Functionally, what I needed was a multi-core computer that would let me run lots of virtual machines simultaneously for testing and development. I’ve subscribed to browsercam for years to test websites which is expensive. I need to be able to test websites on Windows, Mac and Linux and in multiple versions. When Apple switched over to the Intel processor, I decided to get an Intel-based Mac as soon as VMware released a mac version. So, the end result is:

• We kept our PowerPC design station with Adobe Creative Suite, professional fonts and font manager, and various other design and multimedia tools.
• I purchased an Intel Apple and migrated from my Ubuntu workstation to the Apple. This is experimental for me and is what prompted me to write this. I have to admit that I really miss my Ubuntu desktop but I’ll give the mac a fair try. I installed VMware Fusion on my mac, and created a number of VMs:
  • A standard LAMP+RoR+Plone installation (replacing my ancient Dell sc600)
  • A Windows workhorse install for mostly business applications such as QuickBooks, Microsoft Office, etc
  • Multiple installs of Windows for website testing in different versions of Internet Explorer
  • multiple installs of Ubuntu for various client projects that require different versions and packages installed on the server.²
• Most of our internal knowledge management and productivity tools are php, java and ruby on rails web applications. I spent more time than I thought was reasonable setting up hosting on my mac (Leopard), so I ended up creating a separate Ubuntu VM which took about 15 minutes to setup. This is fine since with a quad core, I can comfortably run multiple VMs and all our internal knowledge management is browser based anyway so the vm just has to be on and have an IP address.
• Anyhoo … right before I switched over to the Intel Apple our old Linksys router went belly up. We’ve been pretty loyal to the Cisco-owned Linksys over the years despite the Windows-only admin interface. But I didn’t want to waste money on obsolete technology like G-band so I bought the wireless N Linksys router ….. and promptly returned it.
  We did more research and discovered that with the Apple Airport Extreme, we’d get not only a wireless-N router that we could administrate from our macs, but also a built-in print server which was something else we’d never been satisfied with in our infrastructure (well, dissatisfied isn’t really accurate. We were never able to get our old linksys print server to work nicely with our mac).
  So, the upshot is we got the Apple Airport Extreme and it works great – absolutely no complaints. Both macs can print, I can print from my linux and Windows VMs and, most importantly, I don’t have to think about it.

The end result? pretty good! I replaced 3 older, noisy, energy consuming computers with 1 for a grand total of 2 computers in our office. Which is nice since there are only 2 of us who work in this particular office. This should save us some maintenance time, means we’re consuming less energy and the office is quieter now that our 2 computers are both water cooled. I put away my KVM switch and cleared my desk of cables – I’ll eventually sell it. I put the router and printer in the closet³ and rarely have to “power cycle” the router.

VMware Fusion isn’t perfect yet – I’ll write about the trade offs in a separate post

1. you can use various hacks for running multiple versions of IE on one machine … but the risks are:
   • they may not behave the same way as an installed version
   • a Microsoft update will break them
   • they don’t work consistently across Windows 2000, XP or Vista
   • when testing javascript behavior, I need to trust any errors are the javascript, not the browser install.

   [↩]

2. Although in most cases multiple versions of software can be run side by side in Linux, I find that it’s less time consuming and less fraught with weird errors that must be resolved to just run a separate VM for those special cases rather than running 4 different versions of Python, for example, on the same machine… [↩]
3. laser printers release some nasty particulates when printing largish documents. So it’s a good idea to place them in another room or enclosure. See “SF Gate article” Here’s a link to the Particle Emission Characteristics of Office Printers [↩]