semioticpixels scratch pad

I have 2 old Plone 2.0 websites running on a single Zope instance that I need to:

1. migrate from it’s ancient host to a more modern server
2. upgrade to Plone 3

I’m migrating away from a 32 bit Red Hat 7(!) server to a 64 bit Ubuntu Hardy Virtual Machine. The reasons should be obvious from that statement alone, but in short: Red Hat 7 reached its end-of-life years ago. It’s stable (so far) but all system maintenance has to be done manually with no package or dependency management. There are many benefits to running within a virtualized environment, namely that I can clone it as many times as I like to develop and test other websites without having to set up a whole new server, and it simplifies our redundancy and backup process.

Anyway, this article is both a cheatsheet and an historical commentary for when I or someone else is trying to solve a problem and has a WTF moment regarding how and why things are set up the way they are.

The server environment I’m migrating away from is:
- Server: Zope/(Zope 2.7.3-0, python 2.3.3)
- ZServer/1.1
- Plone/2.0.3

I’ll also be upgrading from Apache 1.3 to Apache 2 and switching from Squid to Varnish for caching

Getting started: Migrating from Red Hat 7.3 to a Ubuntu 8 VM

Zope is an “in-place” installation, which means it runs from the directory that you put it in. This is nice and simple, makes it easier to run different versions of Zope on the same server with associated different versions of Python and Plone and, theoretically it should be easy to move to another system altogether.

So, my first pass, I rsynced the Zope directory over from the old server to the new one and tried to compile Python and Zope to see what would happen. Unfortunately, when I compiled Python, I got a bunch of errors that indicated my version of Python was optimized for a 32 bit system and … the new VM is 64 bit. E.g.

`Parser/tokenizer_pgen.o' is incompatible with i386:x86-64 output

Python 2.3.7 was the oldest version that I could get running on the new VM that didn’t throw 32 bit errors. I briefly considered re-compiling Python 2.3.3 with -m32 option to GCC and I could have spent quite a lot of time researching and very likely could have gotten 2.3.3 running on the VM but 2.3.7 was a bugfix release and the most stable of the 2.3 releases and should work fine so I chose to save time by going with it. We’ll see if I regret that later. I was also reminded in the course of this process to clean up my mess with

$ sudo make clean

When I compiled Python, the following error occurred: Can't locate Tcl/Tk libs and/or headers, so I had to install the tcl/tk libraries (added to setup below)

Plone/2.0.3 is no longer available for download, although I was able to locate a copy of Plone/2.0.5. I don’t have a development server to upgrade the website to 2.0.5 so if I have to use it, I may have to keep my fingers crossed. Since Plone is installed as a Zope Product and doesn’t require compilation, I will first try to copy over the Products directory of the website and see if it’ll just work.

I will also need to install PIL and PyXML

When I compiled Zope the first time, I got error messages that zlib was missing. It turns out that Ubuntu ships with zlib, but not zlib-dev installed. Also, when researching that error, I came across a comment that I would need to install system support (libjpeg62) for PIL so those 2 lines have been added to the VM setup below.

Initial VM setup

update and install basics


$ sudo apt-get install update
$sudo apt-get upgrade
make sure gcc is up to date
$sudo apt-get install build-essential
$sudo apt-get install zlib1g zlib1g-dev libjpeg62 libjpeg62-dev tcl tcl-dev tk tk-dev vim-full lynx


update the locate db so you can find stuff

$sudo updatedb

change root password
$sudo passwd

create a user with sudo privileges:
$ useradd -m -c "real name" -s /bin/bash chris

The flags create a home directory with skel profile defaults and this account is linked to the defaults for bash shell, which is important if you like syntax coloring and tab completion.
$passwd chris
$sudo visudo


Find the line that says #User Privilege specification and add

chris ALL=(ALL) ALL
shift ZZ to save and exit

create zope user
$useradd -m -c “zope” -s /bin/bash zope
$passwd zope

Do not add zope user to the sudoers list.

Disable root login
$sudo vi /etc/ssh/sshd_config

Change “PermitRootLogin yes” to “PermitRootLogin no”
Reload ssh
$sudo /etc/init.d/ssh reload

create directory structure

The Library directory in the below structure may look a little over-organized at first pass. However, my next project after completing this migration will be to upgrade from Plone 2.0.3 to Plone 3 which will require several interim upgrades. This version of the website is running on Zope 2.7.3. Each interim upgrade will be installed to it’s respective directory within Library/Software keeping everything well organized and easy to roll back to if necessary.

in /usr/local

/usr/local/Zope
…./Downloads: where downloads will be stored
…./src: where software will be unarchived
…./Library: where compiled software will be compiled to
………./Software
……………/Zope273
………………../Python
………………./Zope
…/Sites: where each website to be hosted lives
………/instancename1
……../instancename2


# cd /usr/local/Zope/Downloads
# wget http://www.python.org/ftp/python/2.3.7/Python-2.3.7.tgz
# wget http://www.zope.org/Products/Zope/2.7.0/Zope-2.7.0.tgz
# wget http://internap.dl.sourceforge.net/sourceforge/plone/Plone-2.0.5.tar.gz
# wget http://sourceforge.net/projects/pyxml/files/pyxml/0.8.4/PyXML-0.8.4.tar.gz
# wget http://effbot.org/downloads/Imaging-1.1.6.tar.gz

#cp * /usr/local/Zope/src/


Install Python 2.3.7

Build Python 2.3 from source with the following steps:

cd /usr/local/Zope/src/
tar -xvzf Python-2.3.7.tgz
cd Python-2.3.7
./configure --prefix=/usr/local/Zope/Library/Software/Zope273/Python
make
make install


Install PIL

More info

At the time of this writing, PIL 1.1.6 is compatible with Python 1.5.2 and up and fixes some 64 bit compatibility errors in Python 2.5 (which we’re not using yet for this Zope installation)

On Ubuntu Hardy, you have to edit setup.py before installing PIL
#sudo vi /usr/local/Zope/src/Imaging-1.1.6/setup.py

find the line
#TCL_ROOT = None

and replace it with
TCL_ROOT = "/usr/include/tk"

#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py build
#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py install


Install PyXML

cd /usr/local/Zope/src/PyXML-0.8.4/

#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py build
#/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 setup.py install


Other recommended packages

elementree has been recommended, but it’s not installed on the old system so I’m not going to worry about it atm.

Also, the http://plone.org/documentation/kb/setup-from-source plone article recommends installing DocFinder as an invaluable development tool, but again, I don’t need this to get the website running so I’m going to come back to it.

install zope 2.7.3

Build Zope 2.7 from source with the following steps:

cd /usr/local/Zope/src
tar -xvzf Zope-2.7.3.tgz
cd Zope-2.7.3
./configure --with-python=/usr/local/Zope/Library/Software/Zope273/Python/bin/python2.3 --prefix=/usr/local/Zope/Library/Software/Zope273/Zope
make
make install


create zope instance(s)

The zope user must have write access to create the directory. After the instance is created, edit #”effective-user zope” into the etc/zope.conf file, so if you start it as root later it should #su itself to the non-root user. Again: make install should be run as root, #mkzopeinstance.py should not.

Initially, when I tried to create an instance, it failed with the error:
# /usr/local/Zope/Library/Software/Zope273/Zope/bin/mkzopeinstance.py: /usr/local/Zope/bin/python: bad interpreter: No such file or directory


I think this was a result of not cleaning up a bad compilation. To fix this edit the first line in mkzopeinstance.py from
#!/usr/local/Zope/bin/python
To
#!/usr/local/Zope/LIbrary/Software/Zope273/Zope/bin/python

When Zope is compiled, one of the last things it does is create a symbolic link from where you told Zope the python interpreter you wanted it to use lives to Zope/bin/python. mkzopeinstance.py is looking for that symbolic link, and not looking for python itself. This provides further separation so that we could theoretically upgrade python without touching Zope and then that symbolic link would only need to be changed.

The instance needs to be created with the zope user, NOT root

# su zope
# /usr/local/Zope/Library/Software/Zope273/Zope/bin/mkzopeinstance.py
# when prompted for the path to your instance, use:
# /usr/local/Zope/Sites/instance1 .... and so forth for each site.


test Zope

For the first instance, you can test by running /usr/local/Zope/Sites/instance1/bin/runzope. OR running /usr/local/Zope/Sites/instance1/bin/zopectl start

Once loaded, this will make Zope accessible on http://localhost:8080 (unless you changed the port), with the Zope Management Interface available on http://localhost:8080/manage (obviously if you’re accessing a remote server, then localhost might not work and you need to use the IP address)

shut down zone by either hitting ctrl+c if runzope was used or by running /usr/local/Zope/Sites/instance1/bin/zopectl stop if zopectl was used to start zone.

Install the website


cp originalSite/Products/* /usr/local/Zope/Sites/instance1/Products/
cp originalSite/var/Data.fs /usr/local/Zope/Sites/instance1/var/

/usr/local/Zope/Sites/instance1/bin/zopectl start

navigate to the ZMI localhost:8080/manage
(Note that the admin login credentials will match the credentials of the site migrated, not the credentials you compiled Zope with)
The website I migrated over, even though I physically placed the files in the instance1 directory within /usr/local/Zope/Sites/instance1 are in the ZMI under “Plone”

click the Root Folder of the ZMI
check the box beside “Plone”
click the “rename” button and rename it to your instance1 site
(at this point I discovered a dependency on lynx which I’ve added to the apt-get list above)

navigate to localhost:8080/instance1 to view the website

On first pass, everything worked!

users

make sure that all the Zope files belong to the zope user
cd /usr/local
chown zope -R Zope
chgrp zope -R Zope
//////

start Zope

Start Zope with the following command:
su zope
/usr/local/Zope/Sites/dev/bin/zopectl start

check it out

Go to the http://mydomain.com:8080/manage, log in, create a new Plone site object.

resources/references

• setup from source
• plone installation

resolving tcl/tk error when installing PIL on Ubuntu Hardy

Next up:

• VirtualHostMonster and Apache with mod_rewrite & mod_proxy
• caching: Cachefu, Pound, Squid, Varnish

1. Add rotation script to logrotate

sudo vi /etc/logrotate.conf

add

# system-specific logs may be also be configured here.
/usr/local/Zope/Sites/SiteName/log/Z2.log {
rotate 5
weekly
compress
size=100k
sharedscripts
postrotate
#close and re-open all Zope log files (z2.log, event.log) The common idiom after rotating Zope log files
/bin/kill -s SIGUSR2 `cat /usr/local/Zope/Sites/SiteName/var/Z2.pid`
endscript
}

/usr/local/Zope/Sites/SiteName/log/event.log {
rotate 5
weekly
compress
size=100k

}

2. Test log rotation

Do a test run of the rotation without actually rotating anything:

/usr/sbin/logrotate -d /etc/logrotate.conf

if the test run completes without any erros, force a rotation:
/usr/sbin/logrotate -f /etc/logrotate.conf

3. Automate with crontab

On RedHat, crontab may be set up with runparts e.g.

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly


In this case, place a script in the directory where it should run regularly. I’m going to rotate weekly for now, so I’m placing a script in /etc/cron.weekly and naming it zope.cron (you can name it whatever you want. Any script in this directory will run weekely


sudo vi /etc/cron.weekly/zope.cron

#rotate Z2.log and event.log in SiteName

0 01 * * * root /usr/sbin/logrotate /etc/logrotate.conf > /dev/null2>&1

Setting up the Slicehost Account

Update everything
sudo apt-get update
sudo apt-get upgrade

install vim full so you can edit files
apt-get install vim-full

update the “locate” db
sudo updatedb

Change the root password
sudo passwd

Set up Users
useradd zope (I like useradd because there’s no reason to have a home directory. If you want your user to have a home directory then use adduser or useradd -D zope to create ~/home/zope)
passwd zope
useradd -m -c "real name" -s /bin/bash auserwithsudoers The flags create a home directory with skel profile defaults and this account is linked to the defaults for bash shell, which is important if you like syntax coloring and tab completion.
passwd auserwithsudoers
sudo visudo
Find the line that says #User Privilege specification and add
auserwithsudoers ALL=(ALL) ALL
shift ZZ to save and exit

Now log out as root and login as your new admin user and test sudo
su auserwithsudoers
sudo bash

Disable root login
su root
sudo vim /etc/ssh/sshd_config
Change “PermitRootLogin yes” to “PermitRootLogin no”

Reload the ssh config
sudo /etc/init.d/ssh reload

Ok! Now we’re ready to think about plone

1. Follow the plone install instructions here
2. install some products (see plone documentation on using buildout). Here’s what I did to install a Press Release product
   1. download the product. If there’s a choice of files, choose the one with the naming convention Products.package.tar.gz. This is a python egg.
   2. su zope
vi /usr/local/Plone/zeocluster/buildout.cfg
      add the following:
      [buildout]
...
eggs =
Products.PressRelease

      and save (ctrl + ZZ)
   3. sudo /usr/local/Plone/zeocluster/bin/buildout
   4. sudo /usr/local/Plone/zeocluster/bin/plonectl restart
   5. In each Plone site where you want to install the Product, go Site Setup>Add/Remove Products and install the Product.