24. Maintaining Zope

Keeping a Zope site running smoothly involves a number of administrative tasks. This chapter covers some of these tasks, such as:

• Starting Zope automatically at boot time
• Installing new products
• Setting parameters in the Control Panel
• Monitoring
• Cleaning up log files
• Packing and backing up the database
• Database recovery tools

Maintenance often is a very platform-specific task, and Zope runs on many platforms, so you will find instructions for several different operating systems here. It is not possible to provide specifics for every system; instead, we will supply general instructions which should be modified according to your specific needs and platform.

24.1. Starting Zope Automatically at Boot Time

For testing and developing purposes you will start Zope manually most of the time, but for production systems it is necessary to start Zope automatically at boot time. Also, we will want to shut down Zope in an orderly fashion when the system goes down. We will describe the necessary steps for Microsoft Windows and some Linux distributions. Take a look at the Linux section for other Unix-like operating systems. Much of the information presented here also applies to System V like Unices.

24.1.1. Debug Mode and Automatic Startup

If you are planning to run Zope on a Unix production system you should also disable debug mode. This means removing the -D option in startup scripts (e.g. the start script created by Zope at installation time which calls z2.py with the -D switch) and if you’ve manually set it, unsetting the Z_DEBUG_MODE environment variable. In debug mode, Zope does not detach itself from the terminal, which could cause startup scripts to malfunction.

On Windows, running Zope as a service disables debug mode by default. You still can run Zope in debug mode by setting the Z_DEBUG_MODE environment variable or running Zope manually from a startup script with the -D option. Again, this is not recommended for production systems, since debug mode causes performance loss.

24.1.1.1. Automatic Startup for Custom-Built Zopes

Even if you do not want to use the prepackaged Zope that comes with your distribution it should be possible to re-use those startup scripts, eg. by installing the prepackaged Zope and editing the appropriate files and symlinks in /etc/rc.d or by extracting them with a tool like rpm2cpio.

In the following examples we assume you installed your custom Zope to a system-wide directory, eg. /usr/local/zope. If this is not the case please replace every occurence of /usr/local/zope below with your Zope installation directory. There should also be a separate Zope system user present. Below we assume that there is a user zope, group nogroup present on your system. The user zope should of course have read access to the $ZOPE_HOME directory (the directory which contains the “top-level” Zope software and the “z2.py” script) and its descendants, and write access to the contents of the var directory.

If you start Zope as root, which is usually the case when starting Zope automatically on system boot, it is required that the var directory belongs to root. Set the ownership by executing the command:

chown root var

as root.

To set up a Zope binary package with built-in python situated in:: /usr/local/zope running as user zope , with a “WebDAV Source port” set to 8081, you would set:

ZOPE_HOME=/usr/local/zope
PYTHON_BIN=$ZOPE_HOME/bin/python
COMMON_PARAMS="-u zope -z $ZOPE_HOME -Z /var/run/zope.pid -l /var/log/Z2.log -W 8081"

You can also set up a file /etc/sysconfig/zope with variables ZOPE_FTP_PORT, ZOPE_HTTP_PORT:

ZOPE_HTTP_PORT=80
ZOPE_FTP_PORT=21

to set the HTTP and FTP ports. The default is to start them at port 8080 and 8021.

Unfortunately, all Linux distributions start and stop services a little differently, so it is not possible to write a startup script that integrates well with every distribution. We will try to outline a crude version of a generic startup script which you can refine according to your needs.

To do this some shell scripting knowledge and root system access is required.

Linux startup scripts usually reside in:

/etc/init.d

or in:

/etc/rc.d/init.d

For our examples we assume the startup scripts to be in:

/etc/rc.d/init.d

adjust if necessary.

To let the boot process call a startup script, you also have to place a symbolic link to the startup script in the:

/etc/rc.d/rc?.d

directories, where ? is a number from 0-6 which stands for the SystemV run levels. You usually will want to start Zope in run levels 3 and 5 (3 is full multi-user mode, 5 is multiuser mode with X started, according to the “Linux Standard Base”:http://www.linuxbase.org), so you would place two links in the /etc/rc.d’ directories. Be warned that some systems (such as Debian) assume that runlevel 2 is full multiuser mode. As stated above, we assume the main startup script to located in:

/etc/rc.d/init.d/zope

if your system puts the:

init.d

directory somewhere else, you should accomodate the paths below:

# cd /etc/rc.d/rc3.d
# ln -s /etc/rc.d/init.d/zope S99zope
# cd /etc/rc.d/rc5.d
# ln -s /etc/rc.d/init.d/zope S99zope

The scripts are called by the boot process with an argument:

start

when starting up and:

stop

on shutdown.

A simple generic startup script structure could be something like this:

#!/bin/sh

# set paths and startup options
ZOPE_HOME=/usr/local/zope
PYTHON_BIN=$ZOPE_HOME/bin/python
ZOPE_OPTS=" -u zope -P 8000"
EVENT_LOG_FILE=$ZOPE_HOME/var/event.log
EVENT_LOG_SEVERITY=-300
# define more environment variables ...

export EVENT_LOG_FILE  EVENT_LOG_SEVERITY
# export more environment variables ...

umask 077
cd $ZOPE_HOME

case "$1" in

start)
# start service
exec $PYTHON_BIN $ZOPE_HOME/z2.py $ZOPE_OPTS

# if you want to start in debug mode (not recommended for
# production systems):
# exec $PYTHON_BIN $ZOPE_HOME/z2.py $ZOPE_OPTS -D &
;;
stop)
# stop service
kill `cat $ZOPE_HOME/var/Z2.pid`
;;
restart)
# stop service and restart
$0 stop
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac

24.1.1.2. This script lets you perform start / stop / restart operations:

start

Start Zope (and the zdaemon management process)

stop

Stop Zope. Kill Zope and the zdaemon management process

restart

Stop then start Zope

24.1.2. MS Windows

The prevalent way to autostart Zope on MS Windows is to install Zope as a service.

If you installed Zope on Windows NT/2000/XP to be started manually and later on want it started as a service, perform these steps from the command line to register Zope as a Windows service::

> cd c:\Program Files\zope
> bin\lib\win32\PythonService.exe /register
> bin\python.exe ZServer\ZService.py --startup auto install

Replace:

c:\Program Files\zope

with the path to your Zope installation. Zope should now be installed as a service which starts automatically on system boot. To start and stop Zope manually, go to the Windows service administration tool, right-click the Zope service and select the corresponding entry.

24.2. Installing New Products

Zope is a framework for building websites from new and existing software, known as Zope products. A product is a Python package with special conventions that register with the Zope framework. The primary purpose of a Zope product is to create new kinds of objects that appear in the add list. This extensibility through products has spawned a broad market of add-on software for Zope.

The guidelines for packaging a product are given in the “Packaging Products” section in the Zope Products chapter of the Zope Developer Guide. However, since these guidelines are not enforced, many Zope products adhere to different conventions. This section will discuss the different approaches to installing Zope packages.

To install a Zope product, you first download an archive file from a website, such as the Downloads section of zope.org. These archive files come in several varieties, such as tgz (gzipped tar files) zip (the popular ZIP format common on Windows), and others.

In general, unpacking these archives will create a subdirectory containing the Product itself. For instance, the:

Poll-1.0.tgz

archive file in the “Packaging Products” section mentioned above contains a subdirectory of Poll. All the software is contained in this directory.

To install the product, you unarchive the file in the:

lib/python/Products

directory. In the Poll example, this will create a directory:

lib/python/Products/Poll

Unfortunately not all Zope developers adhere to this convention. Often the archive file will have the:

lib/python/Products

part of the path included. Worse, the archive might contain no directory, and instead have all the files in the top-level of the archive. Thus, it is advised to inspect the contents of the archive first.

Once you have the new directory in:

lib/python/Products

you need to tell Zope that a new product has been added. You can do this by restarting your Zope server through the Control Panel of the Zope Management Interface (ZMI), or, on POSIX systems, by sending the Zope process a:

-HUP

signal. For instance, from the Zope directory::

kill -HUP `cat var/Z2.pid`

If your Zope server is running in debug mode, a log message will appear indicating a new product has been discovered and registered.

To confirm that your product is installed, log into your Zope site and visit the Control Panel’s Products section. You should see the new product appear in the list of installed products.

If there was a problem with the installation, the Control Panel will list it as a “Broken Product”. Usually this is because Python had a problem importing a package, or the software had a syntax error. You can visit the broken product in the Control Panel and click on its Traceback tab. You will see the Python traceback generated when the package was imported.

A traceback generally will tell you what went wrong with the import. For instance, a package the software depends on could be missing. To illustrate this take a look at the traceback below - a result of trying to install CMFOODocument:http://www.zope.org/Members/longsleep/CMFOODocument without the (required) CMF package::

Traceback (most recent call last):
File "/usr/share/zope/2.6.0/lib/python/OFS/Application.py", line 541, in import_product
product=__import__(pname, global_dict, global_dict, silly)
File "/usr/share/zope/2.6.0/lib/python/Products/CMFOODocument/__init__.py", line 19, in ?
import OODocument
File "/usr/share/zope/2.6.0/lib/python/Products/CMFOODocument/OODocument.py", line 31, in ?
from Products.CMFCore.PortalContent import NoWL, ResourceLockedError
ImportError: No module named CMFCore.PortalContent

24.3. Server Settings

The Zope server has a number of settings that can be adjusted for performance. Unfortunately, performance tuning is not an exact science, that is, there is no recipe for setting parameters. Rather, you have to test every change. To load test a site, you should run a test setup with easily reproducible results. Load test a few significant spots in your application. The trick is to identify typical situations while still permitting automated testing. There are several tools to load test websites. One of the simple yet surprisingly useful tools is:

ab

which comes with Apache distributions. With ab you can test individual URLs, optionally providing cookies and POST data. Other tools often allow one to create or record a user session and playing it back multiple times. See eg. the Open System Testing Architecture, JMeter, or Microsoft’s Web Application Stress Tool.

24.3.1. Database Cache

The most important is the database cache setting. To adjust these settings, visit the Control Panel and click on the Database link.

There are usually seven database connections to the internal Zope database (see Database Connections below for information about how to change the number of connections). Each connection gets its own database cache. The “Target number of objects in memory per cache” setting controls just that - the system will try not to put more than this number of persistent Zope objects into RAM per database connection. So if this number is set to 400 and there are seven database connections configured, there should not be more than 2800 objects sitting in memory. Obviously, this does not say much about memory consumption, since the objects might be anything in size - from a few hundred bytes upwards. The cache favors commonly used objects - it wholly depends on your application and the kind of objects which memory consumption will result from the number set here. As a rule, Zope objects are about as big as the data they contain. There is only little overhead in wrapping data into Zope objects.

24.3.2. ZServer Threads

This number determines how many ZServer threads Zope starts to service requests. The default number is four (4). You may try to increase this number if you are running a heavily loaded website. If you want to increase this to more than seven (7) threads, you also should increase the number of database connections (see the next section).

24.3.3. Database Connections

We briefly mentioned Zope’s internal database connections in the Database Cache section above. Out of the box, the number of database connections is hardwired to seven (7); but this can be changed. There is no “knob” to change this number so in order to change the number of database connections, you will need to enter quite deep into the systems’ bowels. It is probably a wise idea to back up your Zope installation before following any of the instructions below.

Each database connection maintains its own cache (see above, “Database Cache”), so bumping the number of connections up increases memory requirements. Only change this setting if you’re sure you have the memory to spare.

To change this setting, create a file called “custom_zodb.py” in your Zope installation directory. In this file, put the following code:

import ZODB.FileStorage
import ZODB.DB

filename = os.path.join(INSTANCE_HOME, 'var', 'Data.fs')
Storage = ZODB.FileStorage.FileStorage(filename)
DB = ZODB.DB(Storage, pool_size=25, cache_size=2000)

This only applies if you are using the standard Zope FileStorage storage.

The “pool_size” parameter is the number of database connections. Note that the number of database connections should always be higher than the number of ZServer threads by a few (it doesn’t make sense to have fewer database connections than threads). See above on how to change the number of ZServer threads.

24.4. Signals (POSIX only)

Signals are a POSIX inter-process communications mechanism. If you are using Windows then this documentation does not apply.

Zope responds to signals which are sent to the process id specified in the file ‘$ZOPE_HOME/var/Z2.pid’:

SIGHUP

close open database connections, then restart the server process. The common idiom for restarting a Zope server is:

kill -HUP `cat $ZOPE_HOME/var/Z2.pid`

SIGTERM

close open database connections then shut down. The common idiom for shutting down Zope is:

kill -TERM `cat $ZOPE_HOME/var/Z2.pid`

SIGINT

same as SIGTERM

SIGUSR2

close and re-open all Zope log files (z2.log, event log, detailed log.) The common idiom after rotating Zope log files is:

kill -USR2 `cat $ZOPE_HOME/var/Z2.pid`

The process id written to the:

Z2.pid

file depends on whether Zope is run under the:

zdaemon

management process. If Zope is run under a management process (as it is by default) then the pid of the management process is recorded here. Relevant signals sent to the management process are forwarded on to the server process. Specifically, it forwards all those signals listed above, plus SIGQUIT and SIGUSR1. If Zope is not using a management process (-Z0 on the z2.py command line), the server process records its own pid into z2.pid, but all signals work the same way.

24.5. Monitoring

To detect problems (both present and future) when running Zope on production systems, it is wise to watch a few parameters.

24.5.1. Monitor the Event Log and the Access Log

If you set the EVENT_LOG_FILE (formerly known as the STUPID_LOG_FILE) as an environment variable or a parameter to the startup script, you can find potential problems logged to the file set there. Each log entry is tagged with a severity level, ranging from TRACE (lowest) to PANIC (highest). You can set the verbosity of the event log with the environment variable EVENT_LOG_SEVERITY. You have to set this to an integer value - see below:

TRACE=-300   -- Trace messages

DEBUG=-200   -- Debugging messages

BLATHER=-100 -- Somebody shut this app up.

INFO=0       -- For things like startup and shutdown.

PROBLEM=100  -- This isn't causing any immediate problems, but deserves
                attention.

WARNING=100  -- A wishy-washy alias for PROBLEM.

ERROR=200    -- This is going to have adverse effects.

PANIC=300    -- We're dead!

So, for example setting EVENT_LOG_SEVERITY=-300 should give you all log messages for Zope and Zope applications that use Zopes’ logging system.

You also should look at your access log (usually placed in $ZOPE_HOME/var/Z2.log). The Z2.log file is recorded in the Common Log Format. The sixth field of each line contains the HTTP status code. Look out for status codes of 5xx, server error. Server errors often point to performance problems.

24.5.2. Monitor the HTTP Service

You can find several tools on the net which facilitate monitoring of remote services, for example Nagios or VisualPulse.

For a simple “ping” type of HTTP monitoring, you could also try to put a small DTML Method with a known value on your server, for instance only containing the character “1”. Then, using something along the line of the shell script below, you could periodically request the URL of this DTML Method, and mail an error report if we are getting some other value (note the script below requires a Un*x-like operating system):

#!/bin/sh

# configure the values below
URL="http://localhost/ping"
EXPECTED_ANSWER="1"
MAILTO="your.mailaddress@domain.name"
SUBJECT="There seems to be a problem with your website"
MAIL_BIN="/bin/mail"

resp=`wget -O - -q -t 1 -T 1 $URL`
if [ "$resp" != "$EXPECTED_ANSWER" ]; then
$MAIL_BIN -s "$SUBJECT" $MAILTO <<EOF
The URL
----------------------------------------------
$URL
----------------------------------------------
did not respond with the expected value of $EXPECTED_ANSWER.
EOF
fi;

Run this script eg. every 10 minutes from cron and you should be set for simple tasks. Be aware though that we do not handle connections timeouts well here. If the connection hangs, for instance because of firewall misconfiguration wget will likely wait for quite a while (around 15 minutes) before it reports an error.

24.6. Log Files

There are two main sources of log information in Zope, the access log and the event log.

24.6.1. Access Log

The access log records every request made to the HTTP server. It is recorded in the Common Log Format.

The default target of the access log is the file $ZOPE_HOME/var/Z2.log. Under Unix it is however possible to direct this to the syslog by setting the environment variable ZSYSLOG_ACCESS to the desired domain socket (usually /dev/log)

If you are using syslog, you can also set a facility name by setting the environment variable ZSYSLOG_FACILITY. It is also possible to log to a remote machine. This is also controlled, you might have guessed it, by an environment variable. The variable is called ZSYSLOG_SERVER and should be set to a string of the form “host:port” where host is the remote logging machine name or IP address and port is the port number the syslog daemon is listening on (usually 514).

24.6.2. Event Log

The event log (formerly also called “stupid log”) logs Zope and third-party application message. The ordinary log method is to log to a file specified by the EVENT_LOG_FILE, eg. EVENT_LOG_FILE=$ZOPE_HOME/var/event.log.

On Unix it is also possible to use the syslog daemon by setting the environment variable ZSYSLOG to the desired Unix domain socket, usually /dev/log . Like with access logs (see above), it is possible to set a facility name by setting the ZSYSLOG_FACILITY environment variable, and to log to a remote logging machine by setting the ZSYSLOG_SERVER variable to a string of the form “host:port”, where port usually should be 514.

You can coarsely control how much logging information you want to get by setting the variable EVENT_LOG_SEVERITY to an integer number - see the section “Monitor the Event Log and the Access Log” above.

24.6.3. Log Rotation

Log files always grow, so it is customary to periodically rotate logs. This means logfiles are closed, renamed (and optionally compressed) and new logfiles get created. On Unix, there is the logrotate package which traditionally handles this. A sample configuration might look like this:

compress
/usr/local/zope/var/Z2.log {
rotate 25
weekly
postrotate
/sbin/kill -USR2 `cat /usr/local/zope/var/Z2.pid`
endscript
}

This would tell logrotate to compress all log files (not just Zope’s!), handle Zopes access log file, keep 25 rotated log files, do a log rotation every week, and send the SIGUSR2 signal to Zope after rotation. This will cause Zope to close the logfile and start a new one. See the documentation to logrotate for further details.

On Windows there are no widespread tools for log rotation. You might try the KiWi Syslog Daemon and configure Zope to log to it. Also see the sections “Access Log” and “Event Log” above.

24.7. Packing and Backing Up the FileStorage Database

The storage used by default by Zope’s built-in object database, FileStorage, is an undoable storage. This essentially means changes to Zope objects do not overwrite the old object data, rather the new object gets appended to the database. This makes it possible to recreate an objects previous state, but it also means that the file the objects are kept in (which usually resides in $ZOPE_HOME/var/Data.fs) always keeps growing.

To get rid of obsolete objects, you need to:: pack the ZODB. This can be done manually by opening Zopes Control_Panel and clicking on the “Database Management” link. Zope offers you the option of removing only object version older than an adjustable amount of days.

If you want to automatically pack the ZODB you could tickle the appropriate URL with a small python script (the traditional filesystem based kind, not Zopes “Script (Python)”):

#!/usr/bin/python
import sys, urllib
host = sys.argv[1]
days = sys.argv[2]
url = "%s/Control_Panel/Database/manage_pack?days:float=%s" % (host, days)
try:
    f = urllib.urlopen(url).read()
except IOError:
    print "Cannot open URL %s, aborting" % url
    print "Successfully packed ZODB on host %s" % host

The script takes two arguments, the URL of your server (eg. http://mymachine.com) and the number of days old an object version has to be to get discarded.

On Unix, put this in eg. the file:

/usr/local/sbin/zope_pack

and make it executable with:

chmod +x zope_pack

Then you can put in into your crontab with eg.:

5 4 * * sun     /usr/local/sbin/zope_pack http://localhost 7

This would instruct your system to pack the ZODB on 4:05 every sunday. It would connect to the local machine, and leave object versions younger than 7 days in the ZODB.

Under Windows, you should use the scheduler to periodically start the script. Put the above script in eg.:

c:\Program Files\zope_pack.py

or whereever you keep custom scripts, and create a batch file:

zope_pack.bat

with contents similar to the following::

"C:\Program Files\zope\bin\python.exe" "C:\Program Files\zope_pack.py" "http://localhost" 7

The first parameter to python is the path to the python script we just created. The second is the root URL of the machine you want to pack, and the third is the maximum age of object versions you want to keep. Now instruct the scheduler to run this .bat file every week.

Zope backup is quite straightforward. If you are using the default storage (FileStorage), all you need to do is to save the file:

$ZOPE_HOME/var/Data.fs

This can be done online, because Zope only appends to the Data.fs file - and if a few bytes are missing at the end of the file due to a copy while the file is being written to, ZODB is usually capable of repairing that upon startup. The only thing to worry about would be if someone were to be using the Undo feature during backup. If you cannot ensure that this does not happen, you should take one of two routes. The first is be to shutdown Zope prior to a backup, and the second is to do a packing operation in combination with backup. Packing the ZODB leaves a file Data.fs.old with the previous contents of the ZODB. Since Zope does not write to that file anymore after packing, it is safe to backup this file even if undo operations are performed on the live ZODB.

To backup Data.fs on Linux, you should not tar it directly, because tar will exit with an error if files change in the middle of a tar operation. Simply copying it over first will do the trick.

24.8. Database Recovery Tools

To recover data from corrupted ZODB database file (typically located in $ZOPE_HOME/var/Data.fs ) there is a script fsrecover.py located in $ZOPE_HOME/lib/python/ZODB.

fsrecover.py has the following help output:

python fsrecover.py [ <options> ] inputfile outputfile

Options:

-f -- force output even if output file exists

-v level -- Set the
verbosity level:

0 -- Show progress indicator (default)

1 -- Show transaction times and sizes

2 -- Show transaction times and sizes, and
show object (record) ids, versions, and sizes.

-p -- Copy partial transactions. If a data record in the middle of a
transaction is bad, the data up to the bad data are packed. The
output record is marked as packed. If this option is not used,
transaction with any bad data are skipped.

-P t -- Pack data to t seconds in the past. Note that is the "-p"
option is used, then t should be 0.