Catz service management

This page contains short process descriptions, notes and checks for Catz application engine management and for version upgrades. This page is provided as public to show an example how a Perl- and Mojolicious-based application can be managed. This also gives me online access to the notes.

Starting from February 4th 2012 the system and the server has automated backup routines so backup instructions were removed from this page.

Ship the changed files to the active server instance under data directory.

Run the database loader on server. Use load modes “meta post” or “meta post check” if there is only data changes, all modes if new photos are to be loaded first.

perl load.pl meta post check

After a succesfull loading roll to the latest database version.

perl roll.pl

The production switches immediately to use the new data. Check that the production effectively runs on the new data. This can be done visually from the service front page that displays the data timestamp.

Copy the database file and the key file back to dev.

Check that both the database file and the key file to revert to exists. If the key file doesn't exist, create it manually

Delete any newer key file. The production switches immediately to use the old data.

Clean old database versions from /db (leave only the latest db and the latest db and its key file). Clean logs from /log. Check what is in /temp and clean it if needed.

Fresh start memcached. Run automated tests.

prove -r ../t -I ../lib

All test should pass without warnings. Check that the test stats are reasonable (Files=23, Tests=9437, 436 wallclock secs).

Git Commit the project if not yet committed.

Detect what is the production environment's path from production server processes or which enviroment has the lastest hypnotoad.pid file.

ps -ef | grep catz

Delete all stuff from the NON-production environment's root directory.

...
rm -fR *

Deploy the new version of the application by shipping all files.

Make sure that you have made at least one database loading compared to the version in production (data version must have been changed forward). You can make a pseudo load by manually updating the database file name and the key file name.

Set the database file and the key file to read only.

prove -r ../t -I ../lib

Run automated tests. All test should pass without warnings. Check that the stats are reasonable (Files=23, Tests=9413, 157 wallclock secs).

Fire up the service.

hypnotoad run.pl

Edit nginx config so that it points to the deployed environment's http port.

Test nginx config.

nginx -t

Reload nginx config.

/etc/init.d/nginx reload

Test with browser that the service works properly and that it runs on the correct enviroment (visually by looking at the service or from HTTP headers). Ocassionally run the page walker script/walk.pl to find dead links.

Change the environment on the boot time startup script.

nano /etc/rc.local

Shutdown the NON-production environment.

cat hypnotoad.pid | xargs kill -QUIT

Git Sync the project if not yet synced.

Do the Mojolicious update on dev first to find out any issues.

Update Mojolicious via CPAN on prod.

Run automated tests, although this has a limited meaning since if something is broken, the results goes to memcache and are served thereafter).

Restart Hypnotoad.

 cat hypnotoad.pid | xargs kill -USR2

log/production.log should indicate an new start.

[Sun Jul 17 18:39:38 2014] [info] Server listening (...)

Also hypnotoad.pid should show a fresh pid.

Check with browser that the service is working normally.

Install the latest stable Perl (this will run for a while).

perlbrew install perl-x.yy.z

Use the new Perl only in the current session.

perlbrew use perl-x.yy.z

Install the latest CPAN. Reload CPAN. Install the required Perl modules.

Run automated tests. All should pass without warnings. Check that the stats are reasonable.

Switch to the new Perl.

perlbrew switch perl-x.yy.z

Restart hypnotoad.

kill -USR2 <pid>

System can be refreshed after code and configuration changes.

First make a pseudo data load. In the script directory:

perl load.pl meta post check

And then roll to its version.

perl roll.pl

Then gracefully reload the service.

cat hypnotoad.pid | xargs kill -USR2

If you wish to test the system after this you can run

prove -r ../t -I ../lib

Restarting of memcached is not needed since the roll to the new data version.

The other option is not to do a pseudo data load and roll but restart memcached. Memcached can be restarted as a root:

/etc/init.d/memcached restart