====
LOG
====
Review Lessons from recovery
=============================
#. not having an operational repo + trac + docs : serious hit on ability to recover and build these
#. benefit of plain text sphinx docs (compared to wiki docmentation : hidden away in sqlite DB)
#. self-hosting env (the scripts for building) not a good idea
#. move at least **env** to github / google code ?
#. move important documentation to Sphinx
#. use of older machines generally means source builds for usable versions as distros are behind
Failed to find viable interim server
-------------------------------------
#. cms01, system apache config for heprez too sensitive to touch as currently in use
#. a front to tomcat + exist and static presenter
#. belle7, in use for OUM + dybslv
#. ``sudo yum --enablerepo=epel info trac`` at 0.10.5 **MAYBE NEW ENOUGH : USING BELLE1**
#. belle1, made some progress but balked at having to downgrade 0.11 to 0.10.5 (and expectation that this would entail lots of trac config effort)
cms01 as stand-in server ?
===========================
#. try to apache-edit to use existing trac+svn install on cms01
* cms01 config has been changed for heprez
::
[blyth@cms01 ~]$ sudo apachectl stop
Syntax error on line 17 of /etc/httpd/conf/svnsetup/repos.conf:
Unknown DAV provider: svn
[blyth@cms01 ~]$ sudo apachectl start
Syntax error on line 17 of /etc/httpd/conf/svnsetup/repos.conf:
Unknown DAV provider: svn
[blyth@cms01 conf.d]$ ll /usr/lib/httpd/modules/*dav*
-rwxr-xr-x 1 root root 39324 Oct 25 2011 /usr/lib/httpd/modules/mod_dav_fs.so
-rwxr-xr-x 1 root root 80316 Oct 25 2011 /usr/lib/httpd/modules/mod_dav.so
-rwxr-xr-x 1 root root 426545 Dec 31 2007 /usr/lib/httpd/modules/mod_dav_svn.so
[blyth@cms01 ~]$ ll /usr/lib/httpd/modules/*svn*
-rwxr-xr-x 1 root root 29087 Dec 31 2007 /usr/lib/httpd/modules/mod_authz_svn.so
-rwxr-xr-x 1 root root 426545 Dec 31 2007 /usr/lib/httpd/modules/mod_dav_svn.so
Key precursor for apache config is ``svnsetup-``, mismatch::
[Wed May 02 17:36:03 2012] [notice] Apache/2.0.52 (Scientific Linux) configured -- resuming normal operations
[Wed May 02 17:36:19 2012] [error] sys:1: RuntimeWarning: Python C API version mismatch for module _apache: This Python has API version 1013, module _apache has version 1012.
[Wed May 02 17:36:19 2012] [error] make_obcallback: could not import mod_python.apache.\n
belle1 as stand-in : balked at trac downgrade 0.11 to 0.10.5
==============================================================
EPEL distro trac 0.10.5
------------------------
Yum epel based installation on belle1 : lighttpd/fastcgi/trac + apache/mod_dav_svn + ...
* http://trac.edgewall.org/wiki/0.11/TracUpgrade claims downgrade from 0.11 to 0.10.5 "is easy"
::
[blyth@belle1 ~]$ sudo yum --enablerepo=epel deplist trac
::
sqlite3 db/trac.db "update system set value=20 where name='database_version'"
EPEL distro config is not very smooth::
[blyth@belle1 trac-0.10.5]$ cat /etc/httpd/conf.d/trac.conf
# Replace all occurrences of /srv/trac with your trac root below
# and uncomment the respective SetEnv and PythonOption directives.
#SetEnv TRAC_ENV /srv/trac
SetHandler mod_python
PythonHandler trac.web.modpython_frontend
#PythonOption TracEnv /srv/trac
[blyth@belle1 trac-0.10.5]$
lighttpd fastcgi
------------------
Maybe can avoid apache with lighttpd::
[blyth@belle1 httpd]$ sudo yum --enablerepo=epel install lighttpd
[blyth@belle1 httpd]$ sudo yum --enablerepo=epel install lighttpd-fastcgi
[blyth@belle1 ~]$ rpm -ql trac | grep trac.fcgi
/var/www/cgi-bin/trac.fcgi
moving trac fast cgi into fastcgi avoids error,
2012-05-02 19:49:23: (server.c.1512) server stopped by UID = 0 PID = 17722
2012-05-02 19:49:26: (log.c.166) server started
but no show, nope need to
::
include "conf.d/fastcgi.conf"
gets to http://belle1.nuu.edu.tw/trac/ with::
Internal Error
The user root requires read _and_ write permission to the database file
/var/scm/tracs/test/db/trac.db and the directory it is located in.
Dealing with that::
[blyth@belle1 httpd]$ lighttpd-chown /var/scm/tracs/test
=== lighttpd-chown : sudo chown lighttpd:lighttpd -R /var/scm/tracs/test
gets to raw trac : using lighttpd + fastcgi on the test instance at
* http://belle1.nuu.edu.tw/trac/
apache mod_dav_svn
-------------------
unfortunately cannot do away with apache, as need http access to repo with **mod_dav_svn**
distinct possibility this will mess up the trac install from epel ? Comparing deplist
yields hope it will work::
[blyth@belle1 ~]$ sudo yum deplist mod_dav_svn
[blyth@belle1 ~]$ sudo yum deplist trac
Both **.so** from **mod_dav_svn**::
[blyth@belle1 ~]$ rpm -ql mod_dav_svn
/etc/httpd/conf.d/subversion.conf
/usr/lib/httpd/modules/mod_authz_svn.so
/usr/lib/httpd/modules/mod_dav_svn.so
Problem with using lighttpd/fastcgi/trac + apache/mod_dav_svn is port clash
change the port for apache : as main use if via SVN when ugly URLs are not as glaring as with trac::
[blyth@belle1 ~]$ sudo /sbin/service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
Open the port for me::
IPTABLES_PORT=8080 iptables-webopen-ip 140.112.102.77
Succeeds to make revision 0 of test repo visible
* http://belle1.nuu.edu.tw:8080/repos/test/
Can checkout::
g4pb-2:~ blyth$ svn co http://belle1.nuu.edu.tw:8080/repos/test
Checked out revision 0.
g4pb-2:~ blyth$
g4pb-2:~ blyth$ cd test
g4pb-2:test blyth$ svn st
g4pb-2:test blyth$ svn info
Path: .
Working Copy Root Path: /Users/blyth/test
URL: http://belle1.nuu.edu.tw:8080/repos/test
Repository Root: http://belle1.nuu.edu.tw:8080/repos/test
Repository UUID: 99398d55-88de-4a62-90a3-7cc4ad44927c
Revision: 0
Node Kind: directory
Schedule: normal
Last Changed Rev: 0
Last Changed Date: 2012-05-02 19:31:08 +0800 (Wed, 02 May 2012)
But checkin fails::
g4pb-2:test blyth$ svn add check.txt
A check.txt
g4pb-2:test blyth$ svn ci -m "test mod_dav_svn on belle1 "
svn: E000013: Commit failed (details follow):
svn: E000013: Can't create directory '/var/scm/repos/test/db/transactions/0-1.txn': Permission denied
Fixing apache ownership allows checkin to work::
[blyth@belle1 conf.d]$ apache-chown /var/scm/repos -R
=== apache-chown : sudo chown -R apache:apache /var/scm/repos
g4pb-2:test blyth$ svn ci -m "test mod_dav_svn on belle1 "
Adding check.txt
Transmitting file data .
Committed revision 1.
Visible via mod_dav_svn and trac:
* http://belle1.nuu.edu.tw:8080/repos/test/check.txt
* http://belle1.nuu.edu.tw/trac/browser/check.txt
belle1 next
-------------
* multi "tracs" and "repos"
* authentication + authorization
* trac plugins : which extensions are really needed ? 0.10.4 probably means some difficulties
* http://trac.edgewall.org/wiki/TracFastCgi#SimpleLighttpdConfiguration
change to TRAC_ENV_PARENT_DIR in /etc/lighttpd/conf.d/fastcgi.conf succeeds to list projects and serve them in extensible manner
* http://belle1.nuu.edu.tw/tracs/ "Available Projects"
* http://belle1.nuu.edu.tw/tracs/test/timeline
~/env/trac/tracdep.bash I have played with trac fastcgi previously it seems : with trac 0.11
generalize svnsetup- to the lighttpd/fastcgi/trac + apache/mod_dav_svn
and get working with epel yum trac 10.4 + corresponding AccountManager plugin
::
svnsetup-repos- anon-or-real repos YES
trac version issues
---------------------
http://trac-hacks.org/wiki/AccountManagerPlugin
(before Trac 0.11 this has been a separate trac:WebAdmin plugin)
need users and authz files to proceed, so transfer the backups::
[blyth@cms01 scm]$ rsync -e ssh -razvt /data/var/scm/backup/cms02 belle1.nuu.edu.tw:/var/scm/backup/
furnish these from backup with::
svnsetup-; svnsetup-from-backup-bootstrap
now get permission denied for http://belle1.nuu.edu.tw:8080/repos/test/
cms02 source build
===================
replacement cms02 evaluation
-------------------------------
#. replacement cms02 network accessible + setup sudoer user account
#. setup ssh keys, affording single keystoke login::
g4pb-2:~ blyth$ ssh--putkey cms02.phys.ntu.edu.tw
#. check what we have, hmm older than cms01 ... means a source build of trac for a reasonable version::
[blyth@hfag blyth]$ cat /etc/redhat-release
Scientific Linux SL release 3.0.9 (SL)
[blyth@cms02 ~]$ cat /etc/redhat-release
Scientific Linux SL release 4.5 (Beryllium)
[blyth@cms01 ~]$ cat /etc/redhat-release
Scientific Linux CERN SLC release 4.8 (Beryllium)
[blyth@belle7 repos]$ cat /etc/redhat-release
Scientific Linux SL release 5.1 (Boron)
#. check distro versions
* trac version from EPEL for SL4 is 0.9.3 ``sudo yum --enablerepo=epel info trac`` : too old for compatibility
basic yum installs
-------------------
The former **hep6** has nowt::
sudo yum install gcc
sudo yum install gcc-c++
sudo yum install curl
sudo yum install zlib-devel
tracpreq-again-source one-by-one
----------------------------------
doing the **tracpreq-again-source** one-by-one::
t tracpreq-again-source
tracpreq-again-source is a function
tracpreq-again-source ()
{
local msg="=== $FUNCNAME :";
[ "$(tracpreq-mode)" != "source" ] && echo $msg ABORT this is for tracpreq-mode:source only && return 1;
local ans;
read -p "$msg ENTER YES TO PROCEED" ans;
[ "$ans" != "YES" ] && echo $msg skipping && return 1;
log-;
log-init $FUNCNAME;
python-;
pythonbuild-;
pythonbuild-again | log-- $FUNCNAME pythonbuild-again;
configobj-;
configobj-get | log-- $FUNCNAME configobj-get;
swig-;
swigbuild-;
swigbuild-again | log-- $FUNCNAME swigbuild-again;
apache-;
apache-again | log-- $FUNCNAME apache-again;
svn-;
svnbuild-;
svnbuild-again | log-- $FUNCNAME svnbuild-again;
sqlite-;
sqlite-again | log-- $FUNCNAME sqlite-again
}
Mis-ordering in the build
---------------------------
Hmm ordering seems wrong,
#. configobj-get requires svn
#. swigbuild-get requires svn
subversion : configure failures
----------------------------------
configuring svn::
checking zlib.h presence... no
checking for zlib.h... no
configure: error: subversion requires zlib
make: *** No targets specified and no makefile found. Stop.
make: *** No rule to make target `install'. Stop.
Can't open Makefile: No such file or directory.
Can't open Makefile: No such file or directory.
diff: Makefile.orig: No such file or directory
diff: Makefile: No such file or directory
make: *** No rule to make target `swig-py'. Stop.
make: *** No rule to make target `install-swig-py'. Stop.
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named svn
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named svn
Need the ``zlib-devel``::
[blyth@cms02 subversion-1.4.6]$ rpm -ql zlib
/usr/lib64/libz.so.1
/usr/lib64/libz.so.1.2.1.2
/usr/share/doc/zlib-1.2.1.2
/usr/share/doc/zlib-1.2.1.2/README
/usr/lib/libz.so.1
/usr/lib/libz.so.1.2.1.2
/usr/share/doc/zlib-1.2.1.2
/usr/share/doc/zlib-1.2.1.2/README
[blyth@cms02 subversion-1.4.6]$
Missing a ``--shared`` in subversion config::
cd subversion/libsvn_ra_dav && /bin/sh /data/env/system/svn/build/subversion-1.4.6/libtool --tag=CC --silent --mode=link gcc -g -O2 -g -O2 -pthread -L/data/env/system/svn/build/subversion-1.4.6/apr-util/xml/expat/lib -rpath /data/env/system/svn/subversion-1.4.6/lib -o libsvn_ra_dav-1.la commit.lo fetch.lo file_revs.lo log.lo merge.lo options.lo props.lo replay.lo session.lo util.lo ../../subversion/libsvn_delta/libsvn_delta-1.la ../../subversion/libsvn_subr/libsvn_subr-1.la /data/env/system/svn/build/subversion-1.4.6/apr-util/libaprutil-0.la /data/env/system/svn/build/subversion-1.4.6/apr-util/xml/expat/lib/libexpat.la /data/env/system/svn/build/subversion-1.4.6/apr/libapr-0.la -lrt -lm -lcrypt -lnsl -lpthread -ldl /data/env/system/svn/build/subversion-1.4.6/neon/src/libneon.la -lz
/usr/bin/ld: /data/env/system/svn/build/subversion-1.4.6/neon/src/.libs/libneon.a(ne_request.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/data/env/system/svn/build/subversion-1.4.6/neon/src/.libs/libneon.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [subversion/libsvn_ra_dav/libsvn_ra_dav-1.la] Error 1
* http://svn.apache.org/repos/asf/subversion/site/publish/faq.html#relocation-against-local-symbol
subversion : Link fails
------------------------
Ye olde ``gssapi_krb5`` kludge strikes again::
-0.la /data/env/system/svn/build/subversion-1.4.6/apr-util/xml/expat/lib/libexpat.la /data/env/system/svn/build/subversion-1.4.6/apr/libapr-0.la -lrt -lm -lcrypt -lnsl -lpthread -ldl -L/usr/lib -lgssapi_krb5 -L/usr/lib -lgssapi_krb5 -lz
/usr/bin/ld: cannot find -lgssapi_krb5
collect2: ld returned 1 exit status
Backup Transfer
----------------
Transfer the cms01 backups to the new cms02::
[blyth@cms01 ~]$ rsync -av /data/var/scm/backup/cms02 cms02.phys.ntu.edu.tw:/var/scm/backup/
blyth@cms02.phys.ntu.edu.tw's password:
pysqlite : later handled by fixing python build
-------------------------------------------------
pysqlite failing... is it really needed?::
building 'pysqlite2._sqlite' extension
creating build/temp.linux-x86_64-2.5
creating build/temp.linux-x86_64-2.5/src
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DMODULE_NAME="pysqlite2.dbapi2" -DSQLITE_OMIT_LOAD_EXTENSION=1 -I/data/env/system/python/Python-2.5.6/include/python2.5 -c src/module.c -o build/temp.linux-x86_64-2.5/src/module.o
In file included from src/module.c:24:
src/connection.h:33:21: sqlite3.h: No such file or directory
In file included from src/module.c:24:
src/connection.h:38: error: syntax error before "sqlite3"
src/connection.h:38: warning: no semicolon at end of struct or union
from http://trac.edgewall.org/wiki/0.11/TracInstall maybe not::
If you're using Python 2.3 or 2.4 and need pysqlite,
tracbuild-auto
-----------------
Launch tracbuild-auto, abort as find no easy_install or setuptools::
Checked out revision 4117.
Traceback (most recent call last):
File "setup.py", line 9, in
from setuptools import setup
ImportError: No module named setuptools
=== package-look-version : version in the setup trunk/setup.py
Try ``setuptools-get`` but meet zlib issue, the zlib-devel was not available when building python::
[blyth@cms02 bitextra]$ setuptools-get
python ez_setup.py ... from /tmp/env/setuptools
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg
Traceback (most recent call last):
File "ez_setup.py", line 278, in
main(sys.argv[1:])
File "ez_setup.py", line 212, in main
from setuptools.command.easy_install import main
zipimport.ZipImportError: can't decompress data; zlib not available
[blyth@cms02 bitextra]$
pythonbuild
-------------
back to pythonbuild-configure then make, but think this was not needed... should have just done make : the zlib handling is done in setup.py after interpreter created not at config level ?
many curses errors from python make::
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_curses' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/data/env/system/python/build/Python-2.5.6/./Include -I/data/env/system/python/Python-2.5.6/include -I. -IInclude -I./Include -I/usr/local/include -I/data/env/system/python/build/Python-2.5.6/Include -I/data/env/system/python/build/Python-2.5.6 -c /data/env/system/python/build/Python-2.5.6/Modules/_cursesmodule.c -o build/temp.linux-x86_64-2.5/data/env/system/python/build/Python-2.5.6/Modules/_cursesmodule.o
In file included from /data/env/system/python/build/Python-2.5.6/Modules/_cursesmodule.c:113:
/data/env/system/python/build/Python-2.5.6/./Include/py_curses.h:45:20: curses.h: No such file or directory
In file included from /data/env/system/python/build/Python-2.5.6/Modules/_cursesmodule.c:113:
/data/env/system/python/build/Python-2.5.6/./Include/py_curses.h:73: error: syntax error before "WINDOW"
/data/env/system/python/build/Python-2.5.6/./Include/py_curses.h:73: warning: no semicolon at end of struct or union
eliminated after::
sudo yum install ncurses
sudo yum install ncurses-devel
still one complaint from make::
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
running build_scripts
aftre python ``make install`` ``setuptools-get`` succeeds
back to ``tracbuild-auto`` run into lack of tracdev (duh the server is dead ... need to skip bitextra)::
=== package-get : brn:trunk bnm:trunk pkt:svn tba:trunk url:http://dayabay.phys.ntu.edu.tw/repos/tracdev/annobit/trunk
=== package-get : svn checkout http://dayabay.phys.ntu.edu.tw/repos/tracdev/annobit/trunk rev 123 into /data/env/local/env/trac/package/bitextra with basename trunk
svn: PROPFIND request failed on '/repos/tracdev/annobit/trunk'
svn: PROPFIND of '/repos/tracdev/annobit/trunk': could not connect to server (http://dayabay.phys.ntu.edu.tw)
=== package-get : ABORT failed to checkout ...
skip packages requireing tracdev via temporary hiding::
[blyth@cms02 package]$ mv bitextra.bash.tmp-hide tmp-hide/bitextra.bash
[blyth@cms02 package]$ mv trac2latex.bash tmp-hide/
[blyth@cms02 package]$ mv trac2mediawiki.bash tmp-hide/
[blyth@cms02 package]$ l tmp-hide/
total 24
-rw-r--r-- 1 blyth blyth 2681 Jan 10 09:30 bitextra.bash
-rw-r--r-- 1 blyth blyth 3624 Jan 10 09:30 trac2latex.bash
-rw-r--r-- 1 blyth blyth 3090 Jan 10 09:30 trac2mediawiki.bash
issue with docutils, move it out of way also, install it ordinarily env/python/docutils.bash rather than as a trac package
cms02 : tarball recovery
===========================
Try ``scm-backup-; scm-recover-all cms02``::
=== scm-backup-synctrac : resyncing the instance with the repository ... as repository_dir has changed ... avoiding the yellow banner
=== trac-admin- : trac-admin : /data/env/system/python/Python-2.5.6/bin/trac-admin
=== trac-admin- : python : /data/env/system/python/Python-2.5.6/bin/python
=== trac-admin-
LLP
/data/env/system/sqlite/sqlite-3.3.16/lib
/data/env/system/svn/subversion-1.4.0/lib/svn-python/libsvn
/data/env/system/svn/subversion-1.4.0/lib/svn-python/svn
python: error while loading shared libraries: libpython2.5.so.1.0: cannot open shared object file: No such file or directory
=== trac-admin- : ABORT non-supported sqlite/pysqlite version ... see http://dayabay.phys.ntu.edu.tw/tracs/env/wiki/TracSQLiteMemoryExhaustion
env-abort
=== env-abort : ABORT ... sleeping forever
Reproducible::
[blyth@cms02 ~]$ trac-
[blyth@cms02 ~]$ TRAC_INSTANCE=data trac-admin-- resync
Password:
=== trac-admin- : trac-admin : /data/env/system/python/Python-2.5.6/bin/trac-admin
=== trac-admin- : python : /data/env/system/python/Python-2.5.6/bin/python
=== trac-admin-
LLP
/data/env/system/sqlite/sqlite-3.3.16/lib
/data/env/system/svn/subversion-1.4.0/lib/svn-python/libsvn
/data/env/system/svn/subversion-1.4.0/lib/svn-python/svn
python: error while loading shared libraries: libpython2.5.so.1.0: cannot open shared object file: No such file or directory
=== trac-admin- : ABORT non-supported sqlite/pysqlite version ... see http://dayabay.phys.ntu.edu.tw/tracs/env/wiki/TracSQLiteMemoryExhaustion
env-abort
=== env-abort : ABORT ... sleeping forever
python sqlite issue
---------------------
Seems the pythonbuild did not find the sqlite, both an order problem and fact that the python setup.py looks only in hardcoded locations for
sqlite headers. Kludge the ``setup.py`` of the python build with added path::
sqlite_inc_paths = [ '/usr/include',
'/usr/include/sqlite',
'/usr/include/sqlite3',
'/usr/local/include',
'/usr/local/include/sqlite',
'/usr/local/include/sqlite3',
'/data/env/system/sqlite/sqlite-3.3.16/include',
]
pythonbuild-cd
make
make install
This enables the version check to work::
trac-admin-sqlite-check-
sqlite_version_string:3.3.16 have_pysqlite:2
The ``sudo bash`` environment has the wrong **SVN_HOME**::
[blyth@cms02 Python-2.5.6]$ sudo bash -c "export ENV_HOME=$ENV_HOME ; . $ENV_HOME/env.bash ; env- ; echo \$SVN_HOME "
/data/env/system/svn/subversion-1.4.0
[blyth@cms02 Python-2.5.6]$ echo $SVN_HOME
/data/env/system/svn/subversion-1.4.6
Needed to set versions for NODE_TAG C2R to get correct paths, looked like bug with ``python-path`` but was not, just the C2R issue.
recover-all has some fails in **Resyncing repository history...** namely::
Command failed: /var/scm/svn/dybaux does not appear to be a Subversion repository.
Command failed: /var/scm/svn/dybsvn does not appear to be a Subversion repository.
Command failed: /var/scm/svn/toysvn does not appear to be a Subversion repository.
which correspond to traconlyies(how come?) in backup::
[blyth@cms02 scm]$ l backup/cms02/repos/
total 48
drwxr-xr-x 6 blyth blyth 4096 May 1 13:02 aberdeen
drwxr-xr-x 6 blyth blyth 4096 May 1 13:02 data
drwxr-xr-x 6 blyth blyth 4096 May 1 13:02 env
drwxr-xr-x 6 blyth blyth 4096 May 1 13:02 heprez
drwxr-xr-x 6 blyth blyth 4096 May 1 13:02 newtest
drwxr-xr-x 6 blyth blyth 4096 May 1 13:02 tracdev
[blyth@cms02 scm]$ l backup/cms02/tracs/
total 72
drwxr-xr-x 6 blyth blyth 4096 May 1 13:03 aberdeen
drwxr-xr-x 6 blyth blyth 4096 May 1 13:03 data
drwxr-xr-x 6 blyth blyth 4096 May 1 13:03 env
drwxr-xr-x 6 blyth blyth 4096 May 1 13:04 heprez
drwxr-xr-x 6 blyth blyth 4096 May 1 13:04 newtest
drwxr-xr-x 6 blyth blyth 4096 May 1 13:04 tracdev
drwxr-xr-x 4 blyth blyth 4096 May 1 13:04 toysvn
drwxr-xr-x 6 blyth blyth 4096 May 1 13:03 dybaux
drwxr-xr-x 3 blyth blyth 4096 Oct 17 2011 dybsvn
Also some broken link in htdocs issues::
=== apache-chown : sudo chown -R nobody:nobody heprez
/bin/chown: cannot dereference `heprez/htdocs/docs': No such file or directory
/bin/chown: cannot dereference `tracdev/htdocs/docs': No such file or directory
[blyth@cms02 scm]$ l tracs/*/htdocs/docs
lrwxrwxrwx 1 root root 48 May 4 20:12 tracs/heprez/htdocs/docs -> /data/usr/local/heprez/src/hfag/mods/webapp/docs
lrwxrwxrwx 1 root root 48 May 4 20:13 tracs/tracdev/htdocs/docs -> /data/usr/local/heprez/src/hfag/mods/webapp/docs
cms02 : apache hookup
======================
Next, getting SVN+trac hooked up to source apache with svnsetup-apache
SVN and Trac access working::
http://dayabay.phys.ntu.edu.tw/repos/env/trunk/
Service hookup with ``apache-initd`` symbolic linking to apachectl
Post commit failure::
g4pb-2:env blyth$ ci -m "bring belle1 N1 into the sshconf fold "
...
Transmitting file data .....
Committed revision 3444.
Warning: 'post-commit' hook failed with error output:
/var/scm/repos/env/hooks/post-commit: line 8: /data/env/system/python/Python-2.5.1/bin/python: No such file or directory
cms02 : backups + rsync + chkconfig
=====================================
C2R env hookup
----------------
Hookup **root@cms02** ``ssh C2R`` to env to do backups, in ``.bash_profile``::
export ENV_HOME=/home/blyth/env ; env-(){ [ -r $ENV_HOME/env.bash ] && . $ENV_HOME/env.bash && env-env $* ; }
#env-
scm-backup-all misses svnlook
-------------------------------
Wrong svn version dumped::
[root@cms02 ~]# scm-backup-all
=== scm-backup-all : starting from pwd /root
=== scm-backup-lock : /var/scm/LOCKED/scm-backup-all-started-2012-05-07@18:25:42
/data/env/system/python/Python-2.5.6/bin/python
/data/env/system/sqlite/sqlite-3.3.16/lib
/data/env/system/python/Python-2.5.6/lib
/data/env/system/svn/subversion-1.4.0/lib/svn-python/libsvn
/data/env/system/svn/subversion-1.4.0/lib/svn-python/svn
=== scm-backup-all : svn === skip non-folder /var/scm/svn/*
=== scm-backup-repo : name aberdeen path /var/scm/repos/aberdeen base /var/scm/backup/cms02 stamp 2012/05/07/182542 site ntu ===
=== scm-backup-repo : mkdir -p /var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542 && /data/env/system/svn/build/subversion-1.4.6/tools/backup/hot-backup.py --archive-type=gz /var/scm/repos/aberdeen /var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542 && cd /var/scm/backup/cms02/repos/aberdeen && rm -f last && ln -s 2012/05/07/182542 last
Beginning hot backup of '/var/scm/repos/aberdeen'.
Youngest revision is 1599
Backing up repository to '/var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542/aberdeen-1599'...
Done.
Archiving backup to '/var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542/aberdeen-1599.tar.gz'...
Archive created, removing backup '/var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542/aberdeen-1599'...
-bash: svnlook: command not found
tar: /var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542/aberdeen-.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error exit delayed from previous errors
=== scm-tgzcheck-ztvf : tgz /var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542/aberdeen-.tar.gz integrity check FAILURE 2
=== scm-backup-repo : tgz /var/scm/backup/cms02/repos/aberdeen/2012/05/07/182542/aberdeen-.tar.gz rev integrity check failure 2
[root@cms02 aberdeen]#
[root@cms02 aberdeen]#
scm-backup-rsync
-----------------
Generate ssh config and setup keys for auto rsync::
[root@cms02 ~]# sshconf-
[root@cms02 ~]# local-tags ## corresponds to backup nodes
C N H1
[root@cms02 ~]# sshconf-gen ## generate .ssh/config with sections for the local-tags
[root@cms02 .ssh]# mv id_rsa id_rsa.pub former/ ## move aside old keys that I do not has passphrases for
[root@cms02 .ssh]# ssh--keygen
[root@cms02 ~]# ssh--putkeys ## mammoth session of password and passphrase entry to all backup nodes
[root@cms02 ~]# ssh--agent-start
[root@cms02 ~]# scm-backup-rsync ## manual test of backup
Bogged down by big dybsvn tarballs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Move big fat foreigners into dedicated folder::
[blyth@cms02 scm]$ mkdir -p foreign/cms02/tracs
[blyth@cms02 scm]$ mv backup/cms02/tracs/toysvn foreign/cms02/tracs/
[blyth@cms02 scm]$ mv backup/cms02/tracs/dybsvn foreign/cms02/tracs/
[blyth@cms02 scm]$ mv backup/cms02/tracs/dybaux foreign/cms02/tracs/
Also remote DNA checking
^^^^^^^^^^^^^^^^^^^^^^^^^^
Restrict remote DNA check to the source ``LOCAL_NODE`` to avoid slow irrelevant DNA checks
Add belle1 to backup nodes
^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. add to ``local-tags`` , ``sshconf-gen`` adding to ssh config and ``ssh--putkey N1`` from **C2R**
DNA check failures from a disappearing L (python version difference?)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Digest matches but the size has lost an *L*::
=== scm-backup-dnachecktgzs : OK /var/scm/backup/cms02/tracs/env/2012/05/01/130104/env.tar.gz
1c1
< {'dig': '9e1a36c02cdb837c55404d38b33def8b', 'size': 53648743}
---
> {'dig': '9e1a36c02cdb837c55404d38b33def8b', 'size': 53648743L}
Reproduce::
[root@cms02 ~]# scm-backup-dnachecktgzs /var/scm/backup/cms02/repos/heprez
1c1
< {'dig': 'df6aa49ac8917b8d4144de5abb1a02cc', 'size': 4215148L}
---
> {'dig': 'df6aa49ac8917b8d4144de5abb1a02cc', 'size': 4215148}
=== scm-backup-dnachecktgzs : FAIL /var/scm/backup/cms02/repos/heprez/2012/05/01/130104/heprez-764.tar.gz
=== scm-backup-dnachecktgzs : OK /var/scm/backup/cms02/repos/heprez/2012/05/07/183131/heprez-764.tar.gz
1c1
< {'dig': '6a1024a2b03e128bfd65aad168ff3c90', 'size': 4214891L}
---
> {'dig': '6a1024a2b03e128bfd65aad168ff3c90', 'size': 4214891}
=== scm-backup-dnachecktgzs : FAIL /var/scm/backup/cms02/repos/heprez/2012/04/29/130108/heprez-764.tar.gz
1c1
< {'dig': '2f02f6b5e844ac07c665ebf82850e561', 'size': 4214780L}
---
> {'dig': '2f02f6b5e844ac07c665ebf82850e561', 'size': 4214780}
=== scm-backup-dnachecktgzs : FAIL /var/scm/backup/cms02/repos/heprez/2012/04/30/130103/heprez-764.tar.gz
[root@cms02 ~]#
[root@cms02 ~]#
[root@cms02 ~]#
[root@cms02 ~]# ll /var/scm/backup/cms02/repos/heprez/2012/05/07/183131/heprez-764.tar.gz.dna
-rw-r--r-- 1 root root 61 May 7 18:33 /var/scm/backup/cms02/repos/heprez/2012/05/07/183131/heprez-764.tar.gz.dna
[root@cms02 ~]# cat /var/scm/backup/cms02/repos/heprez/2012/05/07/183131/heprez-764.tar.gz.dna
{'dig': 'f3e450f405704c1cabb3f32f4196af96', 'size': 4226393}
[root@cms02 ~]#
Seems harmless, will go away once have all new ``.tar.gz.dna``
chkconfig setup : allowing auto-revival on reboot
----------------------------------------------------
Add chkconfig lines to::
[blyth@cms02 init.d]$ ls -l httpd
lrwxrwxrwx 1 root root 50 May 7 15:34 httpd -> /data/env/system/apache/httpd-2.0.64/bin/apachectl
::
# chkconfig: 345 95 05
# description: source apache serving Trac + SVN repositories via mod python and mod_dav_svn
cms02 chkconfig for auto-restart
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
June 20 2012, OOM killer httpd issue forced restart of C2, symptom: C2 is ping-able but not ssh-able.
Restart failed to auto-restart httpd. **Need to remember to chkconfig add**
::
[blyth@cms02 ~]$ sudo /sbin/chkconfig --list httpd
service httpd supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add httpd')
[blyth@cms02 ~]$
[blyth@cms02 ~]$ sudo /sbin/chkconfig --add httpd
[blyth@cms02 ~]$
[blyth@cms02 ~]$
[blyth@cms02 ~]$ sudo /sbin/chkconfig --list httpd
httpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
[blyth@cms02 ~]$
[blyth@cms02 ~]$ sudo /sbin/service httpd start
[blyth@cms02 ~]$
cron hookup + notification mail
---------------------------------
Adapt root and blyth cronlines from G::
SHELL = /bin/bash
59 13 * * * ( export HOME=/root ; export NODE=cms02 ; export MAILTO=blyth@hep1.phys.ntu.edu.tw ; export ENV_HOME=/home/blyth/env ; . /home/blyth/env/env.bash ; env- ; scm-backup- ; scm-backup-nightly ) > /var/scm/log/scm-backup-nightly.log 2>&1
hfag chkconfig : starts undesired services
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tidied up the extraneous apache links in /etc/init.d/
Recovering from Yet Another NTU Powercut, Thu 10 May 2012 ~13:30
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. cms01 : cannot access cms01, no ping::
simon:env blyth$ date
Thu 10 May 2012 14:17:36 CST
simon:env blyth$ ping cms01.phys.ntu.edu.tw
PING cms01.phys.ntu.edu.tw (140.112.101.190): 56 data bytes
#. from console, twas stuck at BIOS initialization ... powercycling regained access
#. usual manual mount::
[blyth@cms01 ~]$ sudo mount /data
#. do a manual ``exist-start`` as improper shutdown, this hangs ... but doing a exist-service-start succeeds
and the XMLDB is operational, succeeded to to a heprez-propagate to G for backup
#. hfag : again started too much, manually stop tomcat and exist, httpd OK::
[blyth@hfag blyth]$ sudo /sbin/service tomcat stop
[blyth@hfag blyth]$ sudo /sbin/service exist stop