Notes
======

tgz.py 
        provides `TGZ` :  interface between file system tgz directories and SQLite representation of that 





C2 backups sent to C
------------------------

Cron job runs on C2 at 16:10 TW time::

    [blyth@cms02 env]$ ll $LOCAL_BASE/env/scm/scm_backup_monitor.db
    -rw-r--r--  1 root root 1136640 Nov 13 16:17 /data/env/local/env/scm/scm_backup_monitor.db
    [blyth@cms02 env]$ 
    [blyth@cms02 env]$ date 
    Tue Nov 13 16:38:00 CST 2012

Starting point is the file system strings::

    [blyth@cms02 cms02]$ ll tracs/env/
    total 52
    drwxr-xr-x  10 blyth blyth 4096 Dec  1  2009 2009
    drwxr-xr-x  14 blyth blyth 4096 Dec  1  2010 2010
    drwxr-xr-x  14 blyth blyth 4096 Dec  1  2011 2011
    drwxr-xr-x   8 blyth blyth 4096 May  8  2012 ..
    drwxr-xr-x  13 blyth blyth 4096 Nov  1 16:12 2012
    lrwxrwxrwx   1 root  root    17 Nov 13 16:13 last -> 2012/11/13/161023
    drwxr-xr-x   6 blyth blyth 4096 Nov 13 16:13 .

    [blyth@cms02 cms02]$ find . -name '*.tar.gz' | grep 2012/11/13
    ./folders/svnsetup/2012/11/13/161023/svnsetup.tar.gz
    ./tracs/aberdeen/2012/11/13/161023/aberdeen.tar.gz
    ./tracs/tracdev/2012/11/13/161023/tracdev.tar.gz
    ./tracs/newtest/2012/11/13/161023/newtest.tar.gz
    ./tracs/env/2012/11/13/161023/env.tar.gz
    ./tracs/heprez/2012/11/13/161023/heprez.tar.gz
    ./repos/aberdeen/2012/11/13/161023/aberdeen-1789.tar.gz
    ./repos/tracdev/2012/11/13/161023/tracdev-125.tar.gz
    ./repos/newtest/2012/11/13/161023/newtest-20.tar.gz
    ./repos/data/2012/11/13/161023/data-23.tar.gz
    ./repos/env/2012/11/13/161023/env-3591.tar.gz
    ./repos/heprez/2012/11/13/161023/heprez-822.tar.gz


Which are parsed by `datepath.py` into datetime and written into SQLite DB as strings::


    g4pb-3:scm blyth$ sqlite3 $LOCAL_BASE/env/scm/scm_backup_monitor.db

    sqlite> select date, strftime("%s",date)*1000  from tgzs order by date desc limit 10 ;
    date                 strftime("%s",date)*1000
    -------------------  ------------------------
    2012-07-13T10:20:03  1342174803000           
    2012-07-13T10:20:03  1342174803000           
    2012-07-13T10:20:03  1342174803000           
    2012-07-13T10:20:03  1342174803000           
    2012-07-13T10:20:03  1342174803000           
    2012-07-13T10:20:03  1342174803000           
    2012-07-13T10:20:03  1342174803000           
    2012-07-12T14:10:03  1342102203000           
    2012-07-12T14:10:03  1342102203000           
    2012-07-12T14:10:03  1342102203000           


In the DB::

    [blyth@cms02 env]$ sqlite3 $LOCAL_BASE/env/scm/scm_backup_monitor.db
    SQLite version 3.3.6
    Enter ".help" for instructions
    sqlite> select count(*) from tgzs ;
    3616
    sqlite> select min(date), max(date) from tgzs ;
    2012-04-28T13:01:07|2012-11-13T16:10:23
    sqlite> 

    [blyth@cms02 cms02]$  sqlite3 $LOCAL_BASE/env/scm/scm_backup_monitor.db
    SQLite version 3.3.6
    Enter ".help" for instructions
    sqlite> select * from tgzs order by date desc limit 100 ;
    2012-11-13T16:10:23|C|2.0|/data/var/scm/backup/cms02/tracs/aberdeen|C:/data/var/scm/backup/cms02/tracs/aberdeen/2012/11/13/161023/aberdeen.tar.gz
    2012-11-13T16:10:23|C|9.0|/data/var/scm/backup/cms02/tracs/heprez|C:/data/var/scm/backup/cms02/tracs/heprez/2012/11/13/161023/heprez.tar.gz
    2012-11-13T16:10:23|C|1.0|/data/var/scm/backup/cms02/tracs/newtest|C:/data/var/scm/backup/cms02/tracs/newtest/2012/11/13/161023/newtest.tar.gz
    2012-11-13T16:10:23|C|52.0|/data/var/scm/backup/cms02/tracs/env|C:/data/var/scm/backup/cms02/tracs/env/2012/11/13/161023/env.tar.gz
    2012-11-13T16:10:23|C|1.0|/data/var/scm/backup/cms02/tracs/tracdev|C:/data/var/scm/backup/cms02/tracs/tracdev/2012/11/13/161023/tracdev.tar.gz
    2012-11-13T16:10:23|C|1.0|/data/var/scm/backup/cms02/folders/svnsetup|C:/data/var/scm/backup/cms02/folders/svnsetup/2012/11/13/161023/svnsetup.tar.gz
    2012-11-13T16:10:23|C|184.0|/data/var/scm/backup/cms02/repos/aberdeen|C:/data/var/scm/backup/cms02/repos/aberdeen/2012/11/13/161023/aberdeen-1789.tar.gz
    2012-11-13T16:10:23|C|1.0|/data/var/scm/backup/cms02/repos/data|C:/data/var/scm/backup/cms02/repos/data/2012/11/13/161023/data-23.tar.gz
    2012-11-13T16:10:23|C|5.0|/data/var/scm/backup/cms02/repos/heprez|C:/data/var/scm/backup/cms02/repos/heprez/2012/11/13/161023/heprez-822.tar.gz
    2012-11-13T16:10:23|C|4.0|/data/var/scm/backup/cms02/repos/newtest|C:/data/var/scm/backup/cms02/repos/newtest/2012/11/13/161023/newtest-20.tar.gz
    2012-11-13T16:10:23|C|10.0|/data/var/scm/backup/cms02/repos/env|C:/data/var/scm/backup/cms02/repos/env/2012/11/13/161023/env-3591.tar.gz
    2012-11-13T16:10:23|C|1.0|/data/var/scm/backup/cms02/repos/tracdev|C:/data/var/scm/backup/cms02/repos/tracdev/2012/11/13/161023/tracdev-125.tar.gz
    2012-11-13T16:10:23|H1|1.0|/home/hep/blyth/var/scm/backup/cms02/tracs/tracdev|H1:/home/hep/blyth/var/scm/backup/cms02/tracs/tracdev/2012/11/13/161023/tracdev.tar.gz
    2012-11-13T16:10:23|H1|2.0|/home/hep/blyth/var/scm/backup/cms02/tracs/aberdeen|H1:/home/hep/blyth/var/scm/backup/cms02/tracs/aberdeen/2012/11/13/161023/aberdeen.tar.gz
    2012-11-13T16:10:23|H1|52.0|/home/hep/blyth/var/scm/backup/cms02/tracs/env|H1:/home/hep/blyth/var/scm/backup/cms02/tracs/env/2012/11/13/161023/env.tar.gz
    2012-11-13T16:10:23|H1|9.0|/home/hep/blyth/var/scm/backup/cms02/tracs/heprez|H1:/home/hep/blyth/var/scm/backup/cms02/tracs/heprez/2012/11/13/161023/heprez.tar.gz
    2012-11-13T16:10:23|H1|1.0|/home/hep/blyth/var/scm/backup/cms02/tracs/newtest|H1:/home/hep/blyth/var/scm/backup/cms02/tracs/newtest/2012/11/13/161023/newtest.tar.gz
    2012-11-13T16:10:23|H1|1.0|/home/hep/blyth/var/scm/backup/cms02/folders/svnsetup|H1:/home/hep/blyth/var/scm/backup/cms02/folders/svnsetup/2012/11/13/161023/svnsetup.tar.gz
    2012-11-13T16:10:23|H1|1.0|/home/hep/blyth/var/scm/backup/cms02/repos/tracdev|H1:/home/hep/blyth/var/scm/backup/cms02/repos/tracdev/2012/11/13/161023/tracdev-125.tar.gz
    2012-11-13T16:10:23|H1|184.0|/home/hep/blyth/var/scm/backup/cms02/repos/aberdeen|H1:/home/hep/blyth/var/scm/backup/cms02/repos/aberdeen/2012/11/13/161023/aberdeen-1789.tar.gz
    2012-11-13T16:10:23|H1|1.0|/home/hep/blyth/var/scm/backup/cms02/repos/data|H1:/home/hep/blyth/var/scm/backup/cms02/repos/data/2012/11/13/161023/data-23.tar.gz
    2012-11-13T16:10:23|H1|10.0|/home/hep/blyth/var/scm/backup/cms02/repos/env|H1:/home/hep/blyth/var/scm/backup/cms02/repos/env/2012/11/13/161023/env-3591.tar.gz
    2012-11-13T16:10:23|H1|5.0|/home/hep/blyth/var/scm/backup/cms02/repos/heprez|H1:/home/hep/blyth/var/scm/backup/cms02/repos/heprez/2012/11/13/161023/heprez-822.tar.gz
    2012-11-13T16:10:23|H1|4.0|/home/hep/blyth/var/scm/backup/cms02/repos/newtest|H1:/home/hep/blyth/var/scm/backup/cms02/repos/newtest/2012/11/13/161023/newtest-20.tar.gz
    2012-11-12T16:10:25|C|2.0|/data/var/scm/backup/cms02/tracs/aberdeen|C:/data/var/scm/backup/cms02/tracs/aberdeen/2012/11/12/161025/aberdeen.tar.gz
    2012-11-12T16:10:25|C|9.0|/data/var/scm/backup/cms02/tracs/heprez|C:/data/var/scm/backup/cms02/tracs/heprez/2012/11/12/161025/heprez.tar.gz


SQLite not pulling any tricks::

    [blyth@cms02 cms02]$ echo "select strftime('%s','1970-01-01T00:00:00');" | sqlite3
    0


All so far in localtime

   http://dayabay.phys.ntu.edu.tw/data/scm_backup_monitor_C.json

Now in the JSON, something is shunting it forwards 8hrs (local interpreted as UTC)::

    imon:scm blyth$ ./tgzmon.py 
    INFO:__main__:series OK                   last [1352764800000L, 12] ts 1352764800 stamp 2012-11-13 08:00:00 
    INFO:__main__:series tracs/aberdeen       last [1352823023000L, 20.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series tracs/heprez         last [1352823023000L, 90.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series tracs/env            last [1352823023000L, 52.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series tracs/tracdev        last [1352823023000L, 10.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series repos/aberdeen       last [1352823023000L, 184.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series repos/heprez         last [1352823023000L, 50.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series repos/env            last [1352823023000L, 100.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:__main__:series repos/tracdev        last [1352823023000L, 10.0] ts 1352823023 stamp 2012-11-14 00:10:23 
    INFO:env.plot.highmon:no violations, not sending email
    
datetime.fromtimestamp returns local
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These ms timestamps get converted back to datetimes by::

    In [82]: datetime.fromtimestamp(1342102203000/1000)
    Out[82]: datetime.datetime(2012, 7, 12, 22, 10, 3)

    In [86]: datetime.fromtimestamp(0)               ## treats stamp as local
    Out[86]: datetime.datetime(1970, 1, 1, 8, 0)

    In [176]: datetime.utcfromtimestamp(0)           ## treats stamp as UTC 
    Out[176]: datetime.datetime(1970, 1, 1, 0, 0)



sqlite not offsetting
~~~~~~~~~~~~~~~~~~~~~~~~

Not from SQLite::

    sqlite> select strftime("%s","1970-01-01T00:00:00") ;
    strftime("%s","1970-01-01T00:00:00")
    ------------------------------------
    0                                   
    sqlite> select strftime("%s","1970-01-01T08:00:00") ;
    strftime("%s","1970-01-01T08:00:00")
    ------------------------------------
    28800