FIXED : OptiX 400 Interop Wierdness
========================================

TODO
----

check performance implications of OptiX zeroing ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Is OptiX zeroing significant to performance ? As is done in COMPUTE mode where not actually needed.
Can remove the OpZeroer call. 

TODO: formalize performance measurement into a test


FIXED : missing creation mode metadata
---------------------------------------------

RecordsNPYTest failing due to unset OpticksMode on loading an OpticksEvent, 
need to get **creation_mode** persisted and loaded::

    simon:optickscore blyth$ ls -l /tmp/blyth/opticks/evt/#/#/#/parameters.json
    -rw-r--r--  1 blyth  wheel   980 Aug 16 13:11 /tmp/blyth/opticks/evt/BoxInBox/mdtorch/1/parameters.json
    -rw-r--r--  1 blyth  wheel  1129 Aug 16 18:55 /tmp/blyth/opticks/evt/PmtInBox/mdtorch/-10/parameters.json
    -rw-r--r--  1 blyth  wheel  1059 Aug 17 14:17 /tmp/blyth/opticks/evt/PmtInBox/mdtorch/10/parameters.json
    -rw-r--r--  1 blyth  wheel   668 Aug 16 18:42 /tmp/blyth/opticks/evt/dayabay/mdcerenkov/1/parameters.json
    -rw-r--r--  1 blyth  wheel   618 Aug 17 14:42 /tmp/blyth/opticks/evt/dayabay/mdtorch/1/parameters.json
    -rw-r--r--  1 blyth  wheel  1173 Aug 16 13:03 /tmp/blyth/opticks/evt/prism/mdtorch/1/parameters.json
    -rw-r--r--  1 blyth  wheel  1144 Aug 16 21:07 /tmp/blyth/opticks/evt/reflect/mdtorch/1/parameters.json

There is already a **mode** entry::

    simon:optickscore blyth$ cat /tmp/blyth/opticks/evt/PmtInBox/mdtorch/10/parameters.json
    {
        "TimeStamp": "20160817_141731",
        "Type": "torch",
        "Tag": "10",
        "Detector": "dayabay",
        "Cat": "PmtInBox",
        "UDet": "PmtInBox",
        "RngMax": "3000000",
        "BounceMax": "9",
        "RecordMax": "10",
        "mode": "INTEROP_MODE",
        "cmdline": "--test --testconfig mode=PmtInBox_pmtpath=\/usr\/local\/opticks\/opticksdata\/export\/dpib\/GMergedMesh\/0_control=1,0,0,0_analytic=1_groupvel=0_shape=box_boundary=Rock\/NONE\/perfectAbsorbSurface\/MineralOil_parameters=0,0,0,300 --torch --torchconfig type=disc_photons=100000_wavelength=380_frame=1_source=0,0,300_target=0,0,0_radius=100_zenithazimuth=0,1,0,1_material=Vacuum_mode=_polarization= --timemax 10 --animtimemax 10 --cat PmtInBox --tag 10 --save --eye 0.0,-0.5,0.0 --geocenter ",
        "genstepDigest": "dca680dac412a91174a72ec35159800f",
        "NumGensteps": "1",
        "NumPhotons": "100000",
        "NumRecords": "1000000",
        "photonData": "012d3aab69d6d51fd68db4c4926df1a4",
        "recordData": "0b3e7ebd119fcbad77c79d3866fa1d9d",
        "sequenceData": "902d018daa9d9848921d538ef77a5b23"
    }




WORKAROUND : OptiX zeroing record buffer in generate.cu
-----------------------------------------------------------

* simply filling the record buffer with zeros
  at the start of the OptiX program, avoids the wierdness (swarming photons etc..)

  * so it seems that the OpZeroer is not working in interop
  * actually it is preferable to avoid the separate record zeroing step anyhow 

::

    124 void OpEngine::initRecords()
    125 {
    126     if(!m_evt) return ;
    127 
    128     if(!m_evt->isStep())
    129     {
    130         LOG(info) << "OpEngine::initRecords --nostep mode skipping " ;
    131         return ;
    132     }
    133 
    134 
    135     OContext* ocontext = m_imp->getOContext();
    136     OPropagator* opropagator = m_imp->getOPropagator();
    137 
    138     OpZeroer* zeroer = new OpZeroer(ocontext) ;
    139 
    140     zeroer->setEvent(m_evt);
    141     zeroer->setPropagator(opropagator);  // only used in compute mode
    142 
    143 
    144     if(m_opticks->hasOpt("dbginterop"))
    145     {
    146         LOG(info) << "OpEngine::initRecords skip OpZeroer::zeroRecords as dbginterop " ;
    147     }
    148     else
    149     {
    150         zeroer->zeroRecords();
    151         // zeros on GPU record buffer via OptiX or OpenGL
    152     }
    153 }



Debug Approaches
------------------

* need to do something technically similar to the 
  full interop simulation run, but drastically simpler :
  may be replace the generate.cu with an artificial propagation
  eg just straight line propagate initial photons

* seems only a subset of photons are afflicted, perhaps
  a problem with RNG ?

* loading of persisted evts seems working OK, 
  this just exercises OpenGL no interop needed

* check for buffer overwriting ... when using 
  OpenGL and OptiX together



OOAxisAppCheck 
~~~~~~~~~~~~~~~~


*oglrap-/AxisApp.cc*
      sets up OpenGL viz using a simple scene, frame, composition etc..

      BUT axis data is too simple for realistic debug, so add some fake
      nopstep or record data

*opticksgl-/OAxisTest.cc*
      sets up optixrap-/axisTest.cu

*opticksgl-/tests/OOAxisAppCheck.cc*
      sits in renderloop calling the axisModify via OptiX launch 
      


GGeoViewTest : Interop and Compute Mode not matching
-----------------------------------------------------------

Comparing interop with compute mode events, divergence is apparent::

   // interop
   GGeoViewTest 

   // compute 
   GGeoViewTest --compute --save
   GGeoViewTest --load


Prior to 400 there was precise digest matching agreement between 
interop and compute.  Now the compute mode looks normal but 
interop has several issues.

interop
~~~~~~~~~~

Notice some rafts of parallel slowly propagating photons.
Looking at photons in different history sequences suggests 
those ending in AB (bulk absorb) are primary mis-behavers.

This is suggestive of noise in the buffer.  Normally one would 
expect visualization of a buffer containing random noise bits to 
show little.

BUT for the domain compressed record buffer
practically any set of bits will produce something that will
be visible.



tpmt-- : origin attraction and swarming
------------------------------------------

interop
~~~~~~~~~~

Small numbers of slower photons seem attracted to origin, 
photons exhibit swarming 

compute
~~~~~~~~

None of the wierdness apparent on load, and matching g4::

    tpmt-- --compute 
    tpmt-- --load

    tpmt-- --compute --tcfg4


compute mode still matching g4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    simon:issues blyth$ tpmt.py 
    /Users/blyth/opticks/ana/tpmt.py
    writing opticks environment to /tmp/blyth/opticks/opticks_env.bash 
    INFO:__main__:tag 10 src torch det PmtInBox 
    INFO:__main__: a : PmtInBox/torch/ 10 :  20160816-1853 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/10.npy 
    INFO:__main__: b : PmtInBox/torch/-10 :  20160816-1855 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/-10.npy 
                          10:PmtInBox   -10:PmtInBox           c2 
                     8cd         67948        68252             0.68  [3 ] TO BT SA
                     7cd         21648        21369             1.81  [3 ] TO BT SD
                    8ccd          4581         4539             0.19  [4 ] TO BT BT SA
                      4d          3794         3864             0.64  [2 ] TO AB
                     86d           640          617             0.42  [3 ] TO SC SA
                     4cd           444          427             0.33  [3 ] TO BT AB
                    4ccd           350          362             0.20  [4 ] TO BT BT AB
                     8bd           283          259             1.06  [3 ] TO BR SA
                    8c6d            81           84             0.05  [4 ] TO SC BT SA
                   86ccd            51           57             0.33  [5 ] TO BT BT SC SA
                  8cbbcd            36           53             3.25  [6 ] TO BT BR BR BT SA
                     46d            40           30             1.43  [3 ] TO SC AB
                    7c6d            20           28             1.33  [4 ] TO SC BT SD
                     4bd            28           21             1.00  [3 ] TO BR AB
                8cbc6ccd             9            3             0.00  [8 ] TO BT BT SC BT BR BT SA
                    866d             8            4             0.00  [4 ] TO SC SC SA
                   8cc6d             7            7             0.00  [5 ] TO SC BT BT SA
                    86bd             6            4             0.00  [4 ] TO BR SC SA
                    8b6d             3            6             0.00  [4 ] TO SC BR SA
              cbccbbbbcd             4            0             0.00  [10] TO BT BR BR BR BR BT BT BR BT
                              100000       100000         0.91 
                          10:PmtInBox   -10:PmtInBox           c2 
                     ee4         90040        90048             0.00  [3 ] MO Py Py
                    44e4          4931         4901             0.09  [4 ] MO Py MO MO
                      44          3794         3864             0.64  [2 ] MO MO
                     444           991          927             2.14  [3 ] MO MO MO
                    ee44           101          113             0.67  [4 ] MO MO Py Py
                   444e4            52           58             0.33  [5 ] MO Py MO MO MO
                  44eee4            40           54             2.09  [6 ] MO Py Py Py MO MO
                    4444            17           14             0.29  [4 ] MO MO MO MO
                   44e44             8            7             0.00  [5 ] MO MO Py MO MO
                44ee44e4             6            3             0.00  [8 ] MO Py MO MO Py Py MO MO
                444e44e4             5            0             0.00  [8 ] MO Py MO MO Py MO MO MO
              44e4eeeee4             4            0             0.00  [10] MO Py Py Py Py Py MO Py MO MO
                  ee44e4             0            4             0.00  [6 ] MO Py MO MO Py Py
                   ee444             2            0             0.00  [5 ] MO MO MO Py Py
              44edbe44e4             2            0             0.00  [10] MO Py MO MO Py OV Vm Py MO MO
                  4444e4             0            2             0.00  [6 ] MO Py MO MO MO MO
              4ebdbe44e4             0            1             0.00  [10] MO Py MO MO Py OV Vm OV Py MO
              4e5dbe44e4             0            1             0.00  [10] MO Py MO MO Py OV Vm Bk Py MO
              eebdbe44e4             1            0             0.00  [10] MO Py MO MO Py OV Vm OV Py Py
                 44ee444             1            0             0.00  [7 ] MO MO MO Py Py MO MO
                              100000       100000         0.78 



FIXED : interop : fail to pullback/persist sequence buffer
---------------------------------------------------------------

After zeroing workaround the index seems operational and normal in GUI, 
but in analysis its empty::

    simon:ana blyth$ cd ~/opticks/ana ; ipython -i tevt.py -- --tag 10 --det PmtInBox
    Python 2.7.11 (default, Dec  5 2015, 23:51:51) 
    Type "copyright", "credits" or "license" for more information.

    IPython 1.2.1 -- An enhanced Interactive Python.
    ?         -> Introduction and overview of IPython's features.
    %quickref -> Quick reference.
    help      -> Python's own help system.
    object?   -> Details about 'object', use 'object??' for extra details.
    /Users/blyth/opticks/ana/tevt.py --tag 10 --det PmtInBox
    writing opticks environment to /tmp/blyth/opticks/opticks_env.bash 
    Evt( 10,"torch","PmtInBox","PmtInBox/torch/ 10 : ", seqs="[]") 20160817-1105 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/10.npy
     fdom :            (3, 1, 4) : (metadata) 3*float4 domains of position, time, wavelength (used for compression) 
     idom :            (1, 1, 4) : (metadata) int domain 
       ox :       (100000, 4, 4) : (photons) final photon step 
       wl :            (100000,) : (photons) wavelength 
     post :          (100000, 4) : (photons) final photon step: position, time 
     dirw :          (100000, 4) : (photons) final photon step: direction, weight  
     polw :          (100000, 4) : (photons) final photon step: polarization, wavelength  
    flags :            (100000,) : (photons) final photon step: flags  
       c4 :            (100000,) : (photons) final photon step: dtype split uint8 view of ox flags 
    rx_raw :   (100000, 10, 2, 4) : (records) photon step records RAW:before reshaping 
       rx :   (100000, 10, 2, 4) : (records) photon step records 
       ph :       (100000, 1, 2) : (records) photon history flag/material sequence 
       ps :       (100000, 1, 4) : (photons) phosel sequence frequency index lookups (uniques 34) 
       rs :   (100000, 10, 1, 4) : (records) RAW recsel sequence frequency index lookups (uniques 34) 
      rsr :   (100000, 10, 1, 4) : (records) RESHAPED recsel sequence frequency index lookups (uniques 34) 
                          10:PmtInBox 
                       0        1.000         100000       [1 ] ?0?
                              100000         1.00 
                          10:PmtInBox 
                       0        1.000         100000       [1 ] ?0?
                              100000         1.00 

    In [48]: evt.ph[:,0,0]
    Out[48]: 
    A()sliced
    A([0, 0, 0, ..., 0, 0, 0], dtype=uint64)

    In [49]: evt.ph[:,0,1]
    Out[49]: 
    A()sliced
    A([0, 0, 0, ..., 0, 0, 0], dtype=uint64)

    In [50]: np.unique(evt.ph[:,0,0])
    Out[50]: 
    A()sliced
    A([0], dtype=uint64)

    In [51]: np.unique(evt.ph[:,0,1])
    Out[51]: 
    A()sliced
    A([0], dtype=uint64)


OpticksEvent.cc sequence was recently changed to NON_INTEROP (ie pure OptiX buffer even in INTEROP mode)::

     563     m_sequence_spec = new NPYSpec(sequence_ ,  0,1,2,0,      NPYBase::ULONGLONG , "OPTIX_NON_INTEROP,OPTIX_OUTPUT_ONLY") ;
     564           
     565           // OPTIX_NON_INTEROP  : creates OptiX buffer even in INTEROP mode, this is possible for sequence as 
     566           //                      it is not used by OpenGL shaders so no need for INTEROP
     567           //
     568           //    ULONGLONG -> RT_FORMAT_USER  and size set to ni*nj*nk = num_photons*1*2
     569     

Need to accomodate this change in the downloading::

    1052 void App::saveEvt()
    1053 {
    1054     if(!m_ope) return ;
    1055     if(!isCompute())
    1056     {
    1057         Rdr::download(m_evt);
    1058     }
    1059     m_ope->saveEvt();
    1060 }



Fixed::

    simon:cfg4 blyth$ tpmt.py 
    /Users/blyth/opticks/ana/tpmt.py
    writing opticks environment to /tmp/blyth/opticks/opticks_env.bash 
    INFO:__main__:tag 10 src torch det PmtInBox 
    INFO:__main__: a : PmtInBox/torch/ 10 :  20160817-1417 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/10.npy 
    INFO:__main__: b : PmtInBox/torch/-10 :  20160816-1855 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/-10.npy 
                          10:PmtInBox   -10:PmtInBox           c2 
                     8cd         67948        68252             0.68  [3 ] TO BT SA
                     7cd         21648        21369             1.81  [3 ] TO BT SD
                    8ccd          4581         4539             0.19  [4 ] TO BT BT SA
                      4d          3794         3864             0.64  [2 ] TO AB
                     86d           640          617             0.42  [3 ] TO SC SA
                     4cd           444          427             0.33  [3 ] TO BT AB
                    4ccd           350          362             0.20  [4 ] TO BT BT AB
                     8bd           283          259             1.06  [3 ] TO BR SA
                    8c6d            81           84             0.05  [4 ] TO SC BT SA
                   86ccd            51           57             0.33  [5 ] TO BT BT SC SA
                  8cbbcd            36           53             3.25  [6 ] TO BT BR BR BT SA
                     46d            40           30             1.43  [3 ] TO SC AB
                    7c6d            20           28             1.33  [4 ] TO SC BT SD
                     4bd            28           21             1.00  [3 ] TO BR AB
                8cbc6ccd             9            3             0.00  [8 ] TO BT BT SC BT BR BT SA
                    866d             8            4             0.00  [4 ] TO SC SC SA
                   8cc6d             7            7             0.00  [5 ] TO SC BT BT SA
                    86bd             6            4             0.00  [4 ] TO BR SC SA
                    8b6d             3            6             0.00  [4 ] TO SC BR SA
              cbccbbbbcd             4            0             0.00  [10] TO BT BR BR BR BR BT BT BR BT
                              100000       100000         0.91 
                          10:PmtInBox   -10:PmtInBox           c2 
                     ee4         90040        90048             0.00  [3 ] MO Py Py
                    44e4          4931         4901             0.09  [4 ] MO Py MO MO
                      44          3794         3864             0.64  [2 ] MO MO
                     444           991          927             2.14  [3 ] MO MO MO
                    ee44           101          113             0.67  [4 ] MO MO Py Py
                   444e4            52           58             0.33  [5 ] MO Py MO MO MO
                  44eee4            40           54             2.09  [6 ] MO Py Py Py MO MO
                    4444            17           14             0.29  [4 ] MO MO MO MO
                   44e44             8            7             0.00  [5 ] MO MO Py MO MO
                44ee44e4             6            3             0.00  [8 ] MO Py MO MO Py Py MO MO
                444e44e4             5            0             0.00  [8 ] MO Py MO MO Py MO MO MO
              44e4eeeee4             4            0             0.00  [10] MO Py Py Py Py Py MO Py MO MO
                  ee44e4             0            4             0.00  [6 ] MO Py MO MO Py Py
                   ee444             2            0             0.00  [5 ] MO MO MO Py Py
              44edbe44e4             2            0             0.00  [10] MO Py MO MO Py OV Vm Py MO MO
                  4444e4             0            2             0.00  [6 ] MO Py MO MO MO MO
              4ebdbe44e4             0            1             0.00  [10] MO Py MO MO Py OV Vm OV Py MO
              4e5dbe44e4             0            1             0.00  [10] MO Py MO MO Py OV Vm Bk Py MO
              eebdbe44e4             1            0             0.00  [10] MO Py MO MO Py OV Vm OV Py Py
                 44ee444             1            0             0.00  [7 ] MO MO MO Py Py MO MO
                              100000       100000         0.78 
    simon:cfg4 blyth$ 




op --cerenkov
------------------

interop
~~~~~~~~

::

   op --cerenkov

10 percent (53474) of material sequence selection with NULL label, 
and slow backwards photons. 

Same number of missers (MI) in history selection. 

compute
~~~~~~~~~

::

    op --cerenkov --compute --save 
    op --cerenkov --load 


10 percent NULL still there, no visible photons