FIXED : OptiX 400 Interop Wierdness ======================================== TODO ---- check performance implications of OptiX zeroing ? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Is OptiX zeroing significant to performance ? As is done in COMPUTE mode where not actually needed. Can remove the OpZeroer call. TODO: formalize performance measurement into a test FIXED : missing creation mode metadata --------------------------------------------- RecordsNPYTest failing due to unset OpticksMode on loading an OpticksEvent, need to get **creation_mode** persisted and loaded:: simon:optickscore blyth$ ls -l /tmp/blyth/opticks/evt/#/#/#/parameters.json -rw-r--r-- 1 blyth wheel 980 Aug 16 13:11 /tmp/blyth/opticks/evt/BoxInBox/mdtorch/1/parameters.json -rw-r--r-- 1 blyth wheel 1129 Aug 16 18:55 /tmp/blyth/opticks/evt/PmtInBox/mdtorch/-10/parameters.json -rw-r--r-- 1 blyth wheel 1059 Aug 17 14:17 /tmp/blyth/opticks/evt/PmtInBox/mdtorch/10/parameters.json -rw-r--r-- 1 blyth wheel 668 Aug 16 18:42 /tmp/blyth/opticks/evt/dayabay/mdcerenkov/1/parameters.json -rw-r--r-- 1 blyth wheel 618 Aug 17 14:42 /tmp/blyth/opticks/evt/dayabay/mdtorch/1/parameters.json -rw-r--r-- 1 blyth wheel 1173 Aug 16 13:03 /tmp/blyth/opticks/evt/prism/mdtorch/1/parameters.json -rw-r--r-- 1 blyth wheel 1144 Aug 16 21:07 /tmp/blyth/opticks/evt/reflect/mdtorch/1/parameters.json There is already a **mode** entry:: simon:optickscore blyth$ cat /tmp/blyth/opticks/evt/PmtInBox/mdtorch/10/parameters.json { "TimeStamp": "20160817_141731", "Type": "torch", "Tag": "10", "Detector": "dayabay", "Cat": "PmtInBox", "UDet": "PmtInBox", "RngMax": "3000000", "BounceMax": "9", "RecordMax": "10", "mode": "INTEROP_MODE", "cmdline": "--test --testconfig mode=PmtInBox_pmtpath=\/usr\/local\/opticks\/opticksdata\/export\/dpib\/GMergedMesh\/0_control=1,0,0,0_analytic=1_groupvel=0_shape=box_boundary=Rock\/NONE\/perfectAbsorbSurface\/MineralOil_parameters=0,0,0,300 --torch --torchconfig type=disc_photons=100000_wavelength=380_frame=1_source=0,0,300_target=0,0,0_radius=100_zenithazimuth=0,1,0,1_material=Vacuum_mode=_polarization= --timemax 10 --animtimemax 10 --cat PmtInBox --tag 10 --save --eye 0.0,-0.5,0.0 --geocenter ", "genstepDigest": "dca680dac412a91174a72ec35159800f", "NumGensteps": "1", "NumPhotons": "100000", "NumRecords": "1000000", "photonData": "012d3aab69d6d51fd68db4c4926df1a4", "recordData": "0b3e7ebd119fcbad77c79d3866fa1d9d", "sequenceData": "902d018daa9d9848921d538ef77a5b23" } WORKAROUND : OptiX zeroing record buffer in generate.cu ----------------------------------------------------------- * simply filling the record buffer with zeros at the start of the OptiX program, avoids the wierdness (swarming photons etc..) * so it seems that the OpZeroer is not working in interop * actually it is preferable to avoid the separate record zeroing step anyhow :: 124 void OpEngine::initRecords() 125 { 126 if(!m_evt) return ; 127 128 if(!m_evt->isStep()) 129 { 130 LOG(info) << "OpEngine::initRecords --nostep mode skipping " ; 131 return ; 132 } 133 134 135 OContext* ocontext = m_imp->getOContext(); 136 OPropagator* opropagator = m_imp->getOPropagator(); 137 138 OpZeroer* zeroer = new OpZeroer(ocontext) ; 139 140 zeroer->setEvent(m_evt); 141 zeroer->setPropagator(opropagator); // only used in compute mode 142 143 144 if(m_opticks->hasOpt("dbginterop")) 145 { 146 LOG(info) << "OpEngine::initRecords skip OpZeroer::zeroRecords as dbginterop " ; 147 } 148 else 149 { 150 zeroer->zeroRecords(); 151 // zeros on GPU record buffer via OptiX or OpenGL 152 } 153 } Debug Approaches ------------------ * need to do something technically similar to the full interop simulation run, but drastically simpler : may be replace the generate.cu with an artificial propagation eg just straight line propagate initial photons * seems only a subset of photons are afflicted, perhaps a problem with RNG ? * loading of persisted evts seems working OK, this just exercises OpenGL no interop needed * check for buffer overwriting ... when using OpenGL and OptiX together OOAxisAppCheck ~~~~~~~~~~~~~~~~ *oglrap-/AxisApp.cc* sets up OpenGL viz using a simple scene, frame, composition etc.. BUT axis data is too simple for realistic debug, so add some fake nopstep or record data *opticksgl-/OAxisTest.cc* sets up optixrap-/axisTest.cu *opticksgl-/tests/OOAxisAppCheck.cc* sits in renderloop calling the axisModify via OptiX launch GGeoViewTest : Interop and Compute Mode not matching ----------------------------------------------------------- Comparing interop with compute mode events, divergence is apparent:: // interop GGeoViewTest // compute GGeoViewTest --compute --save GGeoViewTest --load Prior to 400 there was precise digest matching agreement between interop and compute. Now the compute mode looks normal but interop has several issues. interop ~~~~~~~~~~ Notice some rafts of parallel slowly propagating photons. Looking at photons in different history sequences suggests those ending in AB (bulk absorb) are primary mis-behavers. This is suggestive of noise in the buffer. Normally one would expect visualization of a buffer containing random noise bits to show little. BUT for the domain compressed record buffer practically any set of bits will produce something that will be visible. tpmt-- : origin attraction and swarming ------------------------------------------ interop ~~~~~~~~~~ Small numbers of slower photons seem attracted to origin, photons exhibit swarming compute ~~~~~~~~ None of the wierdness apparent on load, and matching g4:: tpmt-- --compute tpmt-- --load tpmt-- --compute --tcfg4 compute mode still matching g4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: simon:issues blyth$ tpmt.py /Users/blyth/opticks/ana/tpmt.py writing opticks environment to /tmp/blyth/opticks/opticks_env.bash INFO:__main__:tag 10 src torch det PmtInBox INFO:__main__: a : PmtInBox/torch/ 10 : 20160816-1853 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/10.npy INFO:__main__: b : PmtInBox/torch/-10 : 20160816-1855 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/-10.npy 10:PmtInBox -10:PmtInBox c2 8cd 67948 68252 0.68 [3 ] TO BT SA 7cd 21648 21369 1.81 [3 ] TO BT SD 8ccd 4581 4539 0.19 [4 ] TO BT BT SA 4d 3794 3864 0.64 [2 ] TO AB 86d 640 617 0.42 [3 ] TO SC SA 4cd 444 427 0.33 [3 ] TO BT AB 4ccd 350 362 0.20 [4 ] TO BT BT AB 8bd 283 259 1.06 [3 ] TO BR SA 8c6d 81 84 0.05 [4 ] TO SC BT SA 86ccd 51 57 0.33 [5 ] TO BT BT SC SA 8cbbcd 36 53 3.25 [6 ] TO BT BR BR BT SA 46d 40 30 1.43 [3 ] TO SC AB 7c6d 20 28 1.33 [4 ] TO SC BT SD 4bd 28 21 1.00 [3 ] TO BR AB 8cbc6ccd 9 3 0.00 [8 ] TO BT BT SC BT BR BT SA 866d 8 4 0.00 [4 ] TO SC SC SA 8cc6d 7 7 0.00 [5 ] TO SC BT BT SA 86bd 6 4 0.00 [4 ] TO BR SC SA 8b6d 3 6 0.00 [4 ] TO SC BR SA cbccbbbbcd 4 0 0.00 [10] TO BT BR BR BR BR BT BT BR BT 100000 100000 0.91 10:PmtInBox -10:PmtInBox c2 ee4 90040 90048 0.00 [3 ] MO Py Py 44e4 4931 4901 0.09 [4 ] MO Py MO MO 44 3794 3864 0.64 [2 ] MO MO 444 991 927 2.14 [3 ] MO MO MO ee44 101 113 0.67 [4 ] MO MO Py Py 444e4 52 58 0.33 [5 ] MO Py MO MO MO 44eee4 40 54 2.09 [6 ] MO Py Py Py MO MO 4444 17 14 0.29 [4 ] MO MO MO MO 44e44 8 7 0.00 [5 ] MO MO Py MO MO 44ee44e4 6 3 0.00 [8 ] MO Py MO MO Py Py MO MO 444e44e4 5 0 0.00 [8 ] MO Py MO MO Py MO MO MO 44e4eeeee4 4 0 0.00 [10] MO Py Py Py Py Py MO Py MO MO ee44e4 0 4 0.00 [6 ] MO Py MO MO Py Py ee444 2 0 0.00 [5 ] MO MO MO Py Py 44edbe44e4 2 0 0.00 [10] MO Py MO MO Py OV Vm Py MO MO 4444e4 0 2 0.00 [6 ] MO Py MO MO MO MO 4ebdbe44e4 0 1 0.00 [10] MO Py MO MO Py OV Vm OV Py MO 4e5dbe44e4 0 1 0.00 [10] MO Py MO MO Py OV Vm Bk Py MO eebdbe44e4 1 0 0.00 [10] MO Py MO MO Py OV Vm OV Py Py 44ee444 1 0 0.00 [7 ] MO MO MO Py Py MO MO 100000 100000 0.78 FIXED : interop : fail to pullback/persist sequence buffer --------------------------------------------------------------- After zeroing workaround the index seems operational and normal in GUI, but in analysis its empty:: simon:ana blyth$ cd ~/opticks/ana ; ipython -i tevt.py -- --tag 10 --det PmtInBox Python 2.7.11 (default, Dec 5 2015, 23:51:51) Type "copyright", "credits" or "license" for more information. IPython 1.2.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. /Users/blyth/opticks/ana/tevt.py --tag 10 --det PmtInBox writing opticks environment to /tmp/blyth/opticks/opticks_env.bash Evt( 10,"torch","PmtInBox","PmtInBox/torch/ 10 : ", seqs="[]") 20160817-1105 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/10.npy fdom : (3, 1, 4) : (metadata) 3*float4 domains of position, time, wavelength (used for compression) idom : (1, 1, 4) : (metadata) int domain ox : (100000, 4, 4) : (photons) final photon step wl : (100000,) : (photons) wavelength post : (100000, 4) : (photons) final photon step: position, time dirw : (100000, 4) : (photons) final photon step: direction, weight polw : (100000, 4) : (photons) final photon step: polarization, wavelength flags : (100000,) : (photons) final photon step: flags c4 : (100000,) : (photons) final photon step: dtype split uint8 view of ox flags rx_raw : (100000, 10, 2, 4) : (records) photon step records RAW:before reshaping rx : (100000, 10, 2, 4) : (records) photon step records ph : (100000, 1, 2) : (records) photon history flag/material sequence ps : (100000, 1, 4) : (photons) phosel sequence frequency index lookups (uniques 34) rs : (100000, 10, 1, 4) : (records) RAW recsel sequence frequency index lookups (uniques 34) rsr : (100000, 10, 1, 4) : (records) RESHAPED recsel sequence frequency index lookups (uniques 34) 10:PmtInBox 0 1.000 100000 [1 ] ?0? 100000 1.00 10:PmtInBox 0 1.000 100000 [1 ] ?0? 100000 1.00 In [48]: evt.ph[:,0,0] Out[48]: A()sliced A([0, 0, 0, ..., 0, 0, 0], dtype=uint64) In [49]: evt.ph[:,0,1] Out[49]: A()sliced A([0, 0, 0, ..., 0, 0, 0], dtype=uint64) In [50]: np.unique(evt.ph[:,0,0]) Out[50]: A()sliced A([0], dtype=uint64) In [51]: np.unique(evt.ph[:,0,1]) Out[51]: A()sliced A([0], dtype=uint64) OpticksEvent.cc sequence was recently changed to NON_INTEROP (ie pure OptiX buffer even in INTEROP mode):: 563 m_sequence_spec = new NPYSpec(sequence_ , 0,1,2,0, NPYBase::ULONGLONG , "OPTIX_NON_INTEROP,OPTIX_OUTPUT_ONLY") ; 564 565 // OPTIX_NON_INTEROP : creates OptiX buffer even in INTEROP mode, this is possible for sequence as 566 // it is not used by OpenGL shaders so no need for INTEROP 567 // 568 // ULONGLONG -> RT_FORMAT_USER and size set to ni*nj*nk = num_photons*1*2 569 Need to accomodate this change in the downloading:: 1052 void App::saveEvt() 1053 { 1054 if(!m_ope) return ; 1055 if(!isCompute()) 1056 { 1057 Rdr::download(m_evt); 1058 } 1059 m_ope->saveEvt(); 1060 } Fixed:: simon:cfg4 blyth$ tpmt.py /Users/blyth/opticks/ana/tpmt.py writing opticks environment to /tmp/blyth/opticks/opticks_env.bash INFO:__main__:tag 10 src torch det PmtInBox INFO:__main__: a : PmtInBox/torch/ 10 : 20160817-1417 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/10.npy INFO:__main__: b : PmtInBox/torch/-10 : 20160816-1855 /tmp/blyth/opticks/evt/PmtInBox/fdomtorch/-10.npy 10:PmtInBox -10:PmtInBox c2 8cd 67948 68252 0.68 [3 ] TO BT SA 7cd 21648 21369 1.81 [3 ] TO BT SD 8ccd 4581 4539 0.19 [4 ] TO BT BT SA 4d 3794 3864 0.64 [2 ] TO AB 86d 640 617 0.42 [3 ] TO SC SA 4cd 444 427 0.33 [3 ] TO BT AB 4ccd 350 362 0.20 [4 ] TO BT BT AB 8bd 283 259 1.06 [3 ] TO BR SA 8c6d 81 84 0.05 [4 ] TO SC BT SA 86ccd 51 57 0.33 [5 ] TO BT BT SC SA 8cbbcd 36 53 3.25 [6 ] TO BT BR BR BT SA 46d 40 30 1.43 [3 ] TO SC AB 7c6d 20 28 1.33 [4 ] TO SC BT SD 4bd 28 21 1.00 [3 ] TO BR AB 8cbc6ccd 9 3 0.00 [8 ] TO BT BT SC BT BR BT SA 866d 8 4 0.00 [4 ] TO SC SC SA 8cc6d 7 7 0.00 [5 ] TO SC BT BT SA 86bd 6 4 0.00 [4 ] TO BR SC SA 8b6d 3 6 0.00 [4 ] TO SC BR SA cbccbbbbcd 4 0 0.00 [10] TO BT BR BR BR BR BT BT BR BT 100000 100000 0.91 10:PmtInBox -10:PmtInBox c2 ee4 90040 90048 0.00 [3 ] MO Py Py 44e4 4931 4901 0.09 [4 ] MO Py MO MO 44 3794 3864 0.64 [2 ] MO MO 444 991 927 2.14 [3 ] MO MO MO ee44 101 113 0.67 [4 ] MO MO Py Py 444e4 52 58 0.33 [5 ] MO Py MO MO MO 44eee4 40 54 2.09 [6 ] MO Py Py Py MO MO 4444 17 14 0.29 [4 ] MO MO MO MO 44e44 8 7 0.00 [5 ] MO MO Py MO MO 44ee44e4 6 3 0.00 [8 ] MO Py MO MO Py Py MO MO 444e44e4 5 0 0.00 [8 ] MO Py MO MO Py MO MO MO 44e4eeeee4 4 0 0.00 [10] MO Py Py Py Py Py MO Py MO MO ee44e4 0 4 0.00 [6 ] MO Py MO MO Py Py ee444 2 0 0.00 [5 ] MO MO MO Py Py 44edbe44e4 2 0 0.00 [10] MO Py MO MO Py OV Vm Py MO MO 4444e4 0 2 0.00 [6 ] MO Py MO MO MO MO 4ebdbe44e4 0 1 0.00 [10] MO Py MO MO Py OV Vm OV Py MO 4e5dbe44e4 0 1 0.00 [10] MO Py MO MO Py OV Vm Bk Py MO eebdbe44e4 1 0 0.00 [10] MO Py MO MO Py OV Vm OV Py Py 44ee444 1 0 0.00 [7 ] MO MO MO Py Py MO MO 100000 100000 0.78 simon:cfg4 blyth$ op --cerenkov ------------------ interop ~~~~~~~~ :: op --cerenkov 10 percent (53474) of material sequence selection with NULL label, and slow backwards photons. Same number of missers (MI) in history selection. compute ~~~~~~~~~ :: op --cerenkov --compute --save op --cerenkov --load 10 percent NULL still there, no visible photons