.. comment Title: GPU Acceleration of Geant4 Simulations of the Daya Bay Anti-Neutrino Detector Abstract: Studies of the acceleration of Geant4 based Monte Carlo Simulations of the Daya Bay anti-neutrino detector using the Chroma GPU optical photon simulation package are presented. Original open source developments made to bring the Geant4 simulation context to the GPU are described. They include a Geant4 geometry exporter package (G4DAE) and a bridging package (G4DAEChroma) that integrates external optical photon propagation with standard Geant4 simulations. The G4DAE geometry exporter package is planned to be incorporated with the 2015 release of the Geant4 simulation toolkit. The export of triangulated geometry data into XML files following the standard COLLADA 3D specification was found to allow both massively parallel CUDA based optical photon propagation and high performance OpenGL visualizations of geometry and event data. 12 Jan 2015, Monday 17:05 .. raw:: html .. role:: i .. role:: alarm .. role:: warn .. role:: ok .. include:: .. s5_background_image:: # # slide titles and background image urls, # including server relative urls like /env/geant4/geometry/collada/daeview/20140419-170713.png # and protocol relative urls like //localhost/env/test/LANS_AD3_CoverGas_Humidity.png # # NB1 slide titles here must match those in body precisely, # NB2 also ensure all slide titles are unique # # first slide id is exceptionally: slide0, not the mangled title slide0 /env/geant4/geometry/collada/g4daeview/20140419-170713.png auto_auto 0px_0px g4daeview.py : Fast OpenGL 3D viewer for G4DAE files /env/geant4/geometry/collada/g4daeview/20140419-170713.png Cerenkov Photons Simulation - Top View /env/geant4/geometry/collada/g4daeview/20141224-115923.png Cerenkov Photons Simulation - Side View /env/geant4/geometry/collada/g4daeview/20141224-115935.png Scintillation Photons Simulation - Top View /env/geant4/geometry/collada/g4daeview/20141224-121444.png Scintillation Photons Simulation - Side View /env/geant4/geometry/collada/g4daeview/20141224-121435.png Standard Geant4 Workflow /env/keynotefigs/G4DAEChroma/G4DAEChroma.001.png External Photon Simulation Workflow /env/keynotefigs/G4DAEChroma/G4DAEChroma.002.png .. comment Generated Scintillation Photons GPU cf Geant4 /env/g4dae/generated_scintillation_time_wavelength.png G4/DetSim Generated Cerenkov Wavelength /env/g4dae/g4_cerenkov_wavelength.png ====================================================================== GPU Accelerated Geant4 Simulation with G4DAE and Chroma ====================================================================== .. class:: small http://simoncblyth.bitbucket.org/env/presentation/gpu_accelerated_geant4_simulation.html http://simoncblyth.bitbucket.org/env/presentation/g4dae_geometry_exporter.html http://simoncblyth.bitbucket.org/env/presentation/gpu_optical_photon_simulation.html .. sidebar:: Drastic 200x Potential Speedup Chroma project claims 200x speedup of optical photon (OP) propagation compared to Geant4. **A goal worthwhile chasing**. .. class:: small * Geometry Model Implications * Introducing Chroma * Geant4 <-> Chroma Integration * G4DAE Geometry Exporter * Validating GPU Geometry * G4DAEChroma bridge * Chroma Forked * Validating Chroma Generated Photons * Next Steps * Visualizations | Simon C Blyth, National Taiwan University | **January 2015** .. comment Title 00 : GPU Accelerated Geant4 Simulation with G4DAE and Chroma Geometry Model Implications 01 : implications of geometry model choice Introducing Chroma 02 : chroma : ultra-fast photon mc, developed by stan seibert, university of pennsylvania Geant4 <-> Chroma Integration 03 : standard geant4 workflow 04 : geant4 <-> chroma integration 05 : external photon simulation workflow G4DAE Geometry Exporter 06 : https://bitbucket.org/simoncblyth/g4dae 07 : collada : standard for 3d digital asset exchange (dae) 08 : liberated geometry allows ubiquitous high peformance visualization 09 : g4daeview.py : fast opengl 3d viewer for g4dae files Validating GPU Geometry 10 : compare dae exports to gdml and vrml2(wrl) 11 : chroma raycasting : exercises geometry intersection 12 : chroma raycast with entire geometry in view G4DAEChroma bridge 13 : g4daechroma : bridging from geant4 to chroma 14 : g4daechroma : for implementation details... Chroma Forked 15 : https://bitbucket.org/simoncblyth/chroma forked: Validating Chroma Generated Photons 16 : chroma generated scintillation photons cf geant4 17 : chroma generated cerenkov photons cf geant4 Next steps Visualizations 17 : cerenkov photons simulation - side view 18 : cerenkov photons simulation - top view 19 : scintillation photons simulation - top view 20 : scintillation photons simulation - side view Extras 21 : extra slides 22 : g4 generated cerenkov wavelengths in material categories 23 : refractive index interpolation explains jump at 200nm 24 : comparison of generated scintillation photon distributions 25 : comparison of generated cerenokov photon distributions 13 : g4daeview.py : dayabay chroma photon propagation (1) :small:`Implications of Geometry Model Choice` ------------------------------------------------------- :red:`Track > Geometry intersection` typically limits simulation performance. Geometry model determines techniques (and hardware) available to accelerate intersection. :green:`Geant4` Geometry Model (:blue:`solid based`, :red:`locked in`) *Tree of nested solids composed of materials, each shape represented by different C++ class* :green:`Chroma` Geometry Model (:blue:`surface based`, :red:`extremely simple > portable`) *List of oriented triangles, each representing boundary between inside and outside materials.* 3D industry focusses on surface models >> frameworks and GPU hardware designed to work with surface based geometries. :small:`Chroma : Ultra-fast Photon MC, Developed by Stan Seibert, University of Pennsylvania` ----------------------------------------------------------------------------------------------- .. class:: small * http://chroma.bitbucket.org Chroma tracks photons through a :red:`triangle-mesh detector geometry`, simulating processes like diffuse and specular reflections, refraction, Rayleigh scattering and absorption. Using triangle meshes eliminate geometry code as :red:`just one code path`. .. sidebar:: Original Motivation for G4DAE exporter Allow use of Chroma GPU accelerated optical photon propagation with existing DayaBay Geant4 geometry. .. class:: small 200x performance claim: *With a CUDA GPU Chroma has propagated 2.5M photons per second in a detector with 29k photomultiplier tubes. This is 200x faster than GEANT4.* .. class:: small **BUT** Chroma needs : :red:`triangles + inside/outside materials` .. class:: tiny http://on-demand.gputechconf.com/gtc/2013/presentations/S3304-Particle-Physics-With-PyCUDA.pdf https://bitbucket.org/chroma/chroma https://bitbucket.org/simoncblyth/chroma (my fork) Standard Geant4 Workflow -------------------------- .. class:: small :i:`Some invisible vertical space` .. class:: large :i:`.` :i:`.` :i:`.` OP Generated in Cerenkov and Scintillation G4VProcess, some reach PMTs and form hits. .. class:: tiny :i:`.` :i:`.` :i:`.` :i:`.` :i:`.` :: // G4VProcess subclass PostStepDoIt(const G4Track&, const G4Step&) { // calculate NumPhotons aParticleChange.SetNumberOfSecondaries(NumPhotons); // generate and collect secondaries for (G4int i = 0; i < NumPhotons; i++) { ... aParticleChange.AddSecondary(aSecondaryTrack); } } // G4VSensitiveDetector subclass bool ProcessHits(G4Step* step, ...) { // form hits from step information } Geant4 <-> Chroma Integration ---------------------------------------- .. sidebar:: Bridging to external OP simulation .. class:: small Optical photons (modulo reemission) are the leaves of the simulation tree, allowing external simulation to be integrated rather simply. .. class:: small Recreate Geometry info on GPU (:blue:`G4DAE`) * export triangulated geometry as COLLADA/DAE files * export PMT identifiers associating sensors with volumes * import geometry and identity files into Chroma and upload to GPU :red:`Runtime bridge` via message queue (:blue:`G4DAEChroma`) * collect generation step parameters and prevent photon processing by not forming secondaries * send generation steps to Chroma and receive hits in reply * include hits into standard Geant4 hit collections, subsequent simulation proceeds normally GPU Optical Photon simulation (:blue:`Chroma`) * receive generation steps from message queue * copy generation steps to GPU and allocate GPU memory for photons * invoke generation and propagation CUDA kernel * copy propagated photons back to CPU * send reply with hits back to :blue:`G4DAEChroma` External Photon Simulation Workflow ------------------------------------- https://bitbucket.org/simoncblyth/g4dae --------------------------------------------------------------------------- .. sidebar:: Inclusion into Geant4 .. class:: small Geometry exporter presented at Sept 2014 Geant4 Collaboration Meeting. Proposal to contribute was accepted, intended to be included with 2015 Geant4 release. .. class:: small Exports *Geant4* Geometry into COLLADA/DAE standard 3D files, based on *GDML writer* code, same *XercesC* dependency for XML handling. Export includes: * volume tree: solid/physical volume/logical volume heirarchy * geometry: vertices, triangles+quads (triangulation from *G4Polyhedron*) * materials and surfaces with properties as function of wavelength (using DAE *extra* XML elements) .. image:: /env/geant4/geometry/collada/g4dae/g4dae_bitbucket.png :width: 700px :align: center :small:`COLLADA : Standard for 3D Digital Asset Exchange (DAE)` ----------------------------------------------------------------- .. class:: small * https://www.khronos.org/collada/ .. image:: /env/graphics/collada/collada_khronos.png :width: 900px :align: center :small:`Liberated Geometry allows Ubiquitous High Peformance Visualization` ----------------------------------------------------------------------------- .. sidebar:: Geometry Set Free .. class:: small Liberating geometry data from Geant4/ROOT gives free choice of visualization packages. Many commercial, open source apps/libs provide high performance visualization of DAE files using GPU efficient OpenGL techniques. :red:`=> Shockingly Smooth Visualization performance` .. class:: small Commercial: * *Finder/Quicklook/Preview/Xcode* (standard OSX 10.8+) * :blue:`Xcode : useful effect tweaking interface` * :blue:`Preview : set black background for visibility` * *Sketchup 8* free (OSX) [also Windows] * *ESKO Studio Viewer* free (iOS on iPad) Open source: * *Meshlab* (OSX/Linux), slow DAE loading, OK interface [also Windows] * *Blender* (OSX), painful GUI [also Windows/Linux] * *g4daeview* (OSX/Linux), :red:`extremely useful for debugging` Frameworks: * *pycollada*, Python COLLADA parsing/construction, :red:`lxml + numpy powered fast parsing` * *threejs*, Javascript COLLADA loading, renders HTML canvas with WebGL * *SceneKit*, Objective C Framework used by OSX apps Many more applications and frameworks listed http://en.wikipedia.org/wiki/COLLADA .. class:: tiny http://meshlab.sourceforge.net https://bitbucket.org/simoncblyth/meshlab (fork improves COLLADA loading speed) :small:`g4daeview.py : Fast OpenGL 3D viewer for G4DAE files` ----------------------------------------------------------------------------- .. sidebar:: PyOpenGL Based .. class:: small Easy to extend due to fast python development, yet access to highly efficient OpenGL techniques such as VBOs, PBOs and GLSL shaders. .. class:: small G4DAE visualization [#g4daeview_usage]_ implemented with PyCollada, **PyOpenGL** and Glumpy * very fast/responsive 3D OpenGL visualization [#vbo]_ * flexible tree/list based partial geometry loading * intuitive virtual trackball translate/rotate * parallel or perspective projections * interactive fov and near/far plane clipping * persistent viewpoint bookmarks * animation by interpolation between bookmarks or orbiting * numerical control via UDP remote messaging * live Geant4 connection, photon, genstep visualization, single stepping * easily extensible python implementation [#g4daeview_code]_ * photon picking interface, with property inspection Also used for testing GPU photon propagation with Chroma project * interactive raycasting, including animated * propagation visualization, with time slider .. class:: tiny .. [#g4daeview_usage] http://simoncblyth.bitbucket.org/env/notes/geant4/geometry/collada/g4daeview/g4daeview_usage/ .. [#vbo] Implemented with single OpenGL Vertex Buffer Object (VBO) for entire geometry .. [#g4daeview_code] https://bitbucket.org/simoncblyth/env/src/tip/geant4/geometry/collada/g4daeview/ :small:`Compare DAE Exports to GDML and VRML2(WRL)` ---------------------------------------------------- .. class:: small Export validation: * Comparison of all vertices/faces reveals :red:`boolean solids are discrepant`. * Perfect [#perfect]_ DAE WRL agreement achieved by *cheating* : :blue:`perform triangulation once and reuse`. * Boolean solid triangulation sensitivity must be kept in mind as a potential systematic for computational uses * **DAE not much bigger than GDML**, despite including all triangles/vertices [#not-repeating]_ .. class:: small ==================== ================= ============== ================== ================= Qty DayaBay Lingao Far Juno x0.5 ==================== ================= ============== ================== ================= Volumes 12,229 12,229 18,903 25,000 Triangles 2,448,064 2,448,064 4,189,680 21,886,158 Vertices 1,245,996 1,245,996 2,128,208 10,993,079 DAE/GDML/WRL (MB) **6.9**/4.0/98 **6.9**/4.0/96 **8.6**/6.0/167 **6.1**/-/- ==================== ================= ============== ================== ================= .. class:: tiny ``VGDX_20140414`` counts using ``g4daeview.py -g 0: --with-chroma``, Juno geometry truncated .. class:: tiny .. [#perfect] Maximum DAE WRL offset < 0.13 mm, after patching VRML2 export precision bug (fixed in current G4). Details: http://simoncblyth.bitbucket.org/env/notes/geant4/geometry/collada/dae_cf_wrl/ .. [#not-repeating] Geometry is not repeated, instead the COLLADA format uses geometry instancing and a tree of transforms, just like GDML :small:`Chroma Raycasting : exercises geometry intersection` ------------------------------------------------------------- .. class:: small Raycasting exercises slowest part of optical photon propagation: :red:`geometry intersection`. .. image:: /env/chroma/chroma_camera/chroma_raycast_illustration.png :width: 600px :align: center .. class:: small * Shoot rays thru every pixel out into geometry, 1 CUDA thread for each, typically >1M rays * Find triangle intersections using BVH [#bvh]_ acceleration structure * Determine color based on ray to triangle normal angle * Alpha blend nearest 10 surfaces, providing transparency effect .. class:: tiny .. [#bvh] Boundary Volume Heirarchy, a tree of bounding boxes with triangles inside leaf nodes :small:`Chroma Raycast with entire geometry in view` ---------------------------------------------------------------- .. class:: small Render Split into 3x3 CUDA kernel launches, 1 thread per pixel, ~1.8s for 1.23M pixels, 2.4M tris (with [#hw]_) .. image:: /env/chroma/chroma_camera/20140423-162109.png :width: 800px :align: center .. class:: tiny .. [#hw] MacBook Pro (2013), NVIDIA GeForce GT 750M 2048 MB ; Workstation GPUs such as NVIDIA Kepler K20 expected at least ~5x faster :small:`G4DAEChroma : Bridging from Geant4 to Chroma` ----------------------------------------------------------- .. class:: small https://bitbucket.org/simoncblyth/env/src/tip/chroma/G4DAEChroma/G4DAEChroma/ .. sidebar:: Transport Infrastructure .. class:: small Collects generation steps from Geant4, sends to Chroma, receives hits in reply, integrates into G4 Hit Collections. :red:`Implemented with core of ~20 C++ classes` .. class:: small * :green:`Two detector specific subclasses required eg` *DybG4DAECollector* *DybG4DAEGeometry* * NumPy array creation from C++, transport via ZeroMQ * JSON metadata communication C++/Python, *map <-> dict* * Geant4 hit integration using additional *G4VSensitiveDetector* that steals hit collection pointers from existing one * Transform Cache to obtain PMT local coordinates from global hit coordinates * SQLite DB integation for monitoring/control .. image:: /env/chroma/G4DAEChroma/G4DAEChroma_bitbucket.png :width: 700px :align: center :small:`G4DAEChroma : For implementation details...` ----------------------------------------------------------- .. class:: small http://simoncblyth.bitbucket.org/env/notes/chroma/G4DAEChroma/G4DAEChroma/G4DAEChroma_implementation/ .. image:: /env/chroma/G4DAEChroma/G4DAEChroma_implementation.png :width: 700px :align: center https://bitbucket.org/simoncblyth/chroma :small:`Forked:` --------------------------------------------------------------------------- .. class:: small Changes in my fork of Chroma: * stability + efficiency improvements, enabling mobile development * OpenGL/CUDA interoperation, using VBOs * add recording of propagation steps into VBO datastructure * more efficient Raycasting using 4x4 matrix uniforms and PBOs * animated photon propagation visualization * change serialization approach from ROOT/TObject to NumPy (NPY) * :red:`generation of Cerenkov and Scintillation Photons based on Geant4 Generation Step inputs` .. image:: /env/chroma/chroma_fork_jan2015.png :width: 700px :align: center :small:`Chroma Generated Scintillation Photons cf Geant4` ------------------------------------------------------------ .. class:: small Chroma interpolates all properties in 20nm bins, stair artifacts in wavelength distribution can be reduced by using eg 10nm property interpolation bins. .. image:: /env/g4dae/generated_scintillation_time_wavelength.png :width: 800px :align: center :small:`Chroma Generated Cerenkov Photons cf Geant4` ------------------------------------------------------- .. class:: small Geant4/DetSim wavelength distribution has a blip at 200nm, corresponding to edge of water refractive index properties. (see the extra slides) .. image:: /env/g4dae/generated_cerenkov_time_wavelength.png :width: 800px :align: center Next Steps ------------------ .. class:: small G4DAE Geometry Exporter * implement parametrized/replica geometry handling, following GDML * investigate issue inherited from GDML of a skipped edge case (when a volume is shared between multiple volume pairs) resulting in missing *G4LogicalBorderSurface* * investigate improving default export appearance in common viewers * test with newer and latest versions of Geant4 * test on more detector geometries * incorporate into Geant4 codebase .. class:: small G4DAEChroma bridge * refactor to remove duplication in *G4DAEHit* *G4DAEHitList* * eliminate use of *cnpy* external in favor of *numpy.hpp* .. class:: small Chroma * Port Cerenkov *ApplyWaterQE* to Chroma (expect quick) * Port *DsPmtSensDet::ProcessHits* QE gymnastics to Chroma * Implement double sided surfaces, to match *G4LogicalBorderSurface* * Compare PMT Hit distributions between *NuWa/DetSimChroma* and *NuWa/DetSim* * :red:`iterate until match achieved or discrepancies explained` * Find desktop GPU on which can test operation and performance :i:`Cerenkov Photons Simulation - Side View` ---------------------------------------------- .. sidebar:: Cerenkov Photons .. class:: tiny Simulated Cerenkov shock front from an 100 GeV muon travelling from right to left across Dayabay AD. Primaries are simulated by Geant4, Cerenkov "steps" of the primaries are transferred to the GPU where **photons are generated, propagated and PMT hits formed**. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white). :i:`Scintillation Photons Simulation - Side View` -------------------------------------------------- .. sidebar:: Scintillation Photons .. class:: tiny Simulated Scintillation photons from an 100 GeV muon travelling from right to left across Dayabay AD. Primaries are simulated by Geant4, Scintillation "steps" of the primaries are transferred to the GPU where **photons are generated, propagated and PMT hits formed**. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white). Cerenkov Photons Simulation - Top View ---------------------------------------------- Scintillation Photons Simulation - Top View ---------------------------------------------- Extra Slides --------------- :small:`g4daeview.py : OpenGL view of Juno Geometry` ------------------------------------------------------------------------------- .. class:: tiny External view of Juno geometry with cutaway. The extreme size of the Juno geometry (50 million nodes in Chroma representation) provides a challenge for development on mobile GPUs. As my developments operate at the Geant4 level wherever possible it was relatively straightforward to apply the machinery developed for Dayabay to the Juno detector. In collaboration with Juno simulation experts the geometry was exported from Geant4 and GPU visualized in under a days work. .. image:: /env/geant4/geometry/collada/g4daeview/20140716-194144.png :height: 650px :align: center :small:`g4daeview.py : Chroma Raycast of Juno Geometry` ------------------------------------------------------------------------------- .. class:: tiny External view of Juno geometry. The extreme size of the Juno geometry (50 million nodes in Chroma representation) provides a challenge for development on mobile GPUs. The black rectangle arises due to aborts to avoid GPU crashes. .. image:: /env/geant4/geometry/collada/g4daeview/20140716-191232.png :height: 650px :align: center :small:`G4 Generated Cerenkov Wavelengths in material categories` ------------------------------------------------------------------------ .. class:: small Blip caused by 200nm edge in water refractive index .. image:: /env/g4dae/g4_cerenkov_wavelength.png :width: 700px :align: center :small:`Refractive Index Interpolation Explains Jump at 200nm` ---------------------------------------------------------------- .. class:: small DeadWater, IwsWater, OwsWater have same RINDEX starting from 200nm. Chroma interpolates properties onto a standard set of wavelengths, getting rid of the jump. .. image:: /env/g4dae/plot_refractive_index_comparison.png :width: 900px :align: center :small:`Comparison of Generated Scintillation Photon Distributions` ------------------------------------------------------------------------ .. class:: small Position, direction, polarization XYZ + time, wavelength, weight .. image:: /env/g4dae/generated_scintillation_3xyzw.png :width: 700px :align: center :small:`Comparison of Generated Cerenokov Photon Distributions` ------------------------------------------------------------------------ .. class:: small Position, direction, polarization XYZ + time, wavelength, weight .. image:: /env/g4dae/generated_cerenkov_3xyzw.png :width: 700px :align: center :small:`g4daeview.py : Dayabay Chroma Photon Propagation (1)` ------------------------------------------------------------------------------- .. class:: tiny Chroma GPU photon propagation at 12 nanoseconds. The photons are generated by Geant4 simulation of a 100 GeV muon travelling from right to left. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white). .. image:: /env/geant4/geometry/collada/g4daeview/20140716-161445.png :height: 600px :align: center