GPU Accelerated Geant4 Simulation with G4DAE and Chroma

http://simoncblyth.bitbucket.io/env/presentation/gpu_accelerated_geant4_simulation.html http://simoncblyth.bitbucket.io/env/presentation/g4dae_geometry_exporter.html http://simoncblyth.bitbucket.io/env/presentation/gpu_optical_photon_simulation.html

Drastic 200x Potential Speedup

Chroma project claims 200x speedup of optical photon (OP) propagation compared to Geant4. A goal worthwhile chasing.

Simon C Blyth, National Taiwan University
January 2015

Implications of Geometry Model Choice

Track > Geometry intersection typically limits simulation performance. Geometry model determines techniques (and hardware) available to accelerate intersection.

Geant4 Geometry Model (solid based, locked in)

Tree of nested solids composed of materials, each shape represented by different C++ class

Chroma Geometry Model (surface based, extremely simple > portable)

List of oriented triangles, each representing boundary between inside and outside materials.

3D industry focusses on surface models >> frameworks and GPU hardware designed to work with surface based geometries.

Chroma : Ultra-fast Photon MC, Developed by Stan Seibert, University of Pennsylvania

Chroma tracks photons through a triangle-mesh detector geometry, simulating processes like diffuse and specular reflections, refraction, Rayleigh scattering and absorption. Using triangle meshes eliminate geometry code as just one code path.

Original Motivation for G4DAE exporter

Allow use of Chroma GPU accelerated optical photon propagation with existing DayaBay Geant4 geometry.

200x performance claim:

With a CUDA GPU Chroma has propagated 2.5M photons per second in a detector with 29k photomultiplier tubes. This is 200x faster than GEANT4.

BUT Chroma needs : triangles + inside/outside materials

http://on-demand.gputechconf.com/gtc/2013/presentations/S3304-Particle-Physics-With-PyCUDA.pdf

https://bitbucket.org/chroma/chroma

https://bitbucket.org/simoncblyth/chroma (my fork)

Standard Geant4 Workflow

Some invisible vertical space

.

.

.

OP Generated in Cerenkov and Scintillation G4VProcess, some reach PMTs and form hits.

.

.

.

.

.

// G4VProcess subclass
PostStepDoIt(const G4Track&, const G4Step&)
{
    // calculate NumPhotons
    aParticleChange.SetNumberOfSecondaries(NumPhotons);

    // generate and collect secondaries
    for (G4int i = 0; i < NumPhotons; i++)
    {
        ...
        aParticleChange.AddSecondary(aSecondaryTrack);
    }
}

// G4VSensitiveDetector subclass
bool ProcessHits(G4Step* step, ...)
{
    // form hits from step information
}

Geant4 <-> Chroma Integration

Bridging to external OP simulation

Optical photons (modulo reemission) are the leaves of the simulation tree, allowing external simulation to be integrated rather simply.

Recreate Geometry info on GPU (G4DAE)

Runtime bridge via message queue (G4DAEChroma)

GPU Optical Photon simulation (Chroma)

External Photon Simulation Workflow

https://bitbucket.org/simoncblyth/g4dae

Inclusion into Geant4

Geometry exporter presented at Sept 2014 Geant4 Collaboration Meeting. Proposal to contribute was accepted, intended to be included with 2015 Geant4 release.

Exports Geant4 Geometry into COLLADA/DAE standard 3D files, based on GDML writer code, same XercesC dependency for XML handling. Export includes:

/env/geant4/geometry/collada/g4dae/g4dae_bitbucket.png

COLLADA : Standard for 3D Digital Asset Exchange (DAE)

/env/graphics/collada/collada_khronos.png

Liberated Geometry allows Ubiquitous High Peformance Visualization

Geometry Set Free

Liberating geometry data from Geant4/ROOT gives free choice of visualization packages. Many commercial, open source apps/libs provide high performance visualization of DAE files using GPU efficient OpenGL techniques. => Shockingly Smooth Visualization performance

Commercial:

Open source:

Frameworks:

Many more applications and frameworks listed http://en.wikipedia.org/wiki/COLLADA

http://meshlab.sourceforge.net https://bitbucket.org/simoncblyth/meshlab (fork improves COLLADA loading speed)

g4daeview.py : Fast OpenGL 3D viewer for G4DAE files

PyOpenGL Based

Easy to extend due to fast python development, yet access to highly efficient OpenGL techniques such as VBOs, PBOs and GLSL shaders.

G4DAE visualization [1] implemented with PyCollada, PyOpenGL and Glumpy

Also used for testing GPU photon propagation with Chroma project

[1]http://simoncblyth.bitbucket.io/env/notes/geant4/geometry/collada/g4daeview/g4daeview_usage/
[2]Implemented with single OpenGL Vertex Buffer Object (VBO) for entire geometry
[3]https://bitbucket.org/simoncblyth/env/src/tip/geant4/geometry/collada/g4daeview/

Compare DAE Exports to GDML and VRML2(WRL)

Export validation:

  • Comparison of all vertices/faces reveals boolean solids are discrepant.
  • Perfect [4] DAE WRL agreement achieved by cheating : perform triangulation once and reuse.
  • Boolean solid triangulation sensitivity must be kept in mind as a potential systematic for computational uses
  • DAE not much bigger than GDML, despite including all triangles/vertices [5]
Qty DayaBay Lingao Far Juno x0.5
Volumes 12,229 12,229 18,903 25,000
Triangles 2,448,064 2,448,064 4,189,680 21,886,158
Vertices 1,245,996 1,245,996 2,128,208 10,993,079
DAE/GDML/WRL (MB) 6.9/4.0/98 6.9/4.0/96 8.6/6.0/167 6.1/-/-

VGDX_20140414 counts using g4daeview.py -g 0: --with-chroma, Juno geometry truncated

[4]Maximum DAE WRL offset < 0.13 mm, after patching VRML2 export precision bug (fixed in current G4). Details: http://simoncblyth.bitbucket.io/env/notes/geant4/geometry/collada/dae_cf_wrl/
[5]Geometry is not repeated, instead the COLLADA format uses geometry instancing and a tree of transforms, just like GDML

Chroma Raycasting : exercises geometry intersection

Raycasting exercises slowest part of optical photon propagation: geometry intersection.

/env/chroma/chroma_camera/chroma_raycast_illustration.png
[6]Boundary Volume Heirarchy, a tree of bounding boxes with triangles inside leaf nodes

Chroma Raycast with entire geometry in view

Render Split into 3x3 CUDA kernel launches, 1 thread per pixel, ~1.8s for 1.23M pixels, 2.4M tris (with [7])

/env/chroma/chroma_camera/20140423-162109.png
[7]MacBook Pro (2013), NVIDIA GeForce GT 750M 2048 MB ; Workstation GPUs such as NVIDIA Kepler K20 expected at least ~5x faster

G4DAEChroma : Bridging from Geant4 to Chroma

https://bitbucket.org/simoncblyth/env/src/tip/chroma/G4DAEChroma/G4DAEChroma/

Transport Infrastructure

Collects generation steps from Geant4, sends to Chroma, receives hits in reply, integrates into G4 Hit Collections. Implemented with core of ~20 C++ classes

/env/chroma/G4DAEChroma/G4DAEChroma_bitbucket.png

G4DAEChroma : For implementation details...

http://simoncblyth.bitbucket.io/env/notes/chroma/G4DAEChroma/G4DAEChroma/G4DAEChroma_implementation/

/env/chroma/G4DAEChroma/G4DAEChroma_implementation.png

https://bitbucket.org/simoncblyth/chroma Forked:

Changes in my fork of Chroma:

/env/chroma/chroma_fork_jan2015.png

Chroma Generated Scintillation Photons cf Geant4

Chroma interpolates all properties in 20nm bins, stair artifacts in wavelength distribution can be reduced by using eg 10nm property interpolation bins.

/env/g4dae/generated_scintillation_time_wavelength.png

Chroma Generated Cerenkov Photons cf Geant4

Geant4/DetSim wavelength distribution has a blip at 200nm, corresponding to edge of water refractive index properties. (see the extra slides)

/env/g4dae/generated_cerenkov_time_wavelength.png

Next Steps

G4DAE Geometry Exporter

G4DAEChroma bridge

Chroma

Cerenkov Photons Simulation - Side View

Cerenkov Photons

Simulated Cerenkov shock front from an 100 GeV muon travelling from right to left across Dayabay AD. Primaries are simulated by Geant4, Cerenkov "steps" of the primaries are transferred to the GPU where photons are generated, propagated and PMT hits formed. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white).

Scintillation Photons Simulation - Side View

Scintillation Photons

Simulated Scintillation photons from an 100 GeV muon travelling from right to left across Dayabay AD. Primaries are simulated by Geant4, Scintillation "steps" of the primaries are transferred to the GPU where photons are generated, propagated and PMT hits formed. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white).

Cerenkov Photons Simulation - Top View

Scintillation Photons Simulation - Top View

Extra Slides

g4daeview.py : OpenGL view of Juno Geometry

External view of Juno geometry with cutaway. The extreme size of the Juno geometry (50 million nodes in Chroma representation) provides a challenge for development on mobile GPUs. As my developments operate at the Geant4 level wherever possible it was relatively straightforward to apply the machinery developed for Dayabay to the Juno detector. In collaboration with Juno simulation experts the geometry was exported from Geant4 and GPU visualized in under a days work.

/env/geant4/geometry/collada/g4daeview/20140716-194144.png

g4daeview.py : Chroma Raycast of Juno Geometry

External view of Juno geometry. The extreme size of the Juno geometry (50 million nodes in Chroma representation) provides a challenge for development on mobile GPUs. The black rectangle arises due to aborts to avoid GPU crashes.

/env/geant4/geometry/collada/g4daeview/20140716-191232.png

G4 Generated Cerenkov Wavelengths in material categories

Blip caused by 200nm edge in water refractive index

/env/g4dae/g4_cerenkov_wavelength.png

Refractive Index Interpolation Explains Jump at 200nm

DeadWater, IwsWater, OwsWater have same RINDEX starting from 200nm. Chroma interpolates properties onto a standard set of wavelengths, getting rid of the jump.

/env/g4dae/plot_refractive_index_comparison.png

Comparison of Generated Scintillation Photon Distributions

Position, direction, polarization XYZ + time, wavelength, weight

/env/g4dae/generated_scintillation_3xyzw.png

Comparison of Generated Cerenokov Photon Distributions

Position, direction, polarization XYZ + time, wavelength, weight

/env/g4dae/generated_cerenkov_3xyzw.png

g4daeview.py : Dayabay Chroma Photon Propagation (1)

Chroma GPU photon propagation at 12 nanoseconds. The photons are generated by Geant4 simulation of a 100 GeV muon travelling from right to left. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white).

/env/geant4/geometry/collada/g4daeview/20140716-161445.png