GPU Accelerated Geant4 Simulation with G4DAE and Chroma

Simon C Blyth, National Taiwan University
January 2015

Implications of Geometry Model Choice

Track > Geometry intersection typically limits simulation performance. Geometry model determines techniques (and hardware) available to accelerate intersection.

Geant4 Geometry Model (solid based, locked in)

Tree of nested solids composed of materials, each shape represented by different C++ class

Chroma Geometry Model (surface based, extremely simple > portable)

List of oriented triangles, each representing boundary between inside and outside materials.

3D industry focusses on surface models >> frameworks and GPU hardware designed to work with surface based geometries.

Chroma : Ultra-fast Photon MC, Developed by Stan Seibert, University of Pennsylvania

Chroma tracks photons through a triangle-mesh detector geometry, simulating processes like diffuse and specular reflections, refraction, Rayleigh scattering and absorption. Using triangle meshes eliminate geometry code as just one code path.

200x performance claim:

With a CUDA GPU Chroma has propagated 2.5M photons per second in a detector with 29k photomultiplier tubes. This is 200x faster than GEANT4.

BUT Chroma needs : triangles + inside/outside materials (my fork)

Standard Geant4 Workflow

Some invisible vertical space




OP Generated in Cerenkov and Scintillation G4VProcess, some reach PMTs and form hits.






// G4VProcess subclass
PostStepDoIt(const G4Track&, const G4Step&)
    // calculate NumPhotons

    // generate and collect secondaries
    for (G4int i = 0; i < NumPhotons; i++)

// G4VSensitiveDetector subclass
bool ProcessHits(G4Step* step, ...)
    // form hits from step information

Geant4 <-> Chroma Integration

Recreate Geometry info on GPU (G4DAE)

Runtime bridge via message queue (G4DAEChroma)

GPU Optical Photon simulation (Chroma)

External Photon Simulation Workflow

Exports Geant4 Geometry into COLLADA/DAE standard 3D files, based on GDML writer code, same XercesC dependency for XML handling. Export includes:


COLLADA : Standard for 3D Digital Asset Exchange (DAE)


Liberated Geometry allows Ubiquitous High Peformance Visualization


Open source:


Many more applications and frameworks listed (fork improves COLLADA loading speed) : Fast OpenGL 3D viewer for G4DAE files

G4DAE visualization [1] implemented with PyCollada, PyOpenGL and Glumpy

Also used for testing GPU photon propagation with Chroma project

[2]Implemented with single OpenGL Vertex Buffer Object (VBO) for entire geometry

Compare DAE Exports to GDML and VRML2(WRL)

Export validation:

  • Comparison of all vertices/faces reveals boolean solids are discrepant.
  • Perfect [4] DAE WRL agreement achieved by cheating : perform triangulation once and reuse.
  • Boolean solid triangulation sensitivity must be kept in mind as a potential systematic for computational uses
  • DAE not much bigger than GDML, despite including all triangles/vertices [5]
Qty DayaBay Lingao Far Juno x0.5
Volumes 12,229 12,229 18,903 25,000
Triangles 2,448,064 2,448,064 4,189,680 21,886,158
Vertices 1,245,996 1,245,996 2,128,208 10,993,079
DAE/GDML/WRL (MB) 6.9/4.0/98 6.9/4.0/96 8.6/6.0/167 6.1/-/-

VGDX_20140414 counts using -g 0: --with-chroma, Juno geometry truncated

[4]Maximum DAE WRL offset < 0.13 mm, after patching VRML2 export precision bug (fixed in current G4). Details:
[5]Geometry is not repeated, instead the COLLADA format uses geometry instancing and a tree of transforms, just like GDML

Chroma Raycasting : exercises geometry intersection

Raycasting exercises slowest part of optical photon propagation: geometry intersection.

[6]Boundary Volume Heirarchy, a tree of bounding boxes with triangles inside leaf nodes

Chroma Raycast with entire geometry in view

Render Split into 3x3 CUDA kernel launches, 1 thread per pixel, ~1.8s for 1.23M pixels, 2.4M tris (with [7])

[7]MacBook Pro (2013), NVIDIA GeForce GT 750M 2048 MB ; Workstation GPUs such as NVIDIA Kepler K20 expected at least ~5x faster

G4DAEChroma : Bridging from Geant4 to Chroma


G4DAEChroma : For implementation details...

/env/chroma/G4DAEChroma/G4DAEChroma_implementation.png Forked:

Changes in my fork of Chroma:


Chroma Generated Scintillation Photons cf Geant4

Chroma interpolates all properties in 20nm bins, stair artifacts in wavelength distribution can be reduced by using eg 10nm property interpolation bins.


Chroma Generated Cerenkov Photons cf Geant4

Geant4/DetSim wavelength distribution has a blip at 200nm, corresponding to edge of water refractive index properties. (see the extra slides)


Next Steps

G4DAE Geometry Exporter

G4DAEChroma bridge


Cerenkov Photons Simulation - Side View

Scintillation Photons Simulation - Side View

Cerenkov Photons Simulation - Top View

Scintillation Photons Simulation - Top View

Extra Slides : OpenGL view of Juno Geometry

External view of Juno geometry with cutaway. The extreme size of the Juno geometry (50 million nodes in Chroma representation) provides a challenge for development on mobile GPUs. As my developments operate at the Geant4 level wherever possible it was relatively straightforward to apply the machinery developed for Dayabay to the Juno detector. In collaboration with Juno simulation experts the geometry was exported from Geant4 and GPU visualized in under a days work.

/env/geant4/geometry/collada/g4daeview/20140716-194144.png : Chroma Raycast of Juno Geometry

External view of Juno geometry. The extreme size of the Juno geometry (50 million nodes in Chroma representation) provides a challenge for development on mobile GPUs. The black rectangle arises due to aborts to avoid GPU crashes.


G4 Generated Cerenkov Wavelengths in material categories

Blip caused by 200nm edge in water refractive index


Refractive Index Interpolation Explains Jump at 200nm

DeadWater, IwsWater, OwsWater have same RINDEX starting from 200nm. Chroma interpolates properties onto a standard set of wavelengths, getting rid of the jump.


Comparison of Generated Scintillation Photon Distributions

Position, direction, polarization XYZ + time, wavelength, weight


Comparison of Generated Cerenokov Photon Distributions

Position, direction, polarization XYZ + time, wavelength, weight

/env/g4dae/generated_cerenkov_3xyzw.png : Dayabay Chroma Photon Propagation (1)

Chroma GPU photon propagation at 12 nanoseconds. The photons are generated by Geant4 simulation of a 100 GeV muon travelling from right to left. Photon colors indicate reemission (green), absorption(red), specular reflection (magenta), scattering(blue), no history (white).