Opticks : GPU Optical Photon Simulation for Particle Physics with NVIDIA OptiX

Meeting the Challenge of JUNO Simulation with Opticks : GPU Optical Photon Acceleration via NVIDIA® OptiX™


Simon C Blyth, IHEP, CAS — DANCE Workshop, October 2019, Houston







Optical Photon Simulation Problem...

Optical Photon Simulation ≈ Ray Traced Image Rendering

Much in common : geometry, light sources, optical physics

Many Applications of ray tracing :

August 2018 : Major Ray Tracing Advance


10 Giga Rays/s

Ray-tracing vs Rasterization

/env/presentation/nvidia/nv_rasterization.png /env/presentation/nvidia/nv_raytrace.png


NVIDIA RTX Metro Exodus

Spatial Index Acceleration Structure

Hardware Traversal of BVH Spatial Index

NVIDIA® OptiX™ Ray Tracing Engine -- http://developer.nvidia.com/optix

OptiX makes GPU ray tracing accessible

NVIDIA expertise:

Opticks provides (Yellow):

[1] Turing RTX GPUs


Opticks : translates G4 optical physics to CUDA/OptiX

OptiX : single-ray programming model -> line-by-line translation

CUDA Ports of Geant4 classes
  • G4Cerenkov (only generation loop)
  • G4Scintillation (only generation loop)
  • G4OpAbsorption
  • G4OpRayleigh
  • G4OpBoundaryProcess (only a few surface types)
Modify Cerenkov + Scintillation Processes
  • collect genstep, copy to GPU for generation
  • avoids copying millions of photons to GPU
Scintillator Reemission
  • fraction of bulk absorbed "reborn" within same thread
  • wavelength generated by reemission texture lookup
Opticks (OptiX/Thrust GPU interoperation)
  • OptiX : upload gensteps
  • Thrust : seeding, distribute genstep indices to photons
  • OptiX : launch photon generation and propagation
  • Thrust : pullback photons that hit PMTs
  • Thrust : index photon step sequences (optional)

Opticks : translates G4 geometry to GPU, without approximation

Automated : Geant4 "World" -> Opticks CSG -> CUDA/OptiX
  • intersection functions for ~10 primitives
  • intersection program for arbitrarily complex CSG shapes
  • repeated geometry instances identified (progeny digests)
  • instance transforms used in OptiX/OpenGL geometry
  • merge CSG trees into global + instance buffers
Material/Surface/Scintillator properties
  • interpolated to standard wavelength domain
  • interleaved into "boundary" texture
  • "reemission" texture for wavelength generation

CUDA/OptiX Intersection functions for ~10 primitives


Sphere, Cylinder, Disc, Cone, Convex Polyhedron, Hyperboloid, Torus, ...

CUDA/OptiX Intersection Program for Arbitrarily Complex CSG shapes

Complete Binary Tree, pick between pairs of nearest intersects:

UNION tA < tB Enter B Exit B Miss B
Enter A ReturnA LoopA ReturnA
Exit A ReturnA ReturnB ReturnA
Miss A ReturnB ReturnB ReturnMiss
[1] Ray Tracing CSG Objects Using Single Hit Intersections, Andrew Kensler (2006)
with corrections by author of XRT Raytracer http://xrt.wikidot.com/doc:csg
[2] https://bitbucket.org/simoncblyth/opticks/src/tip/optixrap/cu/csg_intersect_boolean.h
Similar to binary expression tree evaluation using postorder traverse.



Validation of Opticks Simulation by Comparison with Geant4

Bi-simulations of all JUNO solids, with millions of photons

mis-aligned histories
mostly < 0.25%, < 0.50% for largest solids
deviant photons within matched history
< 0.05% (500/1M)

Primary sources of problems

Primary cause : float vs double

Geant4 uses double everywhere, Opticks only sparingly (observed double costing 10x slowdown with RTX)


Performance : Scanning from 1M to 400M photons

Full JUNO Geometry j1808v5

Production Mode : does the minimum

Multi-Event Running : measure interval and launch

interval : avg time between successive launches, including:

NVIDIA Quadro RTX 8000 (48G)

谢谢 NVIDIA China
for loaning the card



scan-pf-1_Opticks_vs_Geant4 2

JUNO Full, 400M photons from center
Geant4 Extrap. 95,600 s (26 hrs)
Opticks RTX ON (i) 58 s

scan-pf-1_Opticks_Speedup 2

JUNO Full, 400M photons from center Speedup
Opticks RTX ON (i) 58s x1660
Opticks RTX OFF (i) 275s x348
Geant4 Extrap. 95,600s (26 hrs)  


5x Speedup from RTX with full JUNO geometry

Useful Speedup > 1000x : But Why Not Giga Rays/s ? (1 photon ~10 rays)

OptiX Performance Tools and Tricks, David Hart, NVIDIA https://developer.nvidia.com/siggraph/2019/video/sig915-vid

Where Next for Opticks ?

JUNO+Opticks into Production

Geant4+Opticks Integration : Work with Geant4 Collaboration

Expand Community : Webinars, Conference Tutorials ?

Meeting the Challenge of JUNO Simulation with Opticks


Opticks : state-of-the-art GPU ray tracing applied to optical photon simulation and integrated with Geant4 to eliminate memory and time bottlenecks.

  • Drastic speedup -> better detector understanding -> greater precision
    • any simulation limited by optical photons can benefit
https://bitbucket.org/simoncblyth/opticks code repository
https://simoncblyth.bitbucket.io presentations and videos
https://groups.io/g/opticks forum/mailing list archive
email:opticks+subscribe@groups.io subscribe to mailing list