Opticks : GPU Optical Photon Simulation for Particle Physics with NVIDIA OptiX

Opticks Benefits

Outline

• Short video
• Problem and approach to solving
  • Optical Photon Simulation problem
  • Hybrid solution : Geant4 + Opticks
  • NVIDIA OptiX Ray Tracing Engine
• Opticks : optical photon simulation with NVIDIA OptiX
  • geometry workflow : geocache, GPU textures
  • large geometry techniques, geometry modelling
  • Geant4/Opticks event workflow
• Validation, performance comparisons
• Summary

Optical Photon Simulation Problem...

JPMT Before Contact 2

Ray Traced Image Synthesis ≈ Optical Photon Simulation

OptiX Pixel Calculation

http://on-demand.gputechconf.com/siggraph/2013/presentation/SG3106-Building-Ray-Tracing-Applications-OptiX.pdf

Geometry, light sources, optical physics ->

• pixel values at image plane
• photon parameters at detectors (eg PMTs)

Ray tracing has many applications :

• advertising, design, entertainment, games,...
• BUT : most ray tracers just render images

NVIDIA OptiX had foresight to be less specific:

• general geometry intersection API
• OptiX is to ray tracing what OpenGL is to rasterization

rasterization

project 3D primitives onto 2D image plane, combine fragments into pixel values

ray tracing

cast rays thru image pixels into scene, recursively reflect/refract with geometry intersected, combine returns into pixel values

NVIDIA OptiX 1

NVIDIA OptiX 2

https://research.nvidia.com/publication/optix-general-purpose-ray-tracing-engine

Opticks Geometry : Recreate Geant4 "Context" on GPU

Export Tesselated Geometry

• Geant4 -> G4DAE XML https://bitbucket.org/simoncblyth/g4dae

Load G4DAE XML

• label primitives with boundary indices
• find repeated geometry "instances"
• write NumPy serialization .npy geocache

Geocache -> few seconds startup, instead of few minutes

Load/upload geocache

• load geocache from file
• upload OptiX textures to GPU
• upload geometry buffers to GPU

GPU geometry buffers read by:

• OptiX bounding box and intersection programs
• OpenGL GLSL shaders for visualization

Large Geometry Techniques : Instancing Mandatory

Geometry analysed to find instances

JUNO: ~90M --> 0.1M triangles

• 19,000 20" PMTs
• 34,000 3" PMTs

OpenGL/OptiX instancing

Multiple Renderers

• global, full instances, bbox instances
• rapid switching controls

Geometry Modelling : Tesselated vs Analytic Photomultiplier Tubes

Analytic : more realistic, faster, less memory, much more effort

For Dayabay PMT:

• partition CSG solids into 12 single primitive parts (instead of 2928 triangles)
• splitting at geometrical intersections avoids implementing general CSG boolean handling
• geometry provided to OptiX in form of ray intersection and bounding box code

Aim : analytic description of geometry on critical optical path, remainder tesselated

Hybrid Geant4/Opticks Event Workflow

GPU Resident Photons

Seeded on GPU

associate photons -> gensteps (via seed buffer)

Generated on GPU, using gensteps:

~Stacks collected before photon generation:

• number of photons to generate
• start/end position of step
• other quantities needed for GPU generation

Propagated on GPU

Only photons hitting PMTs copied to CPU

Thrust: high level C++ access to CUDA

• https://developer.nvidia.com/Thrust

Geant4/Detector Simulation

• modified Scintillation, Cerenkov processes
  • collect genstep
  • skip optical photon generation

Opticks (OptiX/Thrust GPU interoperation)

• OptiX : upload gensteps
• Thrust : seeding, distribute genstep indices to photons
• OptiX : launch photon generation and propagation
• Thrust : pullback photons that hit PMTs
• Thrust : index photon step sequences (optional)

Geant4/Detector Simulation

• populate standard hit collections
• subsequent electronics simulation proceeds unaltered

Multi-event handling

• reuse/resize OptiX buffers for each event

Compare Opticks/Geant4 Simulations with Simple Lights/Geometries

1M Photons -> Water Sphere (S-Polarized)

0.5M Photons -> Dayabay PMT

Photon step records

128 bit per step : highly compressed position, time, wavelength, polarization vector, material/history codes

Photon flag sequence

16x 4-bit step flags recorded in uint64 sequence, indexed using Thrust GPU sort (1M indexed ~0.040s)

Sequence index -> interactive OpenGL selection of photons by flag sequence

Opticks/Geant4 Rainbow Step Sequence Comparison

Flags:

• BT/BR: boundary transmit/reflect
• TO/SC/SA: torch/scatter/surface absorb

Statistically consistent photon histories in the two simulations : Multiple orders of rainbow apparent

 64-bit uint  Opticks    Geant4    chi2                                      (tag:5,-5)

        8ccd   819160    819654    0.15  [4 ] TO BT BT SA                    (cross droplet)
         8bd   102087    101615    1.09  [3 ] TO BR SA                       (external reflect)
       8cbcd    61869     61890    0.00  [5 ] TO BT BR BT SA                 (bow 1)
      8cbbcd     9618      9577    0.09  [6 ] TO BT BR BR BT SA              (bow 2)
     8cbbbcd     2604      2687    1.30  [7 ] TO BT BR BR BR BT SA           (bow 3)
    8cbbbbcd     1056      1030    0.32  [8 ] TO BT BR BR BR BR BT SA        (bow 4)
       86ccd     1014      1000    0.10  [5 ] TO BT BT SC SA
   8cbbbbbcd      472       516    1.96  [9 ] TO BT BR BR BR BR BR BT SA     (bow 5)
         86d      498       473    0.64  [3 ] TO SC SA
  bbbbbbbbcd      304       294    0.17  [10] TO BT BR BR BR BR BR BR BR BR  (bow 8+ truncated)
  8cbbbbbbcd      272       247    1.20  [10] TO BT BR BR BR BR BR BR BT SA  (bow 6)
  cbbbbbbbcd      183       161    1.41  [10] TO BT BR BR BR BR BR BR BR BT  (bow 7 truncated)

1M Rainbow S-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. S-Polarized intersection (E field perpendicular to plane of incidence) arranged by directing polarization radially.

PMT Opticks/Geant4 step distribution comparison TO BT [SD]

Good agreement reached, after several fixes: geometry, total internal reflection, group velocity

position(xyz), time(t)

polarization(abc), radius(r)

PMT Opticks/Geant4 step distribution comparison : chi2/ndf

Photon Propagation Times Geant4 cf Opticks

• Opticks > 200X Geant4 with only 384 core mobile GPU[1] (multi-GPU workstation up to 20x more cores)

• Interop uses OpenGL buffers allowing visualization, Compute uses OptiX buffers
• Interop/Compute : perfectly identical results, monitored by digest

[1] MacBook Pro (2013), NVIDIA GeForce GT 750M, 2048 MB, 384 cores

OpticksDocs

import numpy as np
a = np.load("photons.npy")

Summary

Opticks enables Geant4 based simulations to benefit from optical photon simulation taking effectively zero time and zero CPU memory, thanks to massive parallelism made accessible by NVIDIA OptiX.

• The more photons the bigger the overall speedup (99% -> 100x)
• Drastic speedup -> better detector understanding -> greater precision
• Large PMT based neutrino experiments, such as JUNO, can benefit the most

https://bitbucket.org/simoncblyth/opticks

YouTube Video

https://www.youtube.com/watch?v=CBpOha4RzIs