Opticks : GPU ray trace accelerated optical photon simulation

Outline

• Optical Photon Simulation : Context and Problem
  • (JUNO) Optical Photon Simulation Problem...
  • Optical photons limit many simulations => lots of interest in Opticks
  • Optical Photon Simulation ≈ Ray Traced Image Rendering
  • NVIDIA RTX Generations
  • NVIDIA OptiX : Ray Tracing Engine
• Opticks : Solution to Optical Photon Simulation Problem
  • Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
  • Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX
  • Full JUNO, Opticks, OptiX 7.5/8.0
  • How much parallelized speedup actually useful to overall speedup?
• Opticks : NEW FEATURES
  • Integrated analytic + triangulated geometry
  • n-Ary CSG "List-Nodes" : now standard (from G4MultiUnion)
  • Interactive ray traced visualization via OpenGL/OptiX interop
• Summary + Links

---

(JUNO) Optical Photon Simulation Problem...

Optical photons limit many simulations => lots of interest in Opticks

EXPT	Reactor neutrino
Daya Bay	neutrino oscillations
JUNO	mass heirarchy + oscillations => NVIDIA CN Contacts
	Long baseline neutrino beam
DUNE	FermiLab->Sanford, LAr TPC, => Assistance from Fermilab Geant4 Group
	Neutrinoless double beta decay, dark matter, other search
LZ	LUX-ZEPLIN dark matter experiment, Sandford => NVIDIA US Contacts
LEGEND	Large Enriched Germanium Experiment, Gran Sasso/SNOLAB
SABRE	dark matter direct-detection, Australia
AMoRE	Mo-based Rare process Experiment, S.Korea
nEXO	next Enriched Xenon Observatory, LLNL
NEXT-CRAB0	High Pressure Gaseous Xenon TPC with a Direct VUV Camera Based Readout
	Neutrino telescope
KM3Net	Cubic Kilometre Neutrino Telescope, Mediterranean
IceCube	IceCube Neutrino Observatory, South Pole
	Air shower : gamma-ray and cosmic-ray observatory
LHAASO	Large High Altitude Air Shower Observatory, Sichuan
	Accelerator
LHCb-RICH	LHCb ring imaging Cherenkov sub-detector, CERN => NVIDIA EU Contacts

Optical Photon Simulation ≈ Ray Traced Image Rendering

simulation

photon parameters at sensors (PMTs)

rendering

pixel values at image plane

Much in common : geometry, light sources, optical physics

• both limited by ray geometry intersection, aka ray tracing

Many Applications of ray tracing :

• advertising, design, architecture, films, games,...
• -> huge efforts to improve hw+sw over 30 yrs

NVIDIA RTX Generations

• RT Core : ray trace dedicated GPU hardware
• Each gen : large ray tracing improvements:
  • Blackwell (2024/5) Expect: ~2x ray trace over Ada
  • Ada (2022) ~2x ray trace over Ampere
  • Ampere (2020) ~2x ray trace over Turing (2018)
• NVIDIA Blackwell 4th Gen RTX : expected Q1 2025

~2x ray trace performance every ~2 years

NVIDIA® OptiX™ Ray Tracing Engine -- Accessible GPU Ray Tracing

OptiX makes GPU ray tracing accessible

• Programmable GPU-accelerated Ray-Tracing Pipeline
• Single-ray shader programming model using CUDA
• ray tracing acceleration using RT Cores (RTX GPUs)
• "...free to use within any application..."

OptiX features

• acceleration structure creation + traversal (eg BVH)
• instanced sharing of geometry + acceleration structures
• compiler optimized for GPU ray tracing

User provides (Green):

• ray generation
• geometry bounding boxes
• intersect functions
• instance transforms

Latest API Release : NVIDIA® OptiX™ 8.0.0 (Aug 2023)

• NEW: Shader Execution Reordering (SER) (Ada: up to 2x)
• SER: reduced execution+data divergence (on-the-fly)

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow

https://bitbucket.org/simoncblyth/opticks

Opticks API : split according to dependency -- Optical photons are GPU "resident", only hits need to be copied to CPU memory

Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7/8

Geant4 Geometry Model (JUNO: 400k PV, deep hierarchy)

PV	G4VPhysicalVolume	placed, refs LV
LV	G4LogicalVolume	unplaced, refs SO
SO	G4VSolid,G4BooleanSolid	binary tree of SO "nodes"

Opticks CSGFoundry Geometry Model (index references)

struct	Notes	Geant4 Equivalent
CSGFoundry	vectors of the below, easily serialized + uploaded + used on GPU	None
qat4	4x4 transform refs CSGSolid using "spare" 4th column (becomes IAS)	Transforms ref from PV
CSGSolid	refs sequence of CSGPrim	Grouped Vols + Remainder
CSGPrim	bbox, refs sequence of CSGNode, root of CSG Tree of nodes	root G4VSolid
CSGNode	CSG node parameters (JUNO: ~23k CSGNode)	node G4VSolid

NVIDIA OptiX 7/8 Geometry Acceleration Structures (JUNO: 1 IAS + 10 GAS, 2-level hierarchy)

IAS	Instance Acceleration Structures	JUNO: 1 IAS created from vector of ~50k qat4 (JUNO)
GAS	Geometry Acceleration Structures	JUNO: 10 GAS created from 10 CSGSolid (which refs CSGPrim,CSGNode )

JUNO : Geant4 ~400k volumes "factorized" into 1 OptiX IAS referencing ~10 GAS

th CPU and GPU

3. be simple and easy to serialize

The CSG prefix refers to : Constructive Solid Geometry which is the basis for finding intersects

Ada_cxr_overview_emm_t0_elv_t_moi__ALL.jpg

Analytic + triangulated geometry

• mostly analytic CSG solids
• user can name solids for triangulation
  • avoids issue with toruses + complex solids
  • BUT : approximate geometry
  • triangulation from G4Polyhedron
  • config per-solid NumberOfRotationSteps by envvar

cxr_min__eye_1,0,0__zoom_1__tmin_0.5__sSurftube_0V1_0:0:-1.jpg

Interactive ray traced visualization via OpenGL/OptiX interop

configure viewpoint, geometry exclusions via envvars

GuideTube : Torus Triangulated

GuideTube (39*2*2 = 156 G4Torus)

split in phi segments, radius breaks

Intersect with torus expensive on GPU

• requires double precision to solve quartic
• even with double precision analytic solution imprecise
• numerical approach favored => triangulation

Triangulation using G4Polyhedron

G4Poly..::SetNumberOfRotationSteps

	NumberOfRotationSteps
HepPolyhedron Default	24
Top Right	48
Bottom Right	480

Adjustable: precision of intersect, number of triangles

GPUs evolved for triangles => fast even with many

How much parallelized speedup actually useful to overall speedup?

optical photon simulation, P ~ 99% of CPU time

• => limit on overall speedup S(n) is 100x
• even with parallel speedup factor >> 1000x

Traditional simulation use:

• speedup beyond 1000x not needed

amdahl_p_sensitive.png

Summary

Opticks : state-of-the-art GPU ray traced optical photon simulation integrated with Geant4, with automated geometry translation into GPU optimized form.

• NVIDIA Ray Trace Performance continues rapid progress (2x each generation)
• any simulation limited by optical photons can benefit from Opticks
• more photon limited -> more overall speedup (99% -> 100x)