nEXO + Opticks ? NVIDIA OptiX accelerated optical photon simulation ?
nEXO + Opticks ?
NVIDIA OptiX accelerated optical photon simulation ?
Open source, https://bitbucket.org/simoncblyth/opticks
Simon C Blyth, IHEP, CAS — nEXO Light Simulations Workshop, Montreal — 25 October 2024
Outline (1) : Current Opticks Description
- Optical Photon Simulation : Context and Problem
- p3: (JUNO) Optical Photon Simulation Problem...
- p4: Optical photons limit many simulations => lots of interest in Opticks
- p5: Optical Photon Simulation ≈ Ray Traced Image Rendering
- p6: NVIDIA RTX Generations : RT performance : ~2x every ~2 years
- p7: NVIDIA OptiX : Ray Tracing Engine
- Opticks : Solution to Optical Photon Simulation Problem
- p8: Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
- p9: Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX
- p10: Full JUNO, Opticks, OptiX 7.5/8.0
- p11: Integrated Analytic + Triangulated Geometry (NEW)
- p12: Interactive ray traced visualization via OpenGL/OptiX interop (NEW)
- p14: GuideTube : Torus Triangulated
- p15: List-node : avoids deep CSG trees
- p16: Pure Optical TorchGenstep scan : 1M to 100M photons
- p18: Optical simulation 4x faster 1st->3rd gen RTX
- p20: How much parallelized speedup actually useful to overall speedup?
- p21: Summary + Links
- p22: Acknowledgements
- p23: NEW Opticks User : Ilker Parmaksiz, NEXT-CRAB0 Prototype
Outline (2) : Review Early Chroma Investigations
- Opticks Early History
- p24: pre-History : Chroma investigations
- Triangulated Geometry, G4DAE Geometry exporter
- Chroma + G4DAE -> NVIDIA OptiX + "Opticks"
- Handling Huge Geometry (eg JUNO) with instancing
- NVIDIA OptiX 5 Ray Tracing Engine
- Triangulated Geometry Problems
- Fully Analytic CSG
- Analytic PMT (12 parts, not 2928 triangles)
- GPU Geometry starts from ray-primitive intersection
- Torus : much more difficult/expensive than other primitives
- Constructive Solid Geometry (CSG) : Which intersect ?
- CSG Problems : deep trees, coincident faces
- Opticks Instancing : "Factorizes" Geometry
- Porting the physics
- Translate Geant4 Optical Physics to GPU (OptiX/CUDA)
- Optical Photon Simulation : Deciding history on way to boundary
- Validation of Opticks Simulation by Comparison with Geant4
- Performance : Scanning from 1M to 400M Photons
- NVIDIA Giveth and NVIDIA Taketh away ...
(JUNO) Optical Photon Simulation Problem...
Optical photons limit many simulations => lots of interest in Opticks
| EXPT |
Reactor neutrino |
| Daya Bay |
neutrino oscillations |
| JUNO |
mass heirarchy + oscillations => NVIDIA CN Contacts |
| |
Long baseline neutrino beam |
| DUNE |
FermiLab->Sanford, LAr TPC, => Assistance from Fermilab Geant4 Group |
| |
Neutrinoless double beta decay, dark matter, other search |
| LZ |
LUX-ZEPLIN dark matter experiment, Sandford => NVIDIA US Contacts |
| LEGEND |
Large Enriched Germanium Experiment, Gran Sasso/SNOLAB |
| SABRE |
dark matter direct-detection, Australia |
| AMoRE |
Mo-based Rare process Experiment, S.Korea |
| nEXO |
next Enriched Xenon Observatory, LLNL |
| NEXT-CRAB0 |
High Pressure Gaseous Xenon TPC with a Direct VUV Camera Based Readout |
| |
Neutrino telescope |
| KM3Net |
Cubic Kilometre Neutrino Telescope, Mediterranean |
| IceCube |
IceCube Neutrino Observatory, South Pole |
| |
Air shower : gamma-ray and cosmic-ray observatory |
| LHAASO |
Large High Altitude Air Shower Observatory, Sichuan |
| |
Accelerator |
| LHCb-RICH |
LHCb ring imaging Cherenkov sub-detector, CERN => NVIDIA EU Contacts |
Optical Photon Simulation ≈ Ray Traced Image Rendering
- simulation
- photon parameters at sensors (PMTs)
- rendering
- pixel values at image plane
Much in common : geometry, light sources, optical physics
- both limited by ray geometry intersection, aka ray tracing
Many Applications of ray tracing :
- advertising, design, architecture, films, games,...
- -> huge efforts to improve hw+sw over 30 yrs
NVIDIA RTX Generations
- RT Core : ray trace dedicated GPU hardware
- Each gen : large ray tracing improvements:
- Blackwell (2024?5) Expect: ~2x ray trace over Ada
- Ada (2022) ~2x ray trace over Ampere
- Ampere (2020) ~2x ray trace over Turing (2018)
- NVIDIA Blackwell 4th Gen RTX : expected Q1 2025
ray trace performance : ~2x every ~2 years
NVIDIA® OptiX™ Ray Tracing Engine -- Accessible GPU Ray Tracing
OptiX makes GPU ray tracing accessible
- Programmable GPU-accelerated Ray-Tracing Pipeline
- Single-ray shader programming model using CUDA
- ray tracing acceleration using RT Cores (RTX GPUs)
- "...free to use within any application..."
OptiX features
- acceleration structure creation + traversal (eg BVH)
- instanced sharing of geometry + acceleration structures
- compiler optimized for GPU ray tracing
User provides (Green):
- ray generation
- geometry bounding boxes
- intersect functions
- instance transforms
Latest Release : NVIDIA® OptiX™ 8.0.0 (Aug 2023) NEW:
- Shader Execution Reordering (SER) (Ada: up to 2x)
- SER: reduced execution+data divergence (on-the-fly)
Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
| https://bitbucket.org/simoncblyth/opticks |
Opticks API : split according to dependency -- Optical photons are GPU "resident", only hits need to be copied to CPU memory
Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7/8
Geant4 Geometry Model (JUNO: 400k PV, deep hierarchy)
| PV |
G4VPhysicalVolume |
placed, refs LV |
| LV |
G4LogicalVolume |
unplaced, refs SO |
| SO |
G4VSolid,G4BooleanSolid |
binary tree of SO "nodes" |
Opticks CSGFoundry Geometry Model (index references)
| struct |
Notes |
Geant4 Equivalent |
|---|
| CSGFoundry |
vectors of the below, easily serialized + uploaded + used on GPU |
None |
| qat4 |
4x4 transform refs CSGSolid using "spare" 4th column (becomes IAS) |
Transforms ref from PV |
| CSGSolid |
refs sequence of CSGPrim |
Grouped Vols + Remainder |
| CSGPrim |
bbox, refs sequence of CSGNode, root of CSG Tree of nodes |
root G4VSolid |
| CSGNode |
CSG node parameters (JUNO: ~23k CSGNode) |
node G4VSolid |
NVIDIA OptiX 7/8 Geometry Acceleration Structures (JUNO: 1 IAS + 10 GAS, 2-level hierarchy)
| IAS |
Instance Acceleration Structures |
JUNO: 1 IAS created from vector of ~50k qat4 (JUNO) |
| GAS |
Geometry Acceleration Structures |
JUNO: 10 GAS created from 10 CSGSolid (which refs CSGPrim,CSGNode ) |
JUNO : Geant4 ~400k volumes "factorized" into 1 OptiX IAS referencing ~10 GAS
Ada_cxr_overview_emm_t0_elv_t_moi__ALL.jpg
Analytic + triangulated geometry
- default : analytic CSG solids
- user can name solids for triangulation
- avoids issue with toruses + complex solids
- BUT : approximate geometry
- triangulation from G4Polyhedron
- config per-solid NumberOfRotationSteps by envvars
- uses OptiX "built-in" triangle intersection
- NEW FEATURE
- Integration of analytic + triangulated geometry
cxr_min__eye_1,0,0__zoom_1__tmin_0.5__sSurftube_0V1_0:0:-1.jpg
Interactive ray traced visualization via OpenGL/OptiX interop
initial viewpoint, geometry exclusions via envvars
WASDQE+mouse 3D navigation
Ada_cxr_min__eye_1,0,0__zoom_1__tmin_0.5__sSurftube_0V1_0:0:-100000.jpg
Render on NVIDIA RTX 5000 Ada Generation in 0.0060 s (not 0.0200 s)
GuideTube : Torus Triangulated
- GuideTube (39*2*2 = 156 G4Torus)
- split in phi segments, radius breaks
Intersect with torus expensive on GPU
- requires double precision to solve quartic
- even with double precision analytic solution imprecise
- numerical approach favored => triangulation
Triangulation using G4Polyhedron
G4Poly..::SetNumberOfRotationSteps
| |
NumberOfRotationSteps |
|---|
| HepPolyhedron Default |
24 |
| Top Right |
48 |
| Bottom Right |
480 |
Adjustable: precision of intersect, number of triangles
GPUs evolved for triangles => fast even with many
List-node avoids deep CSG trees
Problematic deep CSG tree without list-node
+------------------------------------------+
| |
| |
| U |
| / \ |
| / \ |
| / S |
| U / \ |
| / \ I J |
| U H |
| / \ |
| U G |
| / \ |
| U F |
| / \ |
| U E |
| / \ |
| U D |
| / \ |
| U C |
| / \ |
| A B |
| |
+------------------------------------------+
U : Union
S : Subtraction
A-J : Tubs (cylinder) primitive
Simple G4MultiUnion is translated to Opticks list-node
Pure Optical TorchGenstep scan : 1M to 100M photons
TEST=medium_scan ~/opticks/cxs_min.sh
Generate optical only events with 1M->100M photons starting from CD center,
gather and save only Hits.
OPTICKS_RUNNING_MODE=SRM_TORCH ## "Torch" running enables num_photon scan
OPTICKS_NUM_PHOTON=M1,10,20,30,40,50,60,70,80,90,100
OPTICKS_NUM_EVENT=11
OPTICKS_EVENT_MODE=Hit
- uses CSGOptiXSMTest executable (no Geant4 dependency, avoids ~150s of initialization time)
- load and upload geometry in ~2s
Compare simulation scans on two Dell Precision Workstations:
| GPU (VRAM) |
Arch |
GPU Release |
CUDA(RT) Cores |
RTX Gen |
Driver |
CUDA |
OptiX |
|---|
| NVIDIA TITAN RTX(24G) |
Turing |
Dec 2018 |
4,608(72) |
1st |
515.43 |
11.7 |
7.5 |
| NVIDIA RTX 5000(32G) |
Ada |
Aug 2023 |
12,800(100) |
3rd |
550.76 |
12.4 |
8.0 |
- max launch size : 24/32/48G VRAM ~200/266/400M photons
ALL1_scatter_10M_photon_22pc_hit_alt.png
- 4.5M hits from 20M photon TorchGenstep, 4.4(1.1) seconds
- with: NVIDIA TITAN RTX(NVIDIA RTX 5000 Ada) 1st(3rd) gen RTX
AB_Substamp_ALL_Etime_vs_Photon_rtx_gen1_gen3.png
Event Time(s) vs PH(M)
| PH(M) |
G1 |
G3 |
G1/G3 |
|---|
| 1 |
0.47 |
0.14 |
3.28 |
| 10 |
0.44 |
0.13 |
3.48 |
| 20 |
4.39 |
1.10 |
3.99 |
| 30 |
8.87 |
2.26 |
3.93 |
| 40 |
13.29 |
3.38 |
3.93 |
| 50 |
18.13 |
4.49 |
4.03 |
| 60 |
22.64 |
5.70 |
3.97 |
| 70 |
27.31 |
6.78 |
4.03 |
| 80 |
32.24 |
7.99 |
4.03 |
| 90 |
37.92 |
9.33 |
4.06 |
| 100 |
41.93 |
10.42 |
4.03 |
Optical simulation 4x faster 1st->3rd gen RTX, (3rd gen, Ada : 100M photons simulated in 10 seconds) [TMM PMT model]
How much parallelized speedup actually useful to overall speedup?
optical photon simulation, P ~ 99% of CPU time
- => limit on overall speedup S(n) is 100x
- even with parallel speedup factor >> 1000x
Traditional simulation use:
- speedup beyond 1000x not needed
amdahl_p_sensitive.png
Summary and Links
Opticks : state-of-the-art GPU ray traced optical simulation integrated with Geant4,
with automated geometry translation into GPU optimized form.
- NVIDIA Ray Trace Performance continues rapid progress (2x each gen., every ~2 yrs)
- any simulation limited by optical photons can benefit from Opticks
- more photon limited -> more overall speedup (99% -> ~90x)
Acknowledgements
- Opticks users
- ~38 members of forum : https://groups.io/g/opticks
- many thanks to active bug reporting users
- (especially from JUNO, LZ, LHAASO, LHCb-RICH, DUNE, NEXT-CRAB0)
- JUNO Collaboration
- Tao Lin, Yuxiang Hu, ... (+ many more : changing geometry and physics models)
- forced Opticks to continuously improve
- Geant4 collaboration
- especially Hans Wentzel, Fermilab Geant4 group, early adopter of Opticks
- guest invites to Okinawa, Wollongong meetings
- Dark Matter Search Community (XENON,LZ,DARWIN,..) : DANCE invite 2019
- Many NVIDIA Engineers:
- NVIDIA GPU Technology Conferences (San Jose, Suzhou)
- seven dedicated meetings in 2021 : migrating to OptiX 7 API
- UK GPU Hackathon 2022
Ilker Parmaksiz, NEXT-CRAB0 Prototype
New active bug reporting Opticks user : Ilker Parmaksiz
- careful comparison : Data, Geant4, Opticks
- Opticks 181x over Geant4
Opticks pre-History : Chroma investigations for DayaBay Optical Sim
CAVEAT : I LAST USED CHROMA IN 2015
- developed "G4DAE" Geant4 exporter of 3D DAE files
- added DAE import to my Chroma fork
- 2013 : bought macbook pro (NVIDIA Geforce 750M GPU)
Chroma : Disadvantages
- No use of ray trace engine, eg NVIDIA OptiX
- No use of dedicated ray trace hardware RT cores (RTX)
- BYOB : rolls own BVH acceleration structure
- expert tuning work needed for new GPU arch ?
Chroma : Fundamental Problem, triangles only
- approximate geometry : cannot match Geant4
- best available polygonization, G4Polyhedron
- some solids (eg G4Polycone) yield "cleaved" meshes
- viz. bug for Geant4, broken geometry for Chroma
- OpenMesh surgery possible, but not automatic
chroma_camera_raycast
(g4daeview.py) Chroma Raycast of Daya Bay geometry (3x3 CUDA kernel lunches, 1.8s for 1.23M pixels, Geforce 750M GPU)
G4DAE : DYB pool bottom, Chroma raycast
(g4daeview.py) Chroma raycast render of triangulated geometry
G4DAE : DYB pool bottom, OpenGL render
(g4daeview.py) OpenGL rasterized render of triangulated geometry
Triangulated Geometry : Great for Visualization
Many apps/libs can view/edit DAE/COLLADA files
- Meshlab, Blender, Sketchup, ...
- macOS: Finder/Quickview/Preview/Xcode
- threejs, pycollada, SceneKit, ...
- OpenGL, WebGL, pyopengl, glumpy
Triangle Visualization Advantage
- GPUs evolved to rasterize trianglated geometry
- Developed pyopengl/pycollada renderer
- entire geometry in single OpenGL draw call
- shockingly fast on mobile GPU
CUDA/OpenGL interoperation
- share GPU buffers between compute and visualization
- visualize steps of millions of photons within geometry
Daya Bay Chroma Photon Propagation (1)
(g4daeview.py) Chroma GPU photon propagation at 12 nanoseconds. The photons are generated by Geant4
simulation of a 100 GeV muon travelling from right to left.
Photon colors indicate reemission (green), absorption(red),
specular reflection (magenta), scattering(blue), no history (white).
Daya Bay Chroma Photon Propagation (2)
(g4daeview.py) Chroma GPU photon propagation at 14 nanoseconds.
The interface provides interactive control of the propagation time
allowing any stage of the propagation to be viewed by
scrubbing time backwards/forwards. The speed of this visualization
is achieved by interoperation of CUDA kernels and OpenGL shaders accessing
the same GPU resident photon propagation data.
Daya Bay Chroma Photon Propagation (3)
(g4daeview.py) Initial photon positions of a Geant4 simulated muon that crosses
between the Dayabay Near hall ADs. Colors represent photon wavelengths.
Optical photons: collected in G4 StackAction, serialized, sent over ZeroMQ, deserialized,
presented using OpenGL GLSL shaders.
Opticks History : Chroma + G4DAE -> NVIDIA OptiX + "Opticks"
Why switch to NVIDIA OptiX ?
- OptiX : transparent multi-GPU with no effort
- NVIDIA supported package
- NVIDIA expertise on keeping GPUs busy
- Chroma implemented with python/numpy/pycuda
- C++ impl. needed for easy Geant4 integration
DBNS geometry raycast comparison using mobile GPU
- NVIDIA OptiX 5 : interactive ~30 fps raycasting
- Chroma : 1.8s per frame
Performance improvement ~50x
- Opticks + NVIDIA OptiX 8 + Ada RTX 5000 : ~300 fps with JUNO (much bigger geometry)
"Opticks" started as synthesis:
- Chroma : high level propagation loop structure
- Geant4 : simulation details
- Graphics : performance techniques
Package name "Opticks", taken from world changing publication:
- Sir Isaac Newton FRS "Opticks: or, A Treatise of the Reflexions, Refractions, Inflexions and Colours of Light."
TITAN RTX (1st)