Opticks : Optical Photon Simulation for Particle Physics with NVIDIA OptiX

Opticks : Optical Photon Simulation for Particle Physics with NVIDIA® OptiX™

Simon C Blyth, National Taiwan University — https://bitbucket.org/simoncblyth/opticks — July 2016

Opticks Benefits

Outline

/env/presentation/newtons-opticks.png

Photomultiplier Tube based Detectors : Primary Tool of Neutrino Physics

PMT Based Expt Impact/Prospects Nobel
Savannah River 1956: confirmed existence of antineutrino 1995
Kamiokande II, IMB, Baksan 1987: observed supernova neutrino burst  
Super-Kamiokande 1998: up-going muon neutrino deficit -> oscillation -> mass 2015
SNO (heavy water) 2002: evidence for neutrino flavor conversion 2015
KamLAND 2003: disappearance of electron anti-neutrinos, 2005: first observation of geoneutrinos  
Borexino 2010: confirmed geoneutrinos  
Daya Bay 2012: discovered large theta13 neutrino oscillation mixing angle  
RENO, Double Chooz Confirms Daya Bay theta13  
JUNO data in 2020, ~18k 20 inch PMTs, reactor neutrino spectrum at 53km -> mass heirarchy  
KM3Net, Baikal GVD ~5(1) km^3 of Mediterranean(Lake Baikal) -> high energy cosmic neutrinos  
Hyper-K (proposed) 25x Super-K, 99k 20 inch PMTs, switchable muon (anti)neutrino beam Tokai->Kamioka, e-neutrino spectrum difference -> CP phase  

Understanding optical photon generation and propagation : critical for design, operation, analysis

Radical simulation speedup -> short development cycle -> improved understanding

Photomultiplier Tubes (PMTs)

/env/presentation/1px.png /env/presentation/hamamatsu.png

Photomultiplier Tube Operation

/env/presentation/1px.png
/env/presentation/pmt-schematic.png

Single photoelectron amplified by ~10 dynodes to yield measurable signal

Kamiokande II 1

Artists impression of SN1987A ejecta : based on ESO's VLT observation

© ESO/L. Calçada - Artwork based on ESO's VLT observations - http://www.eso.org/public/news/eso1032/

Kamiokande II 3

/env/presentation/CernCourierSN1987ANeutrinoBurst.png
© ESO/L. Calçada - Artwork based on ESO's VLT observations - http://www.eso.org/public/news/eso1032/

Super-Kamiokande PMTs 1

© Kamioka Observatory, ICRR(Institute for Cosmic Ray Research), The University of Tokyo.

Super-Kamiokande PMTs 4

/env/presentation/AtmosphericNeutrinos.png /env/presentation/SKCCNeutrinoNucleusCherenkov.png
© Kamioka Observatory, ICRR(Institute for Cosmic Ray Research), The University of Tokyo.

Super-Kamiokande PMTs 3

/env/presentation/SKRingsELikeMuLike.png
© Kamioka Observatory, ICRR(Institute for Cosmic Ray Research), The University of Tokyo.

Super-Kamiokande PMTs 2

© Kamioka Observatory, ICRR(Institute for Cosmic Ray Research), The University of Tokyo.
/env/presentation/SuperKOscillationNobel.png

Dayabay Reactor Neutrino Expt, Far Site

Daya Bay Far Site 2

/env/presentation/ZheTaupAD.png

Daya Bay Far Site 3

Daya Bay PMT Wall Photo 1

© 2010 The Regents of the University of California, through the Lawrence Berkeley National Laboratory

Daya Bay Antineutrino Detection via Inverse Beta Decay 1

Daya Bay Antineutrino Detection via Inverse Beta Decay 2

Daya Bay nGd Analysis : Most Precise Theta13

/env/presentation/1px.png /env/presentation/PhysRevLett-115-111802.png

Daya Bay : E(Prompt) -> Detector Unfolded Reactor Antineutrino Spectrum

/env/presentation/PromptEnergyPRL-116-061801-2016.png /env/presentation/AntiNeutrinoEnergyPRL-116-061801-2016.png /env/presentation/PhysRevLett-116-2016-06180.png /env/presentation/1px.png

Jiangmen Underground Neutrino Observatory (JUNO)

JUNO : Neutrino Mass Heirarchy (MH) Determination

/env/presentation/juno_anti_nu_e_spectrum.png
/env/presentation/JUNOEnergyResolutionFormula.png

Need Energy Resolution : 3% (at 1 MeV)

/env/presentation/1px.png

As many photons as possible:

Uniformity, Low noise

Baseline Numbers: arXiv:1507.05613v2 (Oct 2015) Neutrino Physics with JUNO

JUNO : A Multi-purpose Neutrino Observatory

/env/presentation/CJ_JUNO_NeuTel2015_NeutrinoRates.png

arXiv:1507.05613v2 (Oct 2015) Neutrino Physics with JUNO

Neutrino Physics with JUNO

/env/presentation/YellowBookAbstract.png

~220 page "Yellow Book" compendium of JUNO Physics http://arxiv.org/abs/1507.05613

Liquid Scintillator (LS) Particle Dependent Non-Linear Light Yield

Intrinsic scintillator quenching, described by Birks' Law

/env/presentation/EW2016_ZhangYiming_QuenchedOverTrue.png

Less light at high dE/dx ; kB Birks constant

/env/presentation/1px.png

Energy model considerations:

Liquid Scintillator LS ; Ternary (LAB/PPO/bis-MSB) Optimization

Requirements:

Solvent
  • LAB (Linear Alkylbenzene, 10-13 carbons + benzene)
  • absorbs most ionization
Primary Solute, 3000 mg/L (Fluor)
  • PPO ( 2,5-diphenyloxaxazole)
  • primary is Fluor to maximize scintillation yield
  • excitation LAB -> PPO, non-radiatively
  • excitation quenches at high density, Birks' Law (empirical)
Secondary Solute, 15 mg/L (Wavelength Shifter)
  • bis-MSB (p-bis(-o-methylstyryl)-benzene)
  • wavelength shift : reduce self absorption, match PMT
  • excitation PPO -> bis-MSB, radiatively:absorption+reemission

Fluor Scintillation + Cerenkov light -> Absorption + Reemission

Geant4 : Monte Carlo Simulation Toolkit

Geant4 : Monte Carlo Simulation Toolkit Generality

But, there is a problem...

JPMT Before Contact 2

Ray Traced Realistic Image Synthesis and Optical Photon Simulation

Fast ray tracing can revolutionize optical photon simulation

NVIDIA OptiX Ray Tracing Engine:

NVIDIA OptiX makes massively parallel intersection accessible

NVIDIA OptiX 1

NVIDIA OptiX 2

Brief History of GPU Optical Photon Simulation Development

2013(Aug-) [liberate geometry from Geant4]

2014 [integrate geometry/event data with Chroma [1]]

2015 [replace Chroma with Opticks]

2016 [Opticks validation against Geant4]

[1] http://chroma.bitbucket.org Developed by Stan Seibert, University of Pennsylvania.

Migrating Geometry from Geant4 to Opticks

Translate Geant4 CSG tree solids into Opticks boundaries:

Properties as function of wavelength interleaved into float4 texture.

Texture lookup simplifies OptiX programs

// material property lookup from OptiX programs
float nmi = (nm - wavelength_domain.x)/wavelength_domain.z + 0.5f ;
float4 material1 = tex2D(boundary_texture, nmi, line + 0.5f );

float refractive_index = material1.x ;
float absorption_length = material1.y ;
float scattering_length = material1.z ;
float reemission_probability = material1.w ;  // >0 for scintillators

Large Geometry/Event Techniques

Geometry analysed to find instances

JUNO: ~90M --> 0.1M triangles

OpenGL/OptiX instancing

Multiple Renderers

Split simulation from visualization

Geometry Modelling : Tesselated vs Analytic Photomultiplier Tubes

/env/graphics/ggeoview/dpib-triangulated-pmt.png /env/nuwa/detdesc/pmt/hemi-pmt-analytic-near-clipped.png /env/presentation/1px.png

Analytic : more realistic, faster, less memory, much more effort

For Dayabay PMT:

Tesselated Photomultiplier : unrealistic disco ball effect

/env/graphics/ggeoview/dpib-test-disco-ball.png

Analytic PMTs together with triangulated geometry

Starting from convenient tesselated geometry, implement analytic geometry for critical parts in primary optical path, to allow very close Geant4/Opticks match.

/env/nuwa/detdesc/pmt/analytic-pmt-optix-geometry.png

Visualizing An Optical Photon Simulation

Compressed Steps, OpenGL Buffer, written by OptiX

OpenGL Geometry Shader: lines -> points/line_strip

Interactive: time scrubbing + history/material selection

// geom.glsl
layout (lines) in;
layout (line_strip, max_vertices = 2) out;
void main ()
{
    vec4 p0 = gl_in[0].gl_Position  ;   // p0.w : step begin time
    vec4 p1 = gl_in[1].gl_Position  ;   // p1.w : step end time
    ...

Compare Opticks/Geant4 Simulations with Simple Lights/Geometries

/env/graphics/ggeoview/rainbow-spol-disc-incident-sphere.png

1M Photons -> Water Sphere (S-Polarized)

/env/graphics/ggeoview/PmtInBox-approach.png

0.5M Photons -> Dayabay PMT

/env/presentation/1px.png

Opticks/Geant4 Rainbow Step Sequence Comparison

Statistically consistent photon histories in the two simulations

 64-bit uint  Opticks    Geant4    chi2                                      (tag:5,-5)

        8ccd   819160    819654    0.15  [4 ] TO BT BT SA                    (cross droplet)
         8bd   102087    101615    1.09  [3 ] TO BR SA                       (external reflect)
       8cbcd    61869     61890    0.00  [5 ] TO BT BR BT SA                 (bow 1)
      8cbbcd     9618      9577    0.09  [6 ] TO BT BR BR BT SA              (bow 2)
     8cbbbcd     2604      2687    1.30  [7 ] TO BT BR BR BR BT SA           (bow 3)
    8cbbbbcd     1056      1030    0.32  [8 ] TO BT BR BR BR BR BT SA        (bow 4)
       86ccd     1014      1000    0.10  [5 ] TO BT BT SC SA
   8cbbbbbcd      472       516    1.96  [9 ] TO BT BR BR BR BR BR BT SA     (bow 5)
         86d      498       473    0.64  [3 ] TO SC SA
  bbbbbbbbcd      304       294    0.17  [10] TO BT BR BR BR BR BR BR BR BR  (bow 8+ truncated)
  8cbbbbbbcd      272       247    1.20  [10] TO BT BR BR BR BR BR BR BT SA  (bow 6)
  cbbbbbbbcd      183       161    1.41  [10] TO BT BR BR BR BR BR BR BR BT  (bow 7 truncated)

1M 64 bit uint flag sequences indexed using CUDA Thrust, 0.040 s (sparse histogram sort)

Rainbow Spectrum for 1st six bows

Spectra obtained by selecting photons by internal reflection counts. Colors obtained from spectra of each bin using CIEXYZ weighting functions converted into sRGB/D65 colorspace. Exposures by normalizing to bin with maximum luminance (CIE-Y) of each bow. White lines indicate geometric optics prediction of deviation angle ranges of the visible range 380-780nm. 180-360 degrees signifies exit on same side of droplet as incidence.

/env/numerics/npy/rainbow6-spectrum.png

1M Rainbow S-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. S-Polarized intersection (E field perpendicular to plane of incidence) arranged by directing polarization radially.

/env/optix/cfg4/rainbow-cfg4-spol.png

PmtInBox after 1

PmtInBox after 2

PMT Opticks/Geant4 step distribution comparison TO BT [SD]

Good agreement reached, after several fixes: geometry, total internal reflection, group velocity

/env/numerics/npy/PmtInBox_TOBTSD_xyzt.png

position(xyz), time(t)

/env/numerics/npy/PmtInBox_TOBTSD_abcr.png

polarization(abc), radius(r)

PMT Opticks/Geant4 step distribution comparison : chi2/ndf

4/PMT In Box/torch : X Y Z T A B C R
340271/340273 : [TO] BT SA 1.15 1.00 0.00 0.00 1.06 1.03 0.00 1.21
340271/340273 : TO [BT] SA 1.15 1.00 1.06 0.91 1.06 1.03 0.00 1.21
340271/340273 : TO BT [SA] 0.97 1.02 1.05 0.99 1.06 1.03 0.00 1.29
107598/107251 : [TO] BT SD 0.91 0.73 0.56 0.56 0.98 1.09 0.56 0.94
107598/107251 : TO [BT] SD 0.91 0.73 0.81 0.93 0.98 1.09 0.56 0.94
107598/107251 : TO BT [SD] 0.99 0.83 0.97 0.99 0.98 1.09 0.56 0.93
23217/23260 : [TO] BT BT SA 0.94 0.82 0.04 0.04 0.97 0.89 0.04 0.57
23217/23260 : TO [BT] BT SA 0.94 0.82 0.70 0.50 0.97 0.89 0.04 0.57
23217/23260 : TO BT [BT] SA 0.91 0.94 0.43 0.60 0.97 0.89 0.04 0.05
23217/23260 : TO BT BT [SA] 0.94 0.88 0.04 0.35 0.97 0.89 0.04 0.72
18866/19048 : [TO] AB 0.99 1.10 0.87 0.87 0.85 0.84 0.87 1.00
18866/19048 : TO [AB] 0.99 1.10 0.93 0.92 0.85 0.84 0.87 1.00
3179/3133 : [TO] SC SA 1.07 0.83 0.34 0.34 0.86 0.96 0.34 0.73
3179/3133 : TO [SC] SA 1.07 0.83 0.98 1.05 0.98 1.06 0.98 0.73
3179/3133 : TO SC [SA] 0.96 1.04 0.93 0.97 0.98 1.06 0.98 1.10
2204/2249 : [TO] BT AB 0.85 1.04 0.45 0.45 0.99 0.92 0.45 1.06
2204/2249 : TO [BT] AB 0.85 1.04 0.95 0.88 0.99 0.92 0.45 1.06
2204/2249 : TO BT [AB] 0.98 0.94 1.01 1.00 0.99 0.92 0.45 0.90
1696/1732 : [TO] BT BT AB 1.05 0.85 0.38 0.38 0.86 1.09 0.38 0.26
1696/1732 : TO [BT] BT AB 1.05 0.85 1.48 1.28 0.86 1.09 0.38 0.26
1696/1732 : TO BT [BT] AB 0.99 0.86 1.17 1.40 0.86 1.09 0.38 0.86
1696/1732 : TO BT BT [AB] 1.15 0.88 1.08 1.06 0.86 1.09 0.38 0.79
1446/1455 : [TO] BR SA 1.21 0.94 0.03 0.03 0.90 0.87 0.03 1.09
1446/1455 : TO [BR] SA 1.21 0.94 1.02 1.01 0.90 0.87 0.03 1.09
1446/1455 : TO BR [SA] 1.00 0.93 0.97 0.99 0.90 0.87 0.03 1.04

Photon Propagation Times Geant4 cf Opticks

/env/optix/cfg4/PmtInBox-cfg4-2.png
Test Geant4 10.2 Opticks Interop Opticks Compute
Rainbow 1M(S) 56 s 1.62 s 0.28 s
Rainbow 1M(P) 58 s 1.71 s 0.25 s
PmtInBox 0.5M 41 s 0.81 s 0.15 s

[1] MacBook Pro (2013), NVIDIA GeForce GT 750M, 2048 MB, 384 cores

Summary

/env/presentation/1px.png

Opticks enables particle physics to benefit from optical photon simulation taking effectively zero time and zero CPU memory, thanks to massive parallelism made accessible by NVIDIA OptiX.

/env/presentation/1px.png
  • The more photons the bigger the overall speedup (99% -> 100x)
  • Drastic speedup -> better detector understanding -> greater precision
  • Large PMT based neutrino experiments, such as JUNO, can benefit the most
/env/presentation/1px.png

https://bitbucket.org/simoncblyth/opticks

YouTube Video

https://www.youtube.com/watch?v=QzH6y0pKXk4

YouKu Video

http://v.youku.com/v_show/id_XMTUxNjM3MzA4OA==.html

JPMT Wide

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-wide_half.png

JPMT After Contact

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-after-contact_half.png

JPMT Approach

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-approach_half.png

JPMT Arrival

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-arrival_half.png

JPMT Inside Outside

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-inside-outside_half.png

OptiX Experience

Geometry Issues Fixed/Avoidable

Use cuRAND in OptiX without large stack

Init in separate CUDA launch, persist cuRAND state, subsequently load into OptiX context : avoids large stack performance hit

Compositing OptiX Ray Trace and OpenGL Rasterized

out vec4 frag_colour;
uniform sampler2D ColorTex ;
void main ()
{
   frag_colour = texture(ColorTex, texcoord);
   float depth = frag_colour.w ;
   frag_colour.w = 1.0 ;
   gl_FragDepth = depth  ;
   ...