Opticks : GPU Optical Photon Simulation

http://simoncblyth.bitbucket.io/env/presentation/opticks_gpu_optical_photon_simulation.html (Jan 2016) http://simoncblyth.bitbucket.io/env/presentation/optical_photon_simulation_with_nvidia_optix.html (July 2015)

Executive Summary

Contents

Simon C Blyth, National Taiwan University
January 2016

Brief History of GPU Optical Photon Simulation Development

winter 2014 (within Chroma)

spring 2015 (start transition to Opticks)

summer/autumn 2015 (Opticks transition completed)

autumn/winter 2015 (Validations begin)

Newton published Opticks in 1704

Based on: NVIDIA OptiX Ray Tracing Engine [C++/C/CUDA]

OptiX provides: CUDA compiler optimized for Ray Tracing

NVIDIA expertise on efficient GPU/multi-GPU usage

IHEP Performance Check

Used IHEP 4 GPU workstation to verify near linear performance scaling across multiple GPUs

https://developer.nvidia.com/optix

https://research.nvidia.com/publication/optix-general-purpose-ray-tracing-engine

NVIDIA OptiX In Action

Introducing Opticks : Recreating G4 Context on GPU

Basis packages

Geometry packages

GPU library interface packages

Main packages

Large Geometry/Event Techniques

Instancing implemented for OpenGL and OptiX

Geometry analysed to find instances

Switchable Rendering

Compute Mode

Ideas to investigate

  1. Interop forced to duplicate data as OpenGL/OptiX geometry sharing not operational

JUNO Geometry Instance Rendering Control

/env/graphics/ggeoview/ggv-juno-instancing.png

JPMT Inside Wide

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-inside-wide_half.png

Mesh Fixing with OpenMesh surgery

Intersection boundaries determines photon material, bad meshes cause incorrect material assignments

G4Polyhedron Tesselation Bug

OpenMeshRap finds/fixes cleaved meshes

Analytic PMT geometry : more realistic, faster, less memory

/env/nuwa/detdesc/pmt/hemi-pmt-solids.png

Analytic PMT in 12 parts instead of 2928 triangles

Geometry provided to OptiX in form of ray intersection code

PMT intersection by comparison with part intersections: cylinder and partial sphere intersection from quadratic roots

Volume to Surface translation

Volume heirarchy: Pyrex/Vacuum/Bialkali puts photocathode inside vacuum but as coincident boundaries it makes no difference for volume description.

Coincident boundaries do not work for surface description, must adopt correct heirarchy: Pyrex/Bialkali/Vacuum

TODO: single PMT testing vs G4 to check implications

Sphere intersection only 2 cases, Cylinder 10 cases (axial, walls, endcap, outside/inside)

OptiX Ray Traced Analytic PMT geometry

/env/nuwa/detdesc/pmt/hemi-pmt-analytic-near-clipped.png

Near clipped, orthographic projection.

Analytic PMTs together with triangulated geometry

Dayabay Ray Trace: PMTs analytic, the rest triangulated

/env/nuwa/detdesc/pmt/analytic-pmt-optix-geometry.png

Opticks/Geant4 : Dynamic Test Geometry

Commandline arguments parsed into geometry/material/surface description. Each test shape requires:

Shape OptiX OpenGL Geant4
sphere Y Y Y
box needs debug Y Y
prism Y Y  
convex lens Y Y  

Compare Opticks/Geant4 propagations with simple test geometries with:

     ggv.sh        --test --testconfig "..."  --torch --torchconfig "..."   # Opticks
     ggv.sh --cfg4 --test --testconfig "..."  --torch --torchconfig "..."   # Geant4

Opticks Absolute Reflection compared to Fresnel expectation

Comparison of simulated absolute reflection of S and P polarized single events against expectation from Fresnel formula. Using uniform planar incident cyclindrically directed light.

/env/numerics/npy/reflection.png

Multiple Rainbows from a Single Drop of Water

/env/presentation/rainbow-mechanism.png

Caustic bunching at least deviation causes rainbow

  1. Primary bow, single reflection : ray f at bow angle
  2. Secondary bow, double reflection : ray g at bow angle
  3. Deviation angles 0:180 degrees hemisphere opposite to incident rays

Jearl D. Walker, 1975, Multiple rainbows from single drops of water and other liquids http://patarnott.com/atms749/pdf/MultipleRainbowsSingleDrops.pdf

Disc beam 1M Photons incident on Water Sphere (S-Pol)

Photons shown by lines with color representing polarization direction. S-Polarized (perpendicular to plane of incidence) intersection by disc radially directed polarization. Geodesic icosahedron tesselation just for OpenGL visualization, actual OptiX geometry is perfect sphere.

/env/graphics/ggeoview/rainbow-spol-disc-incident-sphere.png

2ns later, Several Bows Apparent

/env/graphics/ggeoview/rainbow-spol-disc-several-bows.png

Opticks/Geant4 Photon Step Sequence Comparison

1M flag sequences indexed using CUDA Thrust, 0.040 s

 64-bit uint  Opticks    Geant4    chi2                                      (tag:5,-5)

        8ccd   819160    819654    0.15  [4 ] TO BT BT SA                    (cross droplet)
         8bd   102087    101615    1.09  [3 ] TO BR SA                       (external reflect)
       8cbcd    61869     61890    0.00  [5 ] TO BT BR BT SA                 (bow 1)
      8cbbcd     9618      9577    0.09  [6 ] TO BT BR BR BT SA              (bow 2)
     8cbbbcd     2604      2687    1.30  [7 ] TO BT BR BR BR BT SA           (bow 3)
    8cbbbbcd     1056      1030    0.32  [8 ] TO BT BR BR BR BR BT SA        (bow 4)
       86ccd     1014      1000    0.10  [5 ] TO BT BT SC SA
   8cbbbbbcd      472       516    1.96  [9 ] TO BT BR BR BR BR BR BT SA     (bow 5)
         86d      498       473    0.64  [3 ] TO SC SA
  bbbbbbbbcd      304       294    0.17  [10] TO BT BR BR BR BR BR BR BR BR  (bow 8+ truncated)
  8cbbbbbbcd      272       247    1.20  [10] TO BT BR BR BR BR BR BR BT SA  (bow 6)
  cbbbbbbbcd      183       161    1.41  [10] TO BT BR BR BR BR BR BR BR BT  (bow 7 truncated)
         4cd      161       139    1.61  [3 ] TO BT AB
       8c6cd      153       106    8.53  [5 ] TO BT SC BT SA
        86bd      138       142    0.06  [4 ] TO BR SC SA
        4ccd      100       117    1.33  [4 ] TO BT BT AB

Rainbow deviation angles

Deviation angle distribution of all 3M photons. Photon wavelengths from Plankian 6500K blackbody spectrum (implemented with inverse CDF GPU texture lookup). Simulated "images" obtained from wavelength spectrum of each bin using CIEXYZ weighting functions converted into sRGB/D65 colorspace. The two images exposed for luminance (CIE-Y) of max bin of 1st and 2nd bows.

/env/numerics/npy/rainbow16_deviation_angle.png

Rainbow Spectrum for 1st six bows

Spectra obtained by selecting photons by internal reflection counts. Colors obtained from spectra of each bin using CIEXYZ weighting functions converted into sRGB/D65 colorspace. Exposures by normalizing to bin with maximum luminance (CIE-Y) of each bow. White lines indicate geometric optics prediction of deviation angle ranges of the visible range 380-780nm. 180-360 degrees signifies exit on same side of droplet as incidence.

/env/numerics/npy/rainbow6-spectrum.png

1M Rainbow S-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. S-Polarized intersection (E field perpendicular to plane of incidence) arranged by directing polarization radially.

/env/optix/cfg4/rainbow-cfg4-spol.png

1M Rainbow P-Polarized, Comparison Opticks/Geant4

Deviation angle(degrees) of 1M parallel monochromatic photons in disc shaped beam incident on water sphere. Numbered bands are visible range expectations of first 11 rainbows. P-Polarized intersection (E field within plane of incidence) arranged by directing polarization tangentially.

/env/optix/cfg4/rainbow-cfg4-ppol.png

Performance Comparison Opticks/Geant4

Average Propagate Time for 1M photons

MacBook Pro 2013, NVIDIA GeForce GT 750M 2GB (384 cores)

Rainbow Test Geant4 10.2 Opticks Interop Opticks Compute
1M (S-Pol) 56 s 1.62 s 0.28 s
1M (P-Pol) 58 s 1.71 s 0.25 s

Opticks Interop/Compute Modes

Summary

Opticks Transition Complete

Opticks Validation

Next Tests

JPMT Wide

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-wide_half.png

JPMT Before Contact

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-before-contact_half.png

JPMT After Contact

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-after-contact_half.png

JPMT Approach

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-approach_half.png

JPMT Arrival

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-arrival_half.png

JPMT Inside Outside

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-inside-outside_half.png

JPMT Headview

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-headview_half.png

JPMT Backview

http://simoncblyth.bitbucket.io/env/graphics/ggeoview/jpmt-backview_half.png

EXTRAS : Details and Tests in progress

Details

Tests requiring G4 comparison

Optical Photons now fully GPU resident

All photon operations now done on GPU:

Only PMT hits need to be copied back to CPU

Thrust/CUDA/OptiX interop used to implement

Volume to Surface Translation details need validation

Zoomed view of PMT edge (mm): showing 3mm Pyrex,

/env/nuwa/detdesc/pmt/analytic-pmt-detail.png

Volume based geometry can get away with coincident boundaries, Surface based geometry cannot

OptiX Geometry Experience

Geometry Issues Fixed/Avoidable

  1. more of a problem for testing than actual usage

Prism S-Polarized

Quadrant of cylindrically directed S-polarized photons at 10 wavelengths (from 100 to 800 nm) incident from left.

S-polarization: perpendicular to incident plane

/env/numerics/npy/prism-spol-ripple.png

Prism P-Polarized

Gap in reflection at Brewsters angle is apparent where the P-polarized photons can only be transmitted.

P-polarization: not perpendicular to incident plane

/env/numerics/npy/prism-ppol-ripple.png

Prism Deviation vs Incident angles for 10 wavelengths

Prism geometry and Snell's law at two refractions allows deviation angle vs incident angle to be predicted. Comparison of simulation results with expectations for 10 wavelengths using refractive index of Schott F2 Flint Glass.

/env/numerics/npy/prism-deviation.png

Lens focussing

Lens constructed from intersection of two spheres. Disc parallel beam incident from left. Color represents polarization direction. 2nd reflections are apparent.

/env/graphics/ggeoview/lens-polcolor.png