Review Opticks + JUNO

Review JUNO + Opticks :
GPU Optical Simulation with NVIDIA® OptiX™

Open source,

Simon C Blyth, IHEP, CAS — JUNO Collaboration Meeting, Jan/Feb 2021, Virtual



JUNO Optical Photon Simulation Problem...

NVIDIA® OptiX™ Ray Tracing Engine --

OptiX makes GPU ray tracing accessible

NVIDIA expertise:

User provides (Yellow):

[1] Turing+ GPUs eg NVIDIA TITAN RTX

NVIDIA OptiX 7 : Entirely new thin API

NVIDIA OptiX 7 Features

"managed by application" -> difficult development

Demands much more developer effort than OptiX 6

More control/flexibility over everything.

  • thread-safe : important for distributed Server/Client Opticks


Opticks : Translates G4 Geometry to GPU, Without Approximation

G4 Structure Tree -> Instance+Global Arrays -> OptiX

Group structure into repeated instances + global remainder:

instancing -> huge memory savings for JUNO PMTs



2020 : Leap in Opticks Awareness : Meetings, Publications >>> New Users

(Feb 2020) Workshop on Efficient Computing for HEP, Edinburgh

(May 2020) HSF (HEP Software Foundation) Meeting : GPUs in Simulations

(10+24 June 2020) HSF Meeting : R&D on accelerators in Simulations 3/9 talks discussed Opticks

(Aug 2020) HSF Publication : HL-LHC Computing Review: Common Tools and Community Software

(Sep 2020) Geant4 : R&D Task Force Meeting 3+ Geant4 teams pursuing GPUs, 1 based on Opticks

  1. "G4Opticks for liquid Argon TPCs", Fermilab Geant4 Team, Hans Wenzel
  2. "VecGeom@GPU", CERN Geant4 Team, Andrei Gheata

2020 : Leap in Opticks Awareness >>> Some Users Getting Serious

(Aug 2020) Shanghai SJTU : Next-gen Neutrino Telescope Simulation Workshop

(Aug 2020) Snowmass 2021 : Three LoI based on Opticks : Multi-expt Dark Matter group invited me to sign LoI

  1. "Opticks : GPU photon simulation via NVIDIA OptiX", Simon Blyth
  2. "Fast simulations for Noble Liquid experiments", XENON, DARWIN, LZ teams, Simon Blyth
  3. "Simulating Optical Photons in HEP experiments on GPUs", Fermilab Geant4 team

(Aug 2020) NERSC Users Group (NUG) meeting : LZ aiming to deploy Opticks on Perlmutter supercomputer


G4OpticksTest : Demonstrates Geant4 + Opticks hybrid workflow

(Nov 2020) HSF WLCG Virtual Workshop :

Worked closely with Hans for several years


Unfortunately it took me five releases to notice this showstopper

Any GDML using expt. with > 1 optical surface, should not use these five releases of Geant4.

Important to test ahead, checking lastest Geant4:


Problem noticed in first release

Geant4 team seem willing to follow my suggestion that will avoid any lasting impact.

UK SWIFT-HEP grant : SoftWare InFrastructure and Technology for HEP

SWIFT-HEP : Newly funded (1.5M GBP over 3 yrs)

University of Manchester (UOM), Adam Davis

(Jan 2021) SwiftHep/ExcaliburHep workshop presentation

Ben Morgan, University of Warwick + Geant4

LHCb RICH (Ring Imaging Cherenkov) with Opticks : Sajan Easo

(June 2020) HSF (HEP Software Foundation) Meeting

Opticks+CORSIKA8 for next-generation neutrino telescope optimization ?

(Aug 2020) Shanghai SJTU : Next-gen Neutrino Telescope Simulation Workshop

LZ+Opticks history : 2018-2021 : contacts with >5 LZ people

(Aug 2020) NERSC Users Group (NUG) meeting : LZ aiming to deploy Opticks on Perlmutter supercomputer

2021 : LZ+Opticks simulation on Perlmutter (>6000 NVIDIA A100)

(8 Jan 2021) Zoom Introduction Meeting of >12 people : LZ/NERSC(6), NVIDIA(5), Opticks(1)

Prepared Opticks codebase orientation for developers

From February 8:

Adapt Opticks to all new NVIDIA OptiX 7 API

Opticks + JUNO Progress : Efficiency Hit Culling on GPU

Culling result photon flags set on GPU:

Testing angular efficiency GPU texture machinery

Completing JUNO Hits : G4OpticksHit (4,4) + G4OpticksHitExtra (2,4)

void G4Opticks::getHit(unsigned i, G4OpticksHit* hit, G4OpticksHitExtra* hit_extra ) const

Geant4 approach

// Simulation/DetSimV2/PMTSim/src/
junoHit_PMT* hit = new junoHit_PMT(); // junoSD_PMT_v2::ProcessHits 
// optional extras normally from  NormalTrackInfo/NormalAnaMgr
hit->SetBoundaryPosition(boundary_pos); // pAcrylic/pInnerWater[1] 

[1] set in NormalTrackInfo by NormalAnaMgr::UserSteppingAction

Opticks : Three Levels of Performance

  1. optical: time+memory to simulate photons (1 CPU, N GPUs)
  2. node: overall simulation performance (1 CPU, N GPUs)
  3. cluster: overall batch performance creating large samples

For example with N = 4

Experimentation required to find optimum

Flexibility -> maximizes resource usage -> solves problems:

OpticksServer + Lightweight OpticksClients

General compute server structure:

Exploration phase: find/evaluate open source basis projects

Plan ? too early to set timeline

computing student/postdoc/.. welcome to assist

[1] OKOP package : no Geant4, OpenGL : (Opticks structured as tree of ~20 packages according to external dependencies, only the four highest level packages depend on Geant4)

Releases for Opticks+JUNO validation comparing G4+Opticks vs G4

0.1.0 : "Basic Validation" Release : Enabling Statistical hit comparison : Estimate : During February
Completed: Remaining:
  • full JUNO hits :
    • G4OpticksHit : global+local
    • G4OpticksHitExtra : boundary_pos, origin_time, ..
  • GPU collection angular efficiency culling
    • ε [θ, φ] -> G4Opticks -> GPU "angular" texture
    • G4Opticks::setSensorData
    • reduces CPU hit memory by : ε . ε [θ, φ]
  • angular efficiency culling : make run-time controllable
  • G4OpticksHitExtra : make run-time optional
    • photon size: (4,4)+(2,4) -> (4,4)
    • half GPU->CPU hit copies
  • JUNO Offline+Opticks Bridge updates
    • adapt to new G4Opticks API
    • improve python commandline control of G4Opticks

students/postdocs with Geant4 geometry experience can assist with "Basic Validation" distribution comparison

0.2.0 : "Full Validation" Release : Enabling Random Aligned step-by-step comparison : Estimate March-April
Completed: Remaining:
  • cfg4 : CRandomEngine control random stream
    • -> random aligned bi-simulation
  • cfg4 : Geant4 SteppingAction emulates Opticks
    • -> OpticksEvent arrays -> direct NumPy compare
  • Integrate cfg4 random control with G4Opticks
    • adapt to general G4 environment (prev: optical only)

students/postdocs with NumPy + Geant4 geometry experience can assist with "Full Validation" step-by-step comparison


Integration of JUNO + Opticks : Opticks is a JunoENV External

Add Opticks to an existing installation (SVN trunk r3990+):

bash junoenv libs all opticks

Four mandatory envvars :

export CMTEXTRATAGS=opticks 
# -> Opticks ON in JunoENV + CMT
export OPTICKS_COMPUTE_CAPABILITY=70  # -> nvcc flags
export OPTICKS_CUDA_PREFIX=/usr/local/cuda-10.1
export OPTICKS_OPTIX_PREFIX=/usr/local/OptiX_650
# location of CUDA and OptiX installs 

Build all JUNO externals, including Opticks (takes ~1hr, 13G) :

    mkdir -p $JUNOTOP && cd $JUNOTOP
    svn co && cd $JUNOTOP/junoenv
    local libs=$(bash junoenv libs list | perl -ne 'm, (\S*)@, && print "$1\n"' -)
    for lib in $libs ; do
        bash junoenv libs all $lib || return 1

Integration of Offline + Opticks : Where WITH_G4OPTICKS used ?

Offline Opticks : #ifdef WITH_G4OPTICKS ... #endif + if(m_opticksMode > 0){ ... }

cd $JUNOTOP/offline/Simulation/DetSimV2 ; find . -type f -exec grep -l WITH_G4OPTICKS {} \+

./DetSimOptions/src/                  ## initialize/finalize
./DetSimOptions/src/   ## translate geometry to GPU

./PhysiSim/src/    ## collect Scintillation gensteps, excluding reemission
./PhysiSim/src/  ## collect Cerenkov gensteps
./PhysiSim/src/    ## use LocalG4Cerenkov1042 rather than G4Cerenkov

./PMTSim/src/         ## setup separate PMTHitMerger m_pmthitmerger_opticks
./PMTSim/src/    ## bulk hit creation+collection at EndOfEvent
  • setup extra hitCollection_opticks (temporarily doing both GPU+CPU sim, with separate hits + merger)
  • invoke G4Opticks::propagateOpticalPhotons with the gensteps collected
  • bulk populate hitCollection_opticks

Integration of Offline + Opticks : options options relevant to Opticks usage : TODO: user friendly Opticks control interface

--opticks-mode 0
do not use Opticks GPU propagation, the default
--opticks-mode 1 (currently SVN trunk r3990)
performs both CPU and GPU optical simulations put GPU hits into separate hitCollectionOpticks
change geometry skipping the guide tube torus
change geometry of LPMT necks avoiding use of torus

tds : Example Bash Function :

tds-(){ python $JUNOTOP/offline/Examples/Tutorial/share/ $* ; }
tds(){ tds- --opticks-mode 1 --no-guide_tube --pmt20inch-polycone-neck --evtmax 2 gun ; }

JUNO Geometry : Remove Torus to enable translation to GPU options (in SVN trunk r3990)

skip the guide tube torus

"cylinder - torus" -> polycone

  • Hamamatsu_R12860_PMTSolid

simpler+faster CSG, same geometry

  • avoids Inner_Separator anti-pattern