Review Opticks + JUNO

Outline

• Introduction (pp 2-7)
  • Optical Photon Simulation Problem...
  • NVIDIA OptiX Ray Tracing Engine, NVIDIA OptiX 7
  • Geant4 + Opticks Hybrid Workflow
  • Opticks : Translates G4 Geometry to GPU, Without Approximation
• 2020 : Leap in Opticks awareness and Users (pp 8-10)
• Geant4+Opticks News (pp 11-14)
  • G4OpticksTest : Fermilab Geant4 team
  • Geant4 Bugs 2305,2311
  • UK SWIFT-HEP grant : SoftWare InFrastructure and Technology for HEP
• Opticks User News (pp 15-18)
  • LHCb RICH study, Sajan Easo
  • Opticks+CORSIKA8 next-generation neutrino telescope optimization, Fan Hu
  • LZ(dark matter search) prepare for Perlmutter with NVIDIA/NERSC engineers
• JUNO+Opticks Progress (pp 19-20)
  • Efficiency Hit Culling on GPU
  • Completing JUNO hits
• JUNO+Opticks Plan (pp 21-23)
  • Three levels of performance
  • OpticksServer + Lightweight OpticksClients
  • Releases for Opticks+JUNO validation comparing G4+Opticks vs G4
• Summary and Links (p24)

JUNO Optical Photon Simulation Problem...

NVIDIA® OptiX™ Ray Tracing Engine -- http://developer.nvidia.com/optix

OptiX Raytracing Pipeline

Analogous to OpenGL rasterization pipeline:

OptiX makes GPU ray tracing accessible

• accelerates ray-geometry intersections
• simple : single-ray programming model
• "...free to use within any application..."
• access RT Cores[1] with OptiX 6.0.0+ via RTX™ mode

NVIDIA expertise:

• ~linear scaling up to 4 GPUs
• acceleration structure creation + traversal (Blue)
• instanced sharing of geometry + acceleration structures
• compiler optimized for GPU ray tracing

https://developer.nvidia.com/rtx

User provides (Yellow):

• ray generation
• geometry bounding box, intersects

[1] Turing+ GPUs eg NVIDIA TITAN RTX

NVIDIA OptiX 7 : Entirely new thin API

NVIDIA OptiX 7 Features

• Minimal host state
• GPU memory managed by application
• GPU launches : explicit, asynchronous (CUDA streams)
• All host functions are thread-safe
• Multi-GPU managed by application

"managed by application" -> difficult development

• lots of learning + expert assistance needed

Disadvantage

Demands much more developer effort than OptiX 6

Advantage

More control/flexibility over everything.

• thread-safe : important for distributed Server/Client Opticks

Geant4OpticksWorkflow

Opticks : Translates G4 Geometry to GPU, Without Approximation

G4 Structure Tree -> Instance+Global Arrays -> OptiX

Group structure into repeated instances + global remainder:

• auto-identify repeated geometry with "progeny digests"
  • JUNO (SVN trunk) : 9 distinct instances + 1 global
• instance transforms used in OptiX/OpenGL geometry

instancing -> huge memory savings for JUNO PMTs

j1808_top_rtx

2020 : Leap in Opticks Awareness : Meetings, Publications >>> New Users

(Feb 2020) Workshop on Efficient Computing for HEP, Edinburgh

• https://indico.ph.ed.ac.uk/event/66/ "Full Simulation of HEP Detectors with Geant4", Ben Morgan

(May 2020) HSF (HEP Software Foundation) Meeting : GPUs in Simulations

• https://indico.cern.ch/event/921244/ "Opticks Development Experience : Problems and Successes", Simon Blyth

(10+24 June 2020) HSF Meeting : R&D on accelerators in Simulations 3/9 talks discussed Opticks

• https://indico.cern.ch/event/925887/ https://indico.cern.ch/event/930881/
• "e/γ (GPU) calorimeter simulation.. ", John Apostolakis, "Prototyping simulation .. on GPUs", Andrei Gheata
• "Prospects for (LHCb) RICH detector simulation using OptiX in GPUs", Sajan Easo

(Aug 2020) HSF Publication : HL-LHC Computing Review: Common Tools and Community Software

• https://zenodo.org/record/4009114 Graeme Stewart,... HL-LHC Computing Review discusses Opticks

(Sep 2020) Geant4 : R&D Task Force Meeting 3+ Geant4 teams pursuing GPUs, 1 based on Opticks

• https://indico.cern.ch/event/942142/sessions/363813/#20200915

1. "G4Opticks for liquid Argon TPCs", Fermilab Geant4 Team, Hans Wenzel
2. "VecGeom@GPU", CERN Geant4 Team, Andrei Gheata

2020 : Leap in Opticks Awareness >>> Some Users Getting Serious

(Aug 2020) Shanghai SJTU : Next-gen Neutrino Telescope Simulation Workshop

• https://indico-tdli.sjtu.edu.cn/event/238/sessions/109/#20200813
• "Opticks : GPU/Graphics backgroup + Application to underwater neutrino telescope simulations ?", Simon Blyth
• SJTU team evaluating Opticks for deep ocean expt simulations (Fan Hu, Donglian Xu)

(Aug 2020) Snowmass 2021 : Three LoI based on Opticks : Multi-expt Dark Matter group invited me to sign LoI

1. "Opticks : GPU photon simulation via NVIDIA OptiX", Simon Blyth
2. "Fast simulations for Noble Liquid experiments", XENON, DARWIN, LZ teams, Simon Blyth
3. "Simulating Optical Photons in HEP experiments on GPUs", Fermilab Geant4 team

(Aug 2020) NERSC Users Group (NUG) meeting : LZ aiming to deploy Opticks on Perlmutter supercomputer

• https://www.nersc.gov/users/NUG/annual-meetings/nug-2020/
• https://www.nersc.gov/assets/Uploads/1400-Opticks-on-CoriGPU.pdf
• "Opticks on CoriGPU", O.Creaner, 5 authors from : NERSC/LBL/LZ/Stanford/Bristol
• Opticks in Shifter/Docker container -> CoriGPU -> Perlmutter Cray/NVIDIA supercomputer
• Perlmutter : > 6000 NVIDIA A100 (Ampere Architecture GPU)

https://insidehpc.com/2020/05/perlmutter-supercomputer-to-include-more-than-6000-nvidia-a100-processors/

GROUPS_IO_1

G4OpticksTest : Demonstrates Geant4 + Opticks hybrid workflow

(Nov 2020) HSF WLCG Virtual Workshop : https://indico.cern.ch/event/941278/timetable/#20201123.detailed

• Hans Wenzel, Fermilab Geant4 Team

Worked closely with Hans for several years

• Developed Geant4-Opticks integration API G4Opticks
• Recently reported two Geant4 optical surface issues #2305 #2311

Geant4_Bug_2305

Unfortunately it took me five releases to notice this showstopper

Any GDML using expt. with > 1 optical surface, should not use these five releases of Geant4.

Important to test ahead, checking lastest Geant4:

• natural way to contribute to Geant4 and raises profile
• avoids : surprises, excluded releases
• ease future "official" updating

Geant4_Bug_2311

Problem noticed in first release

Geant4 team seem willing to follow my suggestion that will avoid any lasting impact.

UK SWIFT-HEP grant : SoftWare InFrastructure and Technology for HEP

SWIFT-HEP : Newly funded (1.5M GBP over 3 yrs)

University of Manchester (UOM), Adam Davis

• "... we identified Opticks as a major contribution area ..."

(Jan 2021) SwiftHep/ExcaliburHep workshop presentation

Ben Morgan, University of Warwick + Geant4

• https://indico.cern.ch/event/976081/

LHCb RICH (Ring Imaging Cherenkov) with Opticks : Sajan Easo

(June 2020) HSF (HEP Software Foundation) Meeting

• https://indico.cern.ch/event/930881/
• "Prospects for RICH simulation using Opticks", Sajan Easo (STFC-GB)
• Standard Geant4 costs ~30% CPU time for optical photons
• successful validations

Opticks+CORSIKA8 for next-generation neutrino telescope optimization ?

(Aug 2020) Shanghai SJTU : Next-gen Neutrino Telescope Simulation Workshop

• https://indico-tdli.sjtu.edu.cn/event/238/sessions/109/#20200813
• "Opticks : GPU/Graphics backgroud + Application to underwater neutrino telescope simulations ?", Simon Blyth
• SJTU team evaluating Opticks for deep ocean expt simulations (Fan Hu, Donglian Xu)
  • enormous instrumented volumes O(10) km^3 water/ice
  • strings of pods geometry
  • huge numbers of Cherenkov photons propagating across enormous volumes, how to simulate ?
  • CORSIKA 8 + Opticks : trying to setup genstep transport

LZ+Opticks history : 2018-2021 : contacts with >5 LZ people

• Summer 2020 : LZ+Opticks simulation identified as candidate for new Perlmutter Cray/NVIDIA supercomputer at NERSC
• Autumn 2020 : LZ+NERSC proposed an NVIDIA assisted Hackathon to optimize LZ+Opticks simulation
  • BUT no point optimizing prior to migration : NVIDIA OptiX 6 -> 7 (an entirely new "Vulkan-ized" API)
  • migration to NVIDIA OptiX 7+ is unavoidable, it beings challenges and opportunities
    • multi-GPU scaling no longer provided, thread-safe OptiX 7 opens possibilities, especially for Opticks Server+Client

(Aug 2020) NERSC Users Group (NUG) meeting : LZ aiming to deploy Opticks on Perlmutter supercomputer

• https://www.nersc.gov/users/NUG/annual-meetings/nug-2020/
• "Opticks on CoriGPU", O.Creaner, 5 authors from : NERSC/LBL/LZ/Stanford/Bristol

2021 : LZ+Opticks simulation on Perlmutter (>6000 NVIDIA A100)

(8 Jan 2021) Zoom Introduction Meeting of >12 people : LZ/NERSC(6), NVIDIA(5), Opticks(1)

• NERSC + NVIDIA teams: senior engineers + postdoc "workers"
• NVIDIA team : experienced with OptiX 6->7, multi-GPU + multi-threaded CUDA/OptiX

Prepared Opticks codebase orientation for developers

• https://simoncblyth.bitbucket.io/opticks/docs/orientation.html

From February 8:

• plan bi-weekly working meetings
• migration assistance

Adapt Opticks to all new NVIDIA OptiX 7 API

• necessary : otherwise forever stuck at OptiX 6.5 (2019)
  • pre-requisite to OpticksServer
• re-implement optixrap (OptiX wrapper) package
• first: single-GPU only
• expect no change to G4Opticks API

Opticks + JUNO Progress : Efficiency Hit Culling on GPU

• Reduce CPU memory for hits by a factor of the efficiency
• only collected hits consume CPU memory

Culling result photon flags set on GPU:

• EFFICIENCY_COLLECT
• EFFICIENCY_CULL

Testing angular efficiency GPU texture machinery

• Mock striped theta efficiency, two random categories
• cosine phi efficiency variation for one category

https://simoncblyth.bitbucket.io/opticks/optixrap/tests/OSensorLibGeoTest.cc.html

https://bitbucket.org/simoncblyth/opticks/src/master/notes/progress.rst

Completing JUNO Hits : G4OpticksHit (4,4) + G4OpticksHitExtra (2,4)

void G4Opticks::getHit(unsigned i, G4OpticksHit* hit, G4OpticksHitExtra* hit_extra ) const

struct G4OpticksHit // from photon->hit buffers
{  // global+local,... all from (4,4) photon item
    G4ThreeVector local_position ;
    G4ThreeVector global_position ;
    G4double      time ;
    G4ThreeVector local_direction ;
    G4ThreeVector global_direction ;
    G4double      weight ;
    G4ThreeVector local_polarization ;
    G4ThreeVector global_polarization ;
    G4double      wavelength ;
    G4int         boundary ;
    G4int         sensorIndex ;
    G4int         nodeIndex ;
    G4int         photonIndex ;
    G4int         flag_mask ;
    G4int         sensor_identifier ;
    G4bool        is_cerenkov ;
    G4bool        is_reemission ;
};
struct G4OpticksHitExtra // from way->hiy buffers
{  // extras from (2,4) "way" item
    G4ThreeVector boundary_pos ;
    G4double      boundary_time ;
    G4double      origin_time ;
    G4int         origin_trackID ;
};

Geant4 approach

// Simulation/DetSimV2/PMTSim/src/junoSD_PMT_v2.cc
junoHit_PMT* hit = new junoHit_PMT(); // junoSD_PMT_v2::ProcessHits 
hit->SetPMTID(pmtid);
hit->SetWeight(1.0);
hit->SetTime(hittime);
hit->SetWavelength(wavelength);
hit->SetKineticEnergy(edep);
hit->SetPosition(local_pos);
hit->SetTheta(local_pos.theta());
hit->SetPhi(local_pos.phi());
hit->SetMomentum(local_dir);
hit->SetPolarization(local_pol);
hit->SetGlobalPosition(global_pos);
hit->SetGlobalMomentum(track->GetMomentum());
hit->SetGlobalPolarization(track->GetPolarization());
// optional extras normally from  NormalTrackInfo/NormalAnaMgr
hit->SetProducerID(producerID);
hit->SetFromCerenkov(is_from_cerenkov);
hit->SetReemission(is_reemission);
hit->SetOriginalOP(is_original_op);
hit->SetOriginalOPStartT(t_start);
hit->SetBoundaryPosition(boundary_pos); // pAcrylic/pInnerWater[1] 

• split: Hit (4,4) + HitExtra (2,4) OR combine: HitFull (6,4) ?
• Using split assuming extras switched off in large productions ?

[1] set in NormalTrackInfo by NormalAnaMgr::UserSteppingAction

Opticks : Three Levels of Performance

• https://bitbucket.org/simoncblyth/opticks/src/master/notes/performance.rst

1. optical: time+memory to simulate photons (1 CPU, N GPUs)
2. node: overall simulation performance (1 CPU, N GPUs)
3. cluster: overall batch performance creating large samples

For example with N = 4

• best optical performance
• node : ~same as 1 GPU per CPU (dominated by non-optical)
• cluster : significantly worse : as limiting concurrent jobs

Experimentation required to find optimum

• CPU only OpticksClients -> GPU cluster OpticksServer

Flexibility -> maximizes resource usage -> solves problems:

• limited numbers of NVIDIA GPUs
• GPU starvation : difficult to keep them busy

OpticksServer + Lightweight OpticksClients

General compute server structure:

• receive/queue/reply NumPy arrays (eg gensteps, hits) over TCP
• decouple computation from queue/network infrastructure code

Exploration phase: find/evaluate open source basis projects

• load balancer, proxy : HAProxy, Squid
• task queue : https://taskqueues.com

Plan ? too early to set timeline

• priority depends on experience with GPU constrained Opticks

computing student/postdoc/.. welcome to assist

• needs network/MQ/server/multi-threading experience~interest

[1] OKOP package : no Geant4, OpenGL : (Opticks structured as tree of ~20 packages according to external dependencies, only the four highest level packages depend on Geant4)

Releases for Opticks+JUNO validation comparing G4+Opticks vs G4

Summary and Links

Opticks : state-of-the-art GPU ray traced optical simulation integrated with Geant4. Now integrated with JUNO Offline Pioneer users welcome

• Opticks gained many new users in 2020, some getting serious
• JUNO Opticks users are needed to assist with validation

geocache_360

Integration of JUNO + Opticks : Opticks is a JunoENV External

Add Opticks to an existing installation (SVN trunk r3990+):

bash junoenv libs all opticks

• pre-requisites : NVIDIA Driver, CUDA, OptiX

Four mandatory envvars :

export CMTEXTRATAGS=opticks 
# -> Opticks ON in JunoENV + CMT
export OPTICKS_COMPUTE_CAPABILITY=70  # -> nvcc flags
export OPTICKS_CUDA_PREFIX=/usr/local/cuda-10.1
export OPTICKS_OPTIX_PREFIX=/usr/local/OptiX_650
# location of CUDA and OptiX installs 

Build all JUNO externals, including Opticks (takes ~1hr, 13G) :

jlibs(){
    mkdir -p $JUNOTOP && cd $JUNOTOP
    svn co https://juno.ihep.ac.cn/svn/offline/trunk/installation/junoenv && cd $JUNOTOP/junoenv
    local libs=$(bash junoenv libs list | perl -ne 'm, (\S*)@, && print "$1\n"' -)
    for lib in $libs ; do
        bash junoenv libs all $lib || return 1
    done
}

• https://juno.ihep.ac.cn/mediawiki/index.php/Offline:Installation#Install_Opticks_using_JunoENV

Integration of Offline + Opticks : Where WITH_G4OPTICKS used ?

Offline Opticks : #ifdef WITH_G4OPTICKS ... #endif + if(m_opticksMode > 0){ ... }

cd $JUNOTOP/offline/Simulation/DetSimV2 ; find . -type f -exec grep -l WITH_G4OPTICKS {} \+

./DetSimOptions/src/DetSim0Svc.cc                  ## initialize/finalize
./DetSimOptions/src/LSExpDetectorConstruction.cc   ## translate geometry to GPU

./PhysiSim/src/DsG4Scintillation.cc    ## collect Scintillation gensteps, excluding reemission
./PhysiSim/src/LocalG4Cerenkov1042.cc  ## collect Cerenkov gensteps
./PhysiSim/src/DsPhysConsOptical.cc    ## use LocalG4Cerenkov1042 rather than G4Cerenkov

./PMTSim/src/PMTSDMgr.cc         ## setup separate PMTHitMerger m_pmthitmerger_opticks
./PMTSim/src/junoSD_PMT_v2.cc    ## bulk hit creation+collection at EndOfEvent

junoSD_PMT_v2::Initialize

• setup extra hitCollection_opticks (temporarily doing both GPU+CPU sim, with separate hits + merger)

junoSD_PMT_v2::EndOfEvent_Opticks

• invoke G4Opticks::propagateOpticalPhotons with the gensteps collected
• bulk populate hitCollection_opticks

• https://juno.ihep.ac.cn/mediawiki/index.php/Offline:Installation#building_the_offline_-DWITH_G4OPTICKS

Integration of Offline + Opticks : tut_detsim.py options

tut_detsim.py options relevant to Opticks usage : TODO: user friendly Opticks control interface

--opticks-mode 0

do not use Opticks GPU propagation, the default

--opticks-mode 1 (currently SVN trunk r3990)

performs both CPU and GPU optical simulations put GPU hits into separate hitCollectionOpticks

--no-guide_tube

change geometry skipping the guide tube torus

--pmt20inch-polycone-neck

change geometry of LPMT necks avoiding use of torus

tds : Example Bash Function :

tds-(){ python $JUNOTOP/offline/Examples/Tutorial/share/tut_detsim.py $* ; }
tds(){ tds- --opticks-mode 1 --no-guide_tube --pmt20inch-polycone-neck --evtmax 2 gun ; }

• https://juno.ihep.ac.cn/mediawiki/index.php/Offline:Detsimuserguide/UsingOpticks

JUNO Geometry : Remove Torus to enable translation to GPU

tut_detsim.py options (in SVN trunk r3990)

--no-guide_tube

skip the guide tube torus

--pmt20inch-polycone-neck

"cylinder - torus" -> polycone

• NNVT_MCPPMT_PMTSolid
• Hamamatsu_R12860_PMTSolid

--pmt20inch-simplify-csg

simpler+faster CSG, same geometry

• avoids Inner_Separator anti-pattern