JUNOSW + Opticks : Planning Release

JUNOSW + Opticks : Release Planning

NVIDIA Driver + CUDA + OptiX versions : Versions for GPU Cluster ?
Info on GPU Cluster from Du Ran
JUNOSW + Opticks : "Technical" Release : Which versions ?
Opticks : Geometry Translation now using minimal intermediate model
A/B Comparison : input photon bi-simulation : insitu and standalone
Insitu input photon bi-simulation comparison : Histories involving 3inch PMT discrepant
Insitu input photon bi-simulation comparison : After increase tub3 delta 1e-3=>0.1

Simon C Blyth, IHEP, CAS — JOC Meeting — 7 Sept 2023

U4Mesh_test2

GEOM=xjfcSolid ~/opticks/u4/tests/U4Mesh_test2.sh
GEOM=xjacSolid ~/opticks/u4/tests/U4Mesh_test2.sh

GEOM=sjclSolid ~/opticks/u4/tests/U4Mesh_test2.sh
GEOM=sjfxSolid ~/opticks/u4/tests/U4Mesh_test2.sh
GEOM=sjrcSolid ~/opticks/u4/tests/U4Mesh_test2.sh
GEOM=sjrfSolid ~/opticks/u4/tests/U4Mesh_test2.sh

GEOM=facrSolid ~/opticks/u4/tests/U4Mesh_test2.sh

Handling these solids

j/PMTSim	Classname	Notes
xjfcSolid	XJfixtureConstruction (PE_PA)	lWaterPool : monstrous many box + cyl union
xjacSolid	XJanchorConstruction [2] (Acrylic)	lWaterPool : cone subtract big sphere union cylinder (small-big subtraction precision + perf issue, expect coin)
sjclSolid	SJCLSanchorConstruction [1] (Acrylic)	lTarget : bizarre union box and cone-(bigBox-bigSphere)
sjfxSolid	SJFixtureConstruction [1] (Acrylic)	lTarget : similar bizarre union box and cone-(bigBox-bigSphere) ORIGINAL WITH COMPILATION WARNING + SEGV BUG !!!!
sjrcSolid	SJReceiverConstruction [1] (Acrylic)	lTarget : again union cyl and cone-(bigBox-bigSphere)
sjrfSolid	SJReceiverFasternConstruction (PE_PA)	lTarget : another monstrous many box and cyl union
facrSolid	FastenerAcrylicConstruction (Steel) [3]	lWaterPool, no overlap annulus + 8 screw can become single listnode

The within lTarget SJ shapes marked [1] use eg cone-(bigBox-bigSphere) in order to conform the shape to the inside of the Acrylic ... but they are all Acrylic anyway ... suspect overlaps between these shapes into the Acrylic would give exactly the same simuluation results with much less cost.... as its all just Acrylic things poking into the Acrylic sphere.

Also the lWaterPool xjacSolid [2] subtracts bigSphere to conform to outside of the Acrylic : but again why worry about Acrylic overlapping Acrylic : its all the same material.

The facrSolid [3] looks highly amenable to being a single list node, as the 9 constituents do not overlap.

It looks likely that five of the above can be greatly improved... leaving the two monstrous PE_PA that will both need tree + listnode handling.

xjacSolid : XJanchorConstruction : lWaterPool

U4Mesh_test2_xjfcSolid_review.png

U4Mesh_test2_xjacSolid_review.png

U4Mesh_test2_sjclSolid_review.png

U4Mesh_test2_sjfxSolid_review.png

U4Mesh_test2_sjrcSolid_review.png

U4Mesh_test2_sjrfSolid_review.png

U4Mesh_test2_facrSolid_review.png

NVIDIA Driver + CUDA + OptiX versions : Versions for GPU Cluster ?

Minimal Opticks development expected to update from OptiX 7.0 to 7.1,7.2,...,8.0 (as only minor API changes)
- Opticks users from Fermilab Geant4 group + elsewhere already using OptiX 7.5, 7.6
- BUT : OptiX updates require : minimum NVIDIA Driver + specific CUDA version
- CUDA version => Minimum Linux kernel versions => default gcc version
strong potential for performance improvements just from updates
Measurements on Workstation as upgrade => desired GPU cluster versions : GPU cluster version constraints ?

Release Date	NVIDIA OptiX	Notes	Driver	CUDA	gcc
July 2019	7.0.0	NEW API	435.12(435.21)	10.1	8.3.0	<=current workstation versions
June 2020	7.1.0	Added Curves	450	11.0		Maybe: torus for guide tube
Oct 2020	7.2.0	Specialization	455	11.1
Apr 2021	7.3.0		465	11.1
Oct 2021	7.4.0	Catmull-Rom	495	11.4
June 2022	7.5.0 [1]	Debug, Sphere	515	11.7		LOOKS POSSIBLE ON CLUSTER
Oct 2022	7.6.0 [1]		520	11.8
Mar 2023	7.7.0	More Curves	530	12.0
Aug 2023	8.0.0	SER, Perf	535	12.0

[1] minor Opticks changes to support NVIDIA OptiX 7.5, 7.6 done already for Opticks users from Fermilab Geant4 group

Info on GPU Cluster from Du Ran

NVIDIA driver : 515.65.01
CUDA : 11.7
Linux kernel  :  3.10.0-1160.80.1.el7.x86_64
OS : CentOS Linux release 7.9.2009
GCC :
- default version on worker node: gcc 4.8.5
- higher version : gcc 7.5.0,
  /cvmfs/slurm.ihep.ac.cn/centos7.8/gcc-7.5.0/bin/gcc


182 v100 32GB nvlink GPU cards
  1 a100 40GB pci-e GPU card in partition gpu

JUNO can use 128 GPU cards at most.

=> CAN REQUEST Install : NVIDIA OptiX 7.5.0
Any timeline for Driver updates ?
- 520 (CUDA 11.8) :
- 530, 536 (CUDA 12.0) ?

JUNOSW + Opticks : "Technical" Release : Which versions ?

Objective : enable more people to get involved with Opticks validation

Geant4 10.4.2	Changing Geant4 ? high delay potential to : Custom4 + Opticks + JUNOSW
Custom4 0.1.7	small package : but deeply coupled with : Geant4 + JUNOSW + Opticks
OptiX 7.5	7.0 -> 7.5 : expect straightforward
Opticks v0.2.0 ?	last release https://github.com/simoncblyth/opticks/releases/tag/v0.1.6

Deferred geometry:

--no-guide_tube	OptiX 7.1 has curves which might allow G4Torus translation (dev. needed)
--debug-disable-xj	XJfixture XJanchor	Deep CSG trees require dev. to see if "listnode" (similar to G4MultiUnion) can provide solution
--debug-disable-sj	SJCLSanchor SJFixture SJReceiver SJFixture
--debug-disable-fa	FastenerAcrylic

Known missing physics:

sigma_alpha/polish ground surface handling : impl in progress
- DONE : G4RandGauss::shoot with CUDA using erfcinvf : Inverse complementary error function
- TODO : normal smearing following G4OpBoundaryProcess::GetFacetNormal
- TODO : qsim::propagate generalization to handle sigma_alpha/polish surfaces

Opticks : Geometry Translation now using minimal intermediate model

GGeo : Historical baggage from export/import era

+-------------------------------------+
| Geant4 ----> GGeo ----> CSGFoundry  |
|         X4       CSG_GGeo           |
+-------------------------------------+

Progress with adoption of minimal approach
Aspect of translation	Status
structural factorization	MATCHED
instance transforms	MATCHED
standard CSG	MATCHED
material/surf/boundary	MATCHED
CSG uncoincidence NEW	MATCHED
CSG tree balancing	DEFERRED

HUGE CODE REDUCTION : > HALVED

6 pkgs inactive : hundreds of classes/structs

Geometry translation:

Geant4 -> stree.h -> CSGFoundry -> IAS,GAS (OptiX 7+)

Minimal intermediate stree.h model (much less code, faster)
full translation now taking ~15s (formerly > 1min)
profit from experience : eg n-ary tree serialization, NPFold

U4Tree	G4VPhysicalVolume -> stree/snode (n-ary tree)
U4Solid	G4VSolid -> s_csg/sn.h (n-ary tree of constituent)
U4Material	G4Material -> NPFold/NP
U4Surface	G4LogicalSurface -> NPFold/NP

CSGFoundry/CSGImport: CSGSolid/CSGPrim/CSGNode <- stree/snode/s_csg/sn

A/B Comparison : input photon bi-simulation : insitu and standalone

A	Opticks (stree workflow, no GGeo) -> Opticks/SEvt -> .npy
B	Geant4 with Opticks/U4Recorder -> Opticks/SEvt -> .npy

SEvt : full optical history => statistical comparison of histories

Insitu : Full Geometry (with some skips)	Standalone : selected few volumes
`~/j/jx.bash:ntds3_noxjsjfa` [1]	`~/opticks/g4cx/tests/G4CXTest.sh`
tut_detsim.py : "main" steering GtOpticksTool : integrate Opticks input photons OPTICKS_INTEGRATION_MODE:3 bi-simulate ~/j/ntds/ntds3.sh : SEvt comparison	G4CXTest.cc : main G4CXApp.h : Geant4+Opticks App in single header G4CXTest.py : SEvt comparison

Investigation Steps:

insitu : find deviant histories for photons touching 3 inch PMT tube
standalone : Tub3inchPMTV3Manager reproduces history deviation
simtrace : shows pmt_solid degenerate with cntr_solid (cylindrical tube)
standalone : add Tub3inchPMTV3Manager__VIRTUAL_DELTA_MM control
- increase delta 1e-3 => 0.1 avoids degeneracy, fixes issue
insitu : confirms fix, SPMT now behaving : BUT LPMT virtual apex degeneracy issue
TODO : standalone LPMT check : NEED TO MOVE LPMT Virtual outwards by ~0.1 mm

[1] https://github.com/simoncblyth/j/blob/main/jx.bash

ntds3 : insitu input photon bi-simulation comparison A(OK)/B(G4) : Hist involving 3inch PMT discrepant

QCF qcf : c2sum :  2063.1819 c2n :   114.0000 c2per:    18.0981  C2CUT:   30
 c2sum/c2n:c2per(C2CUT)  2063.18/114:18.098 (30)

 np.c_[siq,_quo,siq,sabo2,sc2,sabo1][0:25]  ## A-B history frequency chi2 comparison
 [[' 0' 'TO BT BT BT BT SA                                         ' ' 0' ' 37494  37425' ' 0.0635' '     8      3']
  [' 1' 'TO BT BT BT BT SD                                         ' ' 1' ' 30866  30874' ' 0.0010' '     4      4']
  [' 2' 'TO BT BT BT BT BT SA                                      ' ' 2' ' 12382  12477' ' 0.3630' '  9412   9096']
  [' 3' 'TO BT BT BT BT BT SR SA                                   ' ' 3' '  3810   3794' ' 0.0337' ' 11059  10892']
  [' 4' 'TO BT BT BT BT BT SR SR SA                                ' ' 4' '  1998   1996' ' 0.0010' ' 10899  10879']
  [' 5' 'TO BT BT AB                                               ' ' 5' '   884    893' ' 0.0456' '    26     28']
  [' 6' 'TO BT BT BT BT BT SR SR SR SA                             ' ' 6' '   572    563' ' 0.0714' ' 14725  14727']
  [' 7' 'TO BT BT BT BT BR BT BT BT BT BT BT AB                    ' ' 7' '   473    440' ' 1.1928' '  3182   4895']
  [' 8' 'TO BT BT BT BT AB                                         ' ' 8' '   319    352' ' 1.6230' '   651     46']
  [' 9' 'TO BT BT BT BT BR BT BT BT BT BT BT SD                    ' ' 9' '   326    342' ' 0.3832' '  5262   5279']
  ['10' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT SA              ' '10' '   326    332' ' 0.0547' '  7444   7463']
  ['11' 'TO BT BT BT BT BT SR BR SA                                ' '11' '   309    328' ' 0.5667' ' 33584  33575']
  ['12' 'TO BT BT BT BT BR BT BT BT BT BT AB                       ' '12' '   321     52' '193.9973' '  1021  17293']
  ['13' 'TO BT BT BT BT BR BT BT BT BT BT SA                       ' '13' '    24    318' '252.7368' '  4471   1017']
  ['14' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT SD              ' '14' '   312    263' ' 4.1757' '  8147   8138']
  ['15' 'TO BT BT BT BT BR BT BT BT BT AB                          ' '15' '   279    264' ' 0.4144' '   646    940']
  ['16' 'TO BT BT BT BT BT SR SR SR BR SA                          ' '16' '   212    240' ' 1.7345' ' 14749  14746']
  ['17' 'TO BT BT BR BT BT BT SA                                   ' '17' '    10    238' '209.6129' '  2991     17']
  ['18' 'TO BT BT BT BT BT SR SR SR BR BT BT BT BT BT BT SA        ' '18' '     0    197' '197.0000' '    -1  15508']
  ['19' 'TO BT BT BT BR BT BT BT BT SA                             ' '19' '     9    194' '168.5961' '  3510    194']
  ['20' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT BT BT BT BT SD  ' '20' '   190    171' ' 1.0000' ' 16931  17569']
  ['21' 'TO BT BT BT BR BT BT BT BT AB                             ' '21' '   187      4' '175.3351' '   206  22156']
  ['22' 'TO BT BT BR BT BT BT AB                                   ' '22' '   183      3' '174.1935' '     2  39342']

 np.c_[siq,_quo,siq,sabo2,sc2,sabo1][bzero]  ## in A but not B
 [['38' 'TO BT BT BT BT BT SR SR SR BR BT BT BT BT BT BT BT AB     ' '38' '    91      0' '91.0000' ' 16654     -1']
  ['44' 'TO BT BT BT BT BT SR SR SR BR BT BT BT BT BT BT AB        ' '44' '    83      0' '83.0000' ' 15529     -1']
  ['56' 'TO BT BT BT BT BT SR SR BT BT BT BT BT BT BT BT SD        ' '56' '    56      0' '56.0000' ' 26920     -1']
  ['63' 'TO BT BT BT SA                                            ' '63' '    42      0' '42.0000' ' 49820     -1']
  ['76' 'TO BT BT BT SD                                            ' '76' '    34      0' '34.0000' ' 49823     -1']
  ['81' 'TO BT BT BT BT BT BT BT BT BT BT BT AB                    ' '81' '    31      0' '31.0000' '  9297     -1']
  ['92' 'TO BT BT BT BT BT SR SR BT BT BT BT BT BT BT BT SA        ' '92' '    26      0' ' 0.0000' ' 27573     -1']
  ['105' 'TO BT BT BT BT BT SR SR BT BT BT BT BT BT BT SD           ' '105' '    22      0' ' 0.0000' ' 26717     -1']]

 np.c_[siq,_quo,siq,sabo2,sc2,sabo1][azero]  ## in B but not A
 [['18' 'TO BT BT BT BT BT SR SR SR BR BT BT BT BT BT BT SA        ' '18' '     0    197' '197.0000' '    -1  15508']
  ['26' 'TO BT BT BT BT BT SR SR BT BT BT BT BT BT SA              ' '26' '     0    161' '161.0000' '    -1  26558']
  ['77' 'TO BT BT BT BT BT BT BT BT BT BT BT SA                    ' '77' '     0     33' '33.0000' '    -1   9210']]
 PICK=A MODE=2 SEL=0 ~/j/ntds/ntds3.sh

ntds3_V1J010_A16654_3inch_issue.png

MODE=2 PICK=A APID=16654 ~/j/ntds/ntds3.sh ana

FewPMT_tub3_degenerate.png

Murky edge : indicates pmt_solid degenerate with cntr_solid

G4CXTest_FewPMT_one_pmt_circle_inwards_100_A1000_Tub3_delta_1.png

export Tub3inchPMTV3Manager__VIRTUAL_DELTA_MM=1 # avoids issue

PICK=A D=2 APID=1000 FOCUS=0,0,80 ~/opticks/g4cx/tests/G4CXTest.sh ana

ntds3 : insitu input photon bi-simulation comparison A(OK)/B(G4) : After increase tub3 delta 1e-3=>0.1

QCF qcf :  c2sum :   162.4843 c2n :   108.0000 c2per:     1.5045  C2CUT:   30
 c2sum/c2n:c2per(C2CUT)  162.48/108:1.504 (30)
 np.c_[siq,_quo,siq,sabo2,sc2,sabo1][0:25]  ## A-B history frequency chi2 comparison
 [[' 0' 'TO BT BT BT BT SA                                             ' ' 0' ' 37494  37620' ' 0.2114' '     8      1']
  [' 1' 'TO BT BT BT BT SD                                             ' ' 1' ' 30866  30749' ' 0.2222' '     4     13']
  [' 2' 'TO BT BT BT BT BT SA                                          ' ' 2' ' 12382  12416' ' 0.0466' '  9412   8882']
  [' 3' 'TO BT BT BT BT BT SR SA                                       ' ' 3' '  3810   3831' ' 0.0577' ' 11059  11054']
  [' 4' 'TO BT BT BT BT BT SR SR SA                                    ' ' 4' '  1998   1969' ' 0.2120' ' 10899  10889']
  [' 5' 'TO BT BT AB                                                   ' ' 5' '   884    895' ' 0.0680' '    26     20']
  [' 6' 'TO BT BT BT BT BT SR SR SR SA                                 ' ' 6' '   572    604' ' 0.8707' ' 14725  14758']
  [' 7' 'TO BT BT BT BT BR BT BT BT BT BT BT AB                        ' ' 7' '   411    451' ' 1.8561' ' 11875   5071']
  [' 8' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT SA                  ' ' 8' '   337    346' ' 0.1186' '  7444   7444']
  [' 9' 'TO BT BT BT BT AB                                             ' ' 9' '   319    345' ' 1.0181' '   651     14']
  ['10' 'TO BT BT BT BT BR BT BT BT BT BT BT SD                        ' '10' '   314    335' ' 0.6795' '  5262   5252']
  ['11' 'TO BT BT BT BT BR BT BT BT BT BT SA                           ' '11' '   332    312' ' 0.6211' '  1021   1025']
  ['12' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT SD                  ' '12' '   320    289' ' 1.5780' '  8147   8170']
  ['13' 'TO BT BT BT BT BT SR BR SA                                    ' '13' '   309    319' ' 0.1592' ' 33584  33568']
  ['14' 'TO BT BT BT BT BR BT BT BT BT AB                              ' '14' '   279    248' ' 1.8235' '   646   9164']
  ['15' 'TO BT BT BR BT BT BT SA                                       ' '15' '   243    211' ' 2.2555' '     2      2']
  ['16' 'TO BT BT BT BT BT SR SR SR BR SA                              ' '16' '   212    239' ' 1.6164' ' 14749  14761']
  ['17' 'TO BT BT BT BR BT BT BT BT SA                                 ' '17' '   216    214' ' 0.0093' '   206    226']
  ['18' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT BT BT BT BT SD      ' '18' '   190    166' ' 1.6180' ' 16931  11835']
  ['19' 'TO BT BT BT BT BT SR SR SR BR BT BT BT BT BT BT SA            ' '19' '   176    187' ' 0.3333' ' 15529  15388']
  ['20' 'TO BT BT BT BT BR BT BT BT BT BT BT SC BT BT BT BT BT BT SA   ' '20' '   148    171' ' 1.6583' ' 17266  16930']
  ['21' 'TO BT BT BT BT BT SR SR SR BR BR SR SA                        ' '21' '   168    155' ' 0.5232' ' 15414  15512']
  ['22' 'TO BT BT BT BT BR BT BT BT BT BT BT BT BT BT BT BT BT SA      ' '22' '   163    167' ' 0.0485' ' 11832  17198']
  ['23' 'TO BT BT BT BT BR BT BT BT BT BT BT SC BT BT BT BT BT BT SD   ' '23' '   149    159' ' 0.3247' ' 16930  16725']
  ['24' 'TO BT BT BT BT BT SR SR BT BT BT BT BT BT SA                  ' '24' '   143    151' ' 0.2177' ' 26577  26568']]

 np.c_[siq,_quo,siq,sabo2,sc2,sabo1][bzero]  ## in A not in B
 [['55' 'TO BT BT BT SA                                                ' '55' '    42      0' '42.0000' ' 49820     -1']
  ['68' 'TO BT BT BT SD                                                ' '68' '    34      0' '34.0000' ' 49823     -1']]

ntds3_V1J011_A49820_LPMT_virtual_apex_degeneracy.png

PICK=A D=2 APID=49820 ~/j/ntds/ntds3.sh ana

ntds3_V1J011_A49820_LPMT_apex_degenerate_in_red.png

PICK=A D=2 APID=49820 SEL="TO BT BT BT SA,TO BT BT BT SD" ~/j/ntds/ntds3.sh ana