JUNOSW + Opticks : Status and Plan
JUNOSW + Opticks : Status and Plan
- Opticks Reminder
- Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
- Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7+
- JUNOSW+Opticks (2024/06)
- Geometry Issues Summary
- Opticks: n-ary CSG Compound "List-Nodes" => Much Smaller CSG trees
- WaterPoolConstruction
- WaterPoolConstruction "sibling soup" vs HIERARCHY
- WaterPoolConstruction__CONFIG : testing alternative geometry
- FastenerAcrylicConstruction
- XJ triangulated
- SJ triangulated
- NEXT STEPS/PLAN
- Opticks Links, JUNOSW+Opticks DocDB
Simon C Blyth, IHEP, CAS — Simulation Meeting — 2 July 2024
Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
| https://bitbucket.org/simoncblyth/opticks |
Opticks API : split according to dependency -- Optical photons are GPU "resident", only hits need to be copied to CPU memory
Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7+
CSGFoundry Model
- array-based -> simple serialization + upload
- entire geometry in 4 GPU allocations
- factorized using subtree digests
Geant4 Geometry Model (JUNO: 400k PV, deep hierarchy)
| PV | G4VPhysicalVolume | placed, refs LV |
| LV | G4LogicalVolume | unplaced, refs SO |
| SO | G4VSolid,G4BooleanSolid | binary tree of SO "nodes" |
Opticks CSGFoundry Geometry Model (index references)
| struct | Notes | Geant4 Equivalent |
|---|---|---|
| CSGFoundry | vectors of the below, easily serialized + uploaded + used on GPU | None |
| qat4 | 4x4 transform refs CSGSolid using "spare" 4th column (becomes IAS) | Transforms ref from PV |
| CSGSolid | refs sequence of CSGPrim | Grouped Vols + Remainder |
| CSGPrim | bbox, refs sequence of CSGNode, root of CSG Tree of nodes | root G4VSolid |
| CSGNode | CSG node parameters (JUNO: ~23k CSGNode) | node G4VSolid |
NVIDIA OptiX 7+ Geometry Acceleration Structures (JUNO: 1 IAS + 10 GAS, 2-level hierarchy)
| IAS | Instance Acceleration Structures | JUNO: 1 IAS created from vector of ~50k qat4 (JUNO) |
| GAS | Geometry Acceleration Structures | JUNO: 10 GAS created from 10 CSGSolid (which refs CSGPrim,CSGNode ) |
JUNO : Geant4 ~400k volumes "factorized" into 1 OptiX IAS referencing ~10 GAS
JUNOSW+Opticks (2024/06) : Geometry Issues Summary
WaterPoolConstruction + HBeamConstruction : Addition of DeadWater + Tyvek
- "sibling soup" model forces complex CSG gymnastics (>60 holes subtracted)
- very deep CSG tree breaks Opticks translation
- lower HBeam cross pieces and all lower pillar sections impinge into pool bottom
- may cause Geant4 "stuck" tracks
FIX (MR 458) under test : simplify geometry using hierarchy approach (WaterPoolConstruction__CONFIG envvar)
FastenerAcrylicConstruction
- Deep CSG tree causes Opticks translation issue
FIX (MR 468) under test : using G4MultiUnion/list-node (FastenerAcrylicConstruction__CONFIG envvar)
n-ary CSG Compound "List-Nodes" => Much Smaller CSG trees
Complex CSG => Tree Overheads
Generalizes binary to n-ary CSG trees
- using three-level Opticks CSG : tree < node < leaf
+---------------+----------------------+ | 2-ary tree | n-ary "list-node" | +===============+======================+ | U | U[A,B,C] | | / | | | U | | | / \ | | | U A | | | / \ | | | C B | | +---------------+----------------------+
- list-node can be within binary trees
- list-node references sub-nodes by subNum subOffset
- Opticks translates G4MultiUnion into list-node
- Hints in G4VSolid name : Specify shape more precisely
- => better suited intersect alg => less resources => faster
- eg: CSG_DISCONTIGUOUS Union
- user guarantees no overlaps, eg "union of holes" to be CSG subtracted : => simple, low resource intersect
cxr_min__eye_1,0,0__zoom_1__tmin_0.5__World_solid.jpg
- 3x3x3 grid of MultiUnion/list-node each with 7x7x7=343 Orb
- would not be possible without list-node
WaterPoolConstruction "Sibling Soup" vs HIERARCHY
HIERARCHY : Radically Simple
+---Air-----------------------+
| |
| +--DeadWater--------------+ |
| | +---Tyvek-------------+ | |
| | | +----vetoWater----+ | | |
| | | | | | | |
| | | | +-+ +-+ | | | |
| | | | |Steel | | | | | |
| | | | +-+ +-+ | | | |
| | | +-----------------+ | | |
| | +---------------------+ | |
| +-------------------------+ |
+-----------------------------+
Geant4 "Russian doll" hierarchy
- daughter material overrides mother
- => simple cylinders G4Polycone (rmin 0.)
BUT: pillars need to be within vetoWater
"Sibling Soup" : (DeadWater, Tyvek, Air, Steel) in vetoWater
sDeadWater,sTyvekFilm : 6 plane G4Polycone with 31 holes +---------------------+ +----------------+---+ 1 plane | | | | + +-----------------+ +----------------+ + 2 planes | | | | | | | | | | | | | | | | | | | | | | | | | | | | + + + + + | | | | | | | | | | | | | | | | | | | | | | | | | | | | + +--+ +---+ +------+------------------+ + 2 planes | | | | | | +------+ +---+ +------+----------------------+ 1 plane
- complex CSG : to avoid sibling overlaps
ISSUES : with current "Sibling soup" geometry
- many (~120) pool bottom impingments
- complex CSG breaks Opticks translation
WaterPoolConstruction__CONFIG : testing alternative geometry
blyth-WaterPoolConstruction__CONFIG-to-fix-pillar-impingement-and-opticks-translation
| CONFIG | Notes on WaterPoolConstruction HBeamConstruction |
|---|---|
| 0:ASIS | sibling soup => very complicated CSG |
| 1:ZERO_HOLES | skip 60 one-by-one hole subtractions from DeadWater Tyvek |
| 2:MULTIUNION_HOLES | collect holes into G4MultiUnion(listnode), subtract once |
| 3:HIERARCHY | radical simplification, minor geometry change :
|
https://code.ihep.ac.cn/JUNO/offline/junosw/-/merge_requests/458 MERGED: 2024-06-17
Set CONFIG envvar to change geometry, default is no change "sibling soup":
unset WaterPoolConstruction__CONFIG export WaterPoolConstruction__CONFIG=0 # ASIS : same as unset export WaterPoolConstruction__CONFIG=3 # HIERARCHY
cxr_min__eye_0,1,0__zoom_1__tmin_0.1__GZ1.B06_07_FlangeI_Web_FlangeII:0:-1.jpg
Using G4MultiUnion/listnode for 62 subtracted holes works, BUT...
cxr_min__eye_0,1,0__zoom_1__tmin_0.1__GZ1.B06_07_FlangeI_Web_FlangeII:0:-100000.jpg
Impingment at bottom of WaterPool apparent, from angled cross-piece
cxr_min__eye_0,1,0__zoom_1__tmin_0.1__sDeadWater:0:-1.jpg
Also impingement from vertical and tilted pillars
cxr_min__eye_-1,0,0__zoom_1__tmin_0.1__GZ1.A06_07_FlangeI_Web_FlangeII:0:-100001.jpg
WaterPoolConstruction__CONFIG=HIERARCHY (much simpler)
* skip bottom angled cross piece
* raise pillar base 126mm (keep within vetoWater)
FastenerAcrylicConstruction
With list-node : shrink CSG tree
+------------------------------+ | U | | / \ | | / \ | | D U[A,B,C,D,E,F,G,H] | | / \ | | I J | +------------------------------+
Problematic deep CSG tree without list-node
+------------------------------------------+ | | | | | U | | / \ | | / \ | | / D | | U / \ | | / \ I J | | U H | | / \ | | U G | | / \ | | U F | | / \ | | U E | | / \ | | U D | | / \ | | U C | | / \ | | A B | | | +------------------------------------------+ U : Union A-J : Tubs (cylinder) primitive
cxr_min__eye_0,3,-1.5__zoom_1__tmin_0.5__uni1:0:0.jpg
FastenerAcrylicConstruction : hidden within support base
cxr_min__eye_0,3,-1.5__zoom_1__tmin_0.5__uni1:0:000000.jpg
- FastenerAcrylicConstruction : 8 cylinders
- modelled with G4MultiUnion/list-node
XJ triangulated
(0):xjfc:XJfixture
(1):xjac:XJanchor
| Construction | # |
|---|---|
| XJfixture | 56 |
| XJanchor | 56 |
XJfixture
- messy union of tubs, box
- many coincident faces => spurious intersects
XJanchor
- union of tubs, cons
- big sphere subtraction
- poor precision intersect => spurious
TODO: compare validation + performance:
- ASIS
- with list-node
- triangulated
cxr_min__eye_1,0,0__zoom_1__tmin_0.5__solidXJfixture:0:-1.jpg
- XJfixture + XJanchor : "hold" GuideTube
- (trying triangulated)
SJ triangulated
Counts in Geometry
| Construction | # |
|---|---|
| SJCLSanchor | 2 |
| SJfixture | 36 |
| SJReceiver | 8 |
| SJReceiverFastern | 8 |
SJCLSanchor
SJfixture
SJ triangulated render
(0):sjcl:SJCLSanchor
(1):sjfx:SJfixture
(2):sjrc:SJReceiver
(3):sjrf:SJReceiverFastern
cxr_min__eye_1,0,0__zoom_1__tmin_0.5__sSurftube_0V1_0:0:-1.jpg
- Testing triangulated GuideTube + XJ + SJ solids
- (no choice for GuideTube torii, maybe some XJ + SJ can use "list-node")
Remaining CPU/GPU Geometry Differences
Virtual surface shifts used to avoid degeneracy, together with defaults (shifts avoid chi2 discrepancies from degenerate surfaces):
export Tub3inchPMTV3Manager__VIRTUAL_DELTA_MM=0.10 ## 1.e-3 export HamamatsuMaskManager__MAGIC_virtual_thickness_MM=0.10 ## 0.05 export NNVTMaskManager__MAGIC_virtual_thickness_MM=0.10 ## 0.05
- TODO: check virtual shifts have no performance effects or overlap issues
- convince offline group to enlarge defaults : avoiding these Opticks only shifts
- TODO: add optional triangulated geometry handling : use for guide tube
- torus quartic analytic solution painful : expect triangulation approximation more robust+precise
- TODO: test "listnode" solution for handling deep CSG trees (complex solid shape)
Completing these three : will match GPU and CPU geometry
Opticks Links, JUNOSW+Opticks DocDB
| https://bitbucket.org/simoncblyth/opticks | code repository (day-to-day) |
| https://github.com/simoncblyth/opticks | code repository (~month-to-month), releases |
| https://simoncblyth.bitbucket.io https://simoncblyth.github.io https://juno.ihep.ac.cn/~blyth/ | publications, presentations, videos |
| https://groups.io/g/opticks | forum/mailing list archive |
| email:opticks+subscribe@groups.io | subscribe to mailing list |
JUNOSW+Opticks
| DocDB-10968 | 2023/12/19 | Using first JUNOSW+Opticks Pre-Release at IHEP GPU cluster |
| DocDB-10929 | 2023/12/11 | JUNOSW + Opticks : Profiling and Status |
2024/06 JUNO+Opticks Geom Fix
WaterPool Overlaps+Opticks FIXED with simpler approach
FastenerAcrylic works with G4MultiUnion/"list-node"
Testing triangulated GuideTube + XJ + SJ solids
- Re-implemented WaterPool with hierarchy approach
- avoid : ~120 overlaps, complex CSG
- enables translation to Opticks/GPU
- FastenerAcrylic (590) working using "list-node"
- discontiguous => simple + fast
- GuideTube (39*2*2 = 156 G4Torus)
- torus intersect v.expensive on GPU => triangulate
- adjust precision with NumberOfRotationSteps
- XJ + SJ solids (less numerous)
- many concident surfaces, try triangulate
2024/06 JUNOSW+Opticks : Status + Plan
| Option | Approach to equalize CPU/GPU geometry | Status | ||
|---|---|---|---|---|
| --no-guide_tube | triangulated with NumberOfRotationSteps precision control | GOOD | ||
| --debug-disable-fa | FastenerAcrylic using G4MultiUnion/"list-node" works | GOOD | ||
| --debug-disable-xj | XJfixture XJanchor | Trying triangulated (if too much validation deviation use "listnode") | ?? | |
| --debug-disable-sj | SJCLSanchor SJFixture SJReceiver SJReceiverFastern | |||
Immediate Steps : Geometry + Performance + Validation
- generalize Opticks geometry for fine-grained triangulation control
- geometry performance scan + validation check
- Make full geometry JUNO+Opticks release
- automate performance + validation monitoring
- convince offline group to change defaults, adopting geometry fixes+simplifications
Optimization : measure performance as:
- slim Release kernel
- change instancing criteria, reduce remainder volumes by instancing more
- get support stick volumes instanced together
Production Prep : auto event splitting, depending on VRAM