JUNOSW + Opticks : Status and Plan

Opticks Reminder + "list-nodes"
- Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
- Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7+
- Opticks: n-ary CSG Compound "List-Nodes" => Much Smaller CSG trees
JUNOSW+Opticks (2024/06) Geometry Issues
- WaterPoolConstruction "sibling soup" vs HIERARCHY
- WaterPoolConstruction__CONFIG : testing alternative geometry
- FastenerAcrylicConstruction (discontiguous list-node working)
- XJ{fixture/anchor} triangulated(?)
- SJ{Fixture/Receiver/ReceiverFastern/CLSanchor} triangulated(?)
- GuideTube : Torus Triangulated (looking good, precision is adaptable)
2024/06 JUNO+Opticks Summary
- Geometry Fixes
- Status + Plan
Opticks Links, JUNOSW+Opticks DocDB

Simon C Blyth, IHEP, CAS — Simulation Meeting — 2 July 2024

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow

https://bitbucket.org/simoncblyth/opticks

Opticks API : split according to dependency -- Optical photons are GPU "resident", only hits need to be copied to CPU memory

Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7+

Geant4 Geometry Model (JUNO: 400k PV, deep hierarchy)

PV	G4VPhysicalVolume	placed, refs LV
LV	G4LogicalVolume	unplaced, refs SO
SO	G4VSolid,G4BooleanSolid	binary tree of SO "nodes"

Opticks CSGFoundry Geometry Model (index references)

struct	Notes	Geant4 Equivalent
CSGFoundry	vectors of the below, easily serialized + uploaded + used on GPU	None
qat4	4x4 transform refs CSGSolid using "spare" 4th column (becomes IAS)	Transforms ref from PV
CSGSolid	refs sequence of CSGPrim	Grouped Vols + Remainder
CSGPrim	bbox, refs sequence of CSGNode, root of CSG Tree of nodes	root G4VSolid
CSGNode	CSG node parameters (JUNO: ~23k CSGNode)	node G4VSolid

NVIDIA OptiX 7+ Geometry Acceleration Structures (JUNO: 1 IAS + 10 GAS, 2-level hierarchy)

IAS	Instance Acceleration Structures	JUNO: 1 IAS created from vector of ~50k qat4 (JUNO)
GAS	Geometry Acceleration Structures	JUNO: 10 GAS created from 10 CSGSolid (which refs CSGPrim,CSGNode )

JUNO : Geant4 ~400k volumes "factorized" into 1 OptiX IAS referencing ~10 GAS

n-ary CSG Compound "List-Nodes" => Much Smaller CSG trees

Generalizes binary to n-ary CSG trees

using three-level Opticks CSG : tree < node < leaf

  +---------------+----------------------+
  | 2-ary tree    |   n-ary "list-node"  |
  +===============+======================+
  |        U      |      U[A,B,C]        |
  |       /       |                      |
  |      U        |                      |
  |     / \       |                      |
  |    U   A      |                      |
  |   / \         |                      |
  |  C   B        |                      |
  +---------------+----------------------+

list-node can be within binary trees
list-node references sub-nodes by subNum subOffset
Opticks translates G4MultiUnion into list-node

Hints in G4VSolid name : Specify shape more precisely: => better suited intersect alg => less resources => faster
eg: CSG_DISCONTIGUOUS Union: user guarantees no overlaps, eg "union of holes" to be CSG subtracted : => simple, low resource intersect

cxr_min__eye_1,0,0__zoom_1__tmin_0.5__World_solid.jpg

3x3x3 grid of MultiUnion/list-node each with 7x7x7=343 Orb: would not be possible without list-node

JUNOSW+Opticks (2024/06) : Geometry Issues Summary

WaterPoolConstruction + HBeamConstruction : Addition of DeadWater + Tyvek

"sibling soup" model forces complex CSG gymnastics (>60 holes subtracted)
- very deep CSG tree breaks Opticks translation
lower HBeam cross pieces and all lower pillar sections impinge into pool bottom
- may cause Geant4 "stuck" tracks

FIX (MR 458, J24.1.1) under test : simplify geometry using hierarchy approach (WaterPoolConstruction__CONFIG envvar)

FastenerAcrylicConstruction

Deep CSG tree causes Opticks translation issue

FIX (MR 468, J24.1.1) under test : using G4MultiUnion/list-node (FastenerAcrylicConstruction__CONFIG envvar)

XJfixtureConstruction + SJReceiverFasternConstruction

Complicated CSG causing performance issue => trying triangulated or list-node
- FIX requires Opticks geometry generalization for triangulation

Calib_GuideTube_Construction

39*2*2 = 156 G4Torus : double precision intersect on GPU : very expensive => triangulated looking OK
FIX requires no JUNOSW change, needs Opticks geometry generalization

sDeadWater

"sDeadWater/sTyvekFilm" issues at WaterPool base

WaterPoolConstruction "Sibling Soup" vs HIERARCHY


       +---Air-----------------------+
       |                             |
       | +--DeadWater--------------+ |
       | | +---Tyvek-------------+ | |
       | | | +----vetoWater----+ | | |
       | | | |                 | | | |
       | | | | +-+         +-+ | | | |
       | | | | |Steel      | | | | | |
       | | | | +-+         +-+ | | | |
       | | | +-----------------+ | | |
       | | +---------------------+ | |
       | +-------------------------+ |
       +-----------------------------+

Geant4 "Russian doll" hierarchy

daughter material overrides mother
=> simple cylinders G4Polycone (rmin 0.)

BUT: pillars need to be within vetoWater

"Sibling Soup" : (DeadWater, Tyvek, Air, Steel) in vetoWater

sDeadWater,sTyvekFilm : 6 plane G4Polycone with 31 holes

  +---------------------+   +----------------+---+    1 plane
  |                     |   |                    |
  +   +-----------------+   +----------------+   +    2 planes
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  +   +                   +                  +   +
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  |   |                                      |   |
  +   +--+  +---+  +------+------------------+   +    2 planes
  |      |  |   |  |                             |
  +------+  +---+  +------+----------------------+    1 plane

complex CSG : to avoid sibling overlaps

ISSUES : with current "Sibling soup" geometry

many (~120) pool bottom impingments
complex CSG breaks Opticks translation

WaterPoolConstruction__CONFIG : testing alternative geometry

branch: blyth-WaterPoolConstruction__CONFIG-to-fix-pillar-impingement-and-opticks-translation

https://code.ihep.ac.cn/JUNO/offline/junosw/-/merge_requests/458 MERGED: 2024-06-17 (J24.1.1)

CONFIG	Notes on WaterPoolConstruction HBeamConstruction
0:ASIS	sibling soup => very complicated CSG
1:ZERO_HOLES	skip 60 one-by-one hole subtractions from DeadWater Tyvek
2:MULTIUNION_HOLES	collect holes into G4MultiUnion(listnode), subtract once
3:HIERARCHY	radical simplification, minor geometry change : skip bottom angled cross piece raise base of pillars within vetoWater

Set CONFIG envvar to change geometry, default is no change "sibling soup":

unset WaterPoolConstruction__CONFIG
export WaterPoolConstruction__CONFIG=0 # ASIS : same as unset

export WaterPoolConstruction__CONFIG=3 # HIERARCHY

cxr_min__eye_0,1,0__zoom_1__tmin_0.1__GZ1.B06_07_FlangeI_Web_FlangeII:0:-1.jpg

Using G4MultiUnion/listnode for 62 subtracted holes works, BUT...

cxr_min__eye_0,1,0__zoom_1__tmin_0.1__GZ1.B06_07_FlangeI_Web_FlangeII:0:-100000.jpg

Impingment at bottom of WaterPool apparent, from angled cross-piece

cxr_min__eye_0,1,0__zoom_1__tmin_0.1__sDeadWater:0:-1.jpg

Also impingement from vertical and tilted pillars

cxr_min__eye_-1,0,0__zoom_1__tmin_0.1__GZ1.A06_07_FlangeI_Web_FlangeII:0:-100001.jpg

WaterPoolConstruction__CONFIG=HIERARCHY (much simpler)

* skip bottom angled cross piece

* raise pillar base 126mm (keep within vetoWater)

FastenerAcrylicConstruction

Problematic deep CSG tree without list-node

+------------------------------------------+
|                                          |
|                                          |
|                           U              |
|                          / \             |
|                         /   \            |
|                        /     S           |
|                       U     / \          |
|                      / \   I   J         |
|                     U   H                |
|                    / \                   |
|                   U   G                  |
|                  / \                     |
|                 U   F                    |
|                / \                       |
|               U   E                      |
|              / \                         |
|             U   D                        |
|            / \                           |
|           U   C                          |
|          / \                             |
|         A   B                            |
|                                          |
+------------------------------------------+

U : Union
S : Subtraction
A-J : Tubs (cylinder) primitive

cxr_min__eye_0,3,-1.5__zoom_1__tmin_0.5__uni1:0:0.jpg

FastenerAcrylicConstruction : hidden within support base

cxr_min__eye_0,3,-1.5__zoom_1__tmin_0.5__uni1:0:000000.jpg

FastenerAcrylicConstruction : 8 cylinders: modelled with G4MultiUnion/list-node

20240626_171555_00000_wireframe.jpg

triangulated wireframe render

XJfixture/anchor triangulated(?)

(0):xjfc:XJfixture

(1):xjac:XJanchor

Construction	#
XJfixture	56
XJanchor	56

XJfixture

messy union of tubs, box
- many coincident faces => spurious intersects

XJanchor

union of tubs, cons
big sphere subtraction
- poor precision intersect => spurious

TODO: compare validation + performance:

ASIS
with list-node
triangulated

cxr_min__eye_1,0,0__zoom_1__tmin_0.5__solidXJfixture:0:-1.jpg

XJfixture + XJanchor : "hold" GuideTube: (trying triangulated)

SJ triangulated render

(0):sjcl:SJCLSanchor [only 2, Acrylic]

(1):sjfx:SJFixture [36, Acrylic]

(2):sjrc:SJReceiver [only 8, Acrylic]

(3):sjrf:SJReceiverFastern [only 8, PE_PA]

GuideTube : Torus Triangulated

GuideTube (39*2*2 = 156 G4Torus): split in phi segments, radius breaks

Intersect with torus expensive on GPU

requires double precision to solve quartic
even with double precision analytic solution imprecise
numerical approach favored => triangulation

Triangulation using G4Polyhedron

G4Poly..::SetNumberOfRotationSteps

	NumberOfRotationSteps
HepPolyhedron Default	24
Top Right	48
Bottom Right	480

Adjustable: precision of intersect, number of triangles

GPUs evolved for triangles => fast even with many

cxr_min__eye_1,0,0__zoom_1__tmin_0.5__sSurftube_0V1_0:0:-1.jpg

Testing triangulated GuideTube + XJ + SJ solids: (no choice for GuideTube torii, maybe some XJ + SJ can use "list-node")

solidXJfixture:27:-1

TRIMESH=1 MOI=solidXJfixture:27:-1 EYE=0,0.1,-20 ~/o/cxr_min.sh

2024/06 JUNO+Opticks Geom Fix

WaterPool Overlaps+Opticks FIXED with simpler approach

FastenerAcrylic works with G4MultiUnion/"list-node"

Testing triangulated GuideTube + XJ + SJ solids

Re-implemented WaterPool with hierarchy approach
- avoid : ~120 overlaps, complex CSG
- enables translation to Opticks/GPU
FastenerAcrylic (590) working using "list-node"
- discontiguous => simple + fast
GuideTube (39*2*2 = 156 G4Torus)
- torus intersect v.expensive on GPU => triangulate
- adjust precision with NumberOfRotationSteps
XJ + SJ solids (less numerous)
- many concident surfaces, try triangulate

2024/06 JUNOSW+Opticks : Status + Plan

Option	Approach to equalize CPU/GPU geometry		Status
--no-guide_tube	triangulated with NumberOfRotationSteps precision control		GOOD
--debug-disable-fa	FastenerAcrylic using G4MultiUnion/"list-node" works		GOOD
--debug-disable-xj	XJfixture XJanchor	Trying triangulated (if too much validation deviation use "listnode")	??
--debug-disable-sj	SJCLSanchor SJFixture SJReceiver SJReceiverFastern		??

Immediate Steps : Geometry + Performance + Validation

generalize Opticks geometry for fine-grained triangulation control
geometry performance scan + validation check
Make full geometry JUNO+Opticks release (aim ~September)
automate performance + validation monitoring
convince offline group to change defaults, adopting geometry fixes+simplifications

Optimization : measure performance as:

slim Release kernel
change instancing criteria, reduce remainder volumes by instancing more
get support stick volumes instanced together

Production Prep : auto event splitting, depending on VRAM

Opticks Links, JUNOSW+Opticks DocDB

https://bitbucket.org/simoncblyth/opticks	code repository (day-to-day)
https://github.com/simoncblyth/opticks	code repository (~month-to-month), releases
https://simoncblyth.bitbucket.io https://simoncblyth.github.io https://juno.ihep.ac.cn/~blyth/	publications, presentations, videos
https://groups.io/g/opticks	forum/mailing list archive
email:opticks+subscribe@groups.io	subscribe to mailing list

JUNOSW+Opticks

DocDB-10968	2023/12/19	Using first JUNOSW+Opticks Pre-Release at IHEP GPU cluster
DocDB-10929	2023/12/11	JUNOSW + Opticks : Profiling and Status