orientation : Opticks Codebase Orientation for Developers

Opticks Objectives

  1. replace Geant4 optical photon simulation with an equivalent GPU implementation based on NVIDIA OptiX
  2. provide automated geometry translation without approximation, Geant4 -> Opticks (GGeo) -> NVIDIA OptiX,
  3. provide a workflow that integrates the Opticks/OptiX simulation of optical photons with the Geant4 simulation of all other particles

Page Overview

The focus of this description is on the Opticks usage of NVIDIA OptiX however necessary summary contextual info for developers unfamiliar with Opticks is also collected here together with references to more extensive documentation.


https://bitbucket.org/simoncblyth/opticks very latest code repository, unstable, breakage common
https://github.com/simoncblyth/opticks “releases” weeks/months behind, more stable
https://simoncblyth.bitbucket.io presentations and videos
https://groups.io/g/opticks forum/mailing list archive
email:opticks+subscribe@groups.io subscribe to mailing list

Geant4-Opticks-NVIDIA_OptiX workflow

Geant4-Opticks-OptiX workflow

G4Opticks : Geant4-Opticks interface class

Despite the classname G4Opticks is an Opticks class from g4ok sub-project that provides a minimal interface to using embedded Opticks. It is intended to be integrated with the Geant4 based simulation framework of an experiment.

Physics Background

Scintillation and Cerenkov are the two physical processes which are the principal sources of light relevant to neutrino and dark matter experiments.

After generation the photons propagate through the detector geometry being scattered, absorbed, reemitted (in the bulk) and reflected, refracted, detected or absorbed (on surfaces encountered).

The objective of simulation is to provide estimates of times and numbers of photons that reach detectors such as Photomultipler tubes (PMTs) by creation of large samples with various input particles types and parameters.

Simulation is the best way to understand complex detectors and as a result form a better understanding of the physics of interest such as neutrinos coming from nuclear reactors or from the sun or from earths mantle or from distant galaxies.

Geometry translation : Geant4->Opticks(GGeo)->OptiX (green arrows)

The green arrows in the above workflow diagram represent the translation of geometry information that happens at initialization. As this translation can take minutes for large geometries the Opticks(GGeo) geometry model is persisted to binary .npy files which can act as a geocache.

Geometry translation is steered by G4Opticks::translateGeometry with X4PhysicalVolume taking the leading role.

The translation entails the serialization of Geant4 C++ geometry objects for materials, surfaces and solid shapes into arrays and the upload of those into GPU buffers and textures.

Structural geometry volumes are translated into the NVIDIA OptiX geometry model using a very small and flat heirarchy by effectively “factorizing” the structural geometry in a way that exploits the large degree of repetition present in typical detector geometries such as JUNO, with many thousands of photomultiplier tubes of various types. This “factorization” is done with the GInstancer as detailed below.

Opticks Usage of NVIDIA OptiX

Direct use of OptiX is primarily in the optixrap subproject OptiXRap Orientation : translates GGeo->OptiX however most of the rest of Opticks is involved with the conversion of the Geant4 geometry into a form that can become an OptiX geometry suitable for optical photon simulation.

Familiarity with Geant4

Some familiarity with the Geant4 geometry model is required to understand Opticks as the bulk of Opticks code is concerned with the automated translation of Geant4 geometries into Opticks(GGeo) geometries and subsequently OptiX geometries.

Opticks provides some simple bash functions to viewing Geant4 source, eg:

g4-   # precursor bash function
g4-cls G4VSolid
  • all Geant4 class names are prefixed with “G4”
  • Opticks/ggeo class names mostly start with “G”
  • Opticks/npy class names mostly start with “N”
  • G4Opticks has an exceptional name, it is an Opticks/g4ok class

Geant4 Links

Gensteps (blue arrows) and hits (red arrows)

Gensteps (blue arrows in the above workflow diagram) are small arrays of shape (num_gensteps,6,4) which act to carry the parameters of the Cerenkov and Scintillation photon generation from their origin in the modified Geant4 process classes to the CUDA ports of the process generation loops running within the NVIDIA OptiX ray generation program.

The parameters of each genstep includes the number of photons to generate and the line segment along which to generate them together with other parameters needed by the port of the G4Cerenkov and G4Scinillation generation loops.

Gensteps are typically several orders of magnitude smaller than the photons that they yield. Photon generation on GPU has double benefits:

  1. no copying of lots of photons from CPU to GPU
  2. no CPU memory allocation for the majority of the photons, only the small fraction of photons that are detected, known as hits, need to have CPU memory allocation (see red arrows in the above workflow diagram)

Gensteps are the inputs to the optical simulation which yield hits as the output.

Geant4 classes which are partially ported to CUDA/OptiX

  • G4Cerenkov (only generation loop)
  • G4Scintillation (only generation loop)
  • G4OpRayleigh (bulk scattering)
  • G4OpAbsorption (bulk absorption)
  • G4OpBoundaryProcess (only a few surface types)

To quickly view the sources of any Geant4 classes use the opticks bash function g4-cls:

g4-;g4-cls G4Cerenkov

Primary Opticks Packages relevant to geometry

The below Opticks sub-projects are the most relevant ones for understanding the geometry translation. The linked sub-project orientation pages highlight a few of the more important classes. The quoted names are common abbeviations used for the sub-projects.

extg4 “x4”
ExtG4 Orientation : Translates Geant4->GGeo only
ggeo “ggeo”
GGeo Orientation : Geometry Modelling and Persisting
optixrap “oxrap”
OptiXRap Orientation : translates GGeo->OptiX
npy “npy”
NPY Orientation : array creation, updating and persisting
g4ok “g4ok”
G4OK Orientation : Top Level Interface to Embedded Opticks

Descriptions of all ~20 packages : Opticks Packages Overview

Highlighted Geometry classes from Geant4 and Opticks

Geometry classes can be split into three categories:

  1. material and surface properties
  2. solid shapes
  3. volume structure

The below sections list the classes from Geant4 and Opticks that you need to be familiar with in order to understand how Opticks translates the Geant4 geometry into an NVIDIA OptiX geomety suitable for optical photon simulation.

Material and Surface property classes

Geant4 material/surface property classes

holds properties such as RINDEX (refractive index), ABSLENGTH (absorption length), RAYLEIGH (scattering length) as a function of energy
name and properties table
surface properties associated with the interface between two placed volumes (PV)
surface properties associated with all placements of an unplaced logical volume (LV)

Opticks/ggeo material/surface property classes

Material and surface properties from Geant4 are interpolated onto a common wavelength domain and stored within instances of the below ggeo classes

GMaterial GSurface
GPropertyMap subclasses
GMaterialLib GSurfaceLib
vectors of GMaterial and GSurface with ability to serialize into NPY arrays
holds vector of int4 where the integers are indices pointing at surfaces and materials inner-material/inner-surface/outer-surface/outer-material

Vital role of Boundary Indices and Boundary Texture

This GGeo/GBndLib boundary lib is converted by oxrap/OBndLib into the GPU boundary texture interleaving properties from the GMaterialLib and GSurfaceLib into the boundary texture.

During initial traversal of the Geant4 volume tree a boundary index is assigned to every volume corresponding to unique combinations of the four indices (inner-material,inner-surface,outer-surface,outer-material).

This boundary index together with other identity information from GVolume::getIdentity is available GPU side in the identity buffer.

The boundary index together with the boundary texture allows the current and next material properties and the relevant surface properties to be obtained via wavlength interpolated lookups on the boundary texture.

Geometrical shape classes

abstract base class for solids such as G4Sphere, G4Cons (cone), … Translated into NCSG containing trees of nnode (npy)
constituent of CSG trees held in NCSG
holder of NCSG nnode trees, with concatenation capability
despite the name this encompasses both triangulated mesh and analytic CSG geometry of distinct solid shapes
merged mesh containing merges of both triangulated and analytic geometry representations from multiple GMesh.
holds the GMergedMesh and handles persisting

Volume Structure classes in Geant4 and Opticks/GGeo

G4LogicalVolume “LV”
unplaced volume having a solid and material
G4VPhysicalVolume “PV”
placed volume positioning the G4LogicalVolume within a hierarchy
converted from Geant4 physical+logical volume, has GMesh and transform constituents

as GMergedMesh holds identity and transform arrays across the entire geometry it straddles both shape and structure geometry categories

GMergedMesh are Translated into optix::Group OR optix::GeometryGroup underneath a top level m_top optix::Group by optixrap/OGeo.

top level geometry object holding instances of GNodeLib, GMeshLib, GBndLib, GMaterialLib, GSurfaceLib, …
converts the GGeo/GGeoLib/GMergedMesh into shallow optix::Group optix::GeometryGroup tree. The approach taken was chosen because it allows instances to have variables assigned as allowing instance indices to be associated and thus providing identity information for all intersects. Details in OGeo

Critical Role of ggeo/GInstancer : GVolume tree -> GMergedMesh

The Geant4 to GGeo conversion first recreates the full volume hierarchy in a tree of GVolume. As detector geometries generally have large numbers of identical assemblies of multiple volumes it is important to fully exploit instancing to allow the geometry to fit into available memory.

For example the JUNO photomultipler tubes of various types are each modelled in geant4 with small tree of less than 10 volumes each. These assemblies are then repeated many thousands of times forming the full geometry. Other pieces of geometry such as very large acrylic spheres and support structures are not sufficiently repeated to warrant instancing.

The ggeo/GInstancer automatically identifies repeated assemblies of volumes using a so called progeny digest for every node of the geometry that incorporates the shapes of the children of a node and their relative transforms. Looking for repetitions of the progeny digest and disallowing repeats that are contained within other repeats allow all nodes of the geometry to be assigned with a repeat index (ridx). Remainder volumes which do not pass instancing criteria such as the number of repeats are assigned repeat index zero.

The JUNO geometry contains only about 10 distinct repeated assemblies of volumes, including 4 or 5 different types of photomultipler tubes, various support structures as well as the remainder miscellaneous non-repeated volumes. Traversals allow the global transforms of each of these repeated assemblies to be collected into arrays. The remainder volumes of course only have one transform : the identity matrix.

The subtree of GVolumes of the first occurrence of each repeated assembly are combined together into GMergedMesh instances. Thus the full JUNO geometry “factorizes” into about 10 GMergedMesh instances with each having arrays of up to 30,000 4x4 transforms.

GMergedMesh -> optix::Group OR optix::GeometryGroup

The details of the OptiX geometry structure are documented in

With simulation it is necessary to know the identity of the instance for every geometry intersect as different instances can have different efficiencies and as it is expensive to reconstruct identity just from position.

The need to assign an index to the instances is the reason behind the choice of NVIDIA OptiX geometry structure.

OptiX raygen function for simulation



Steers photon generation and propagation from input Scintillation/Cerenkov/Torch gensteps


propagate.h : propagate_to_boundary / propagate_at_boundary_geant4_style

Included into generate.cu, provides a CUDA port of Geant4 optical physics.

OptiX closest hit functions


<no title>

Used for photon simulation, populating PerRayData_propagate.h



Used for creation of ray trace images of geometry, populating PerRayData_radiance.h

OptiX ray-geometry intersect and bounds functions


intersect_analytic.cu : General Analytic CSG intersection

CUDA/OptiX bounds and intersect functions for general CSG shapes created into OptiX RTprogram by:

optix::Geometry OGeo::makeAnalyticGeometry(GMergedMesh* mm)

csg_intersect_boolean.h : General CSG intersection using evaluative_csg approach

Evaluative CSG : iterative implementation of recursive algorithm


csg_intersect_part.h : csg_bounds_prim / csg_intersect_part

Large switch statements using the bounds and intersect functions from cu/csg_intersect_primitive.h


csg_intersect_primitive.h : Intersections with ~10 different primitives

Bounds and intersect functions for of order 10 primitive shapes:

csg_bounds_convexpolyhedron, csg_intersect_convexpolyhedron
csg_bounds_cone,             csg_intersect_cone


PACK4 UNPACK4_0/1/2/3 macros


CSG Algorithm Prototyping

CSG algorithm development notes, including “slavish” python of the CSG implementation

Examples Directly Using NVIDIA OptiX

The below sections list examples using OptiX, named after directory names. Many of the examples are standalone in nature, not depending on an Opticks install. The steps to build (and sometimes run) are often simply:

cd ~/opticks/examples/UseOptiX

cd ~/opticks/examples/UseOptiX7GeometryInstanced

OptiX pre7 usage examples

really minimal usage of OptiX C API, checking creation of context and buffer, no kernel launching
OptiX C API creates raygen program and launches it, just dumping launch index
OptiX C++ API variant of the above : provides a command line interface to quickly run simple OptiX code (no buffers in context).
OptiX C API creates raygen program that just writes constant values to a buffer
OptiX C++ API : creates in and out buffers from NPY arrays and launches a program that simply copies from in to out. Provides a command line interface to quickly run variants of the buffer accessing GPU code.
Minimally demonstrate OptiX geometry without using OXRAP, performs a “standalone” raytrace of a box with normal shader coloring.

Minimally demonstrate the use of optix::GeometryTriangles introduced in OptiX 6.0.0. Raytraces an octahedron writing a PPM file. Based on NPY and SYSRAP for buffer and PPM handling. No OXRAP.

Use the OptiXRap.OContext to reimplement UseOptiXBufferPP in a higher level style, hoping to approach close enough to UseOptiXRap for the problem to manifest. But it hasnt.

Uses Opticks higher level OptiXRap API to test changing the sizes of buffers.

Issue with OptiX 6.0.0 : the buffer manipulations seem to work but the rtPrintf output does not appear unless the buffer writing is commented out.

Huh, now rtPrintf seems to be working without any clear fix. Now not working. Now working again, immediately after an oxrap–

Perhaps a problem of host code being updated and PTX not, because the PTX is from oxrap ?

Can change the progname via envvar:


Standalone-ish OptiX 5 or 6 Examples

C API 3D texture creation, with pullback test into out_buffer
Switch from 3D to layered 2D texture, exfill attempt to fill with MapEx failed
Convert to use OptiX 6 C++ API
Start encapsulation into Make2DLayeredTexture
Use ImageNPY::LoadPPM to load images into textures First try at 2d layered tex failed, so reverted to 2d textures.

Ray-traced theta-phi texture mapping onto a sphere, when using an Earth texture this provides Earth view PPM images centered around any latitude-longitude position. This example was used to develop the watertight OptiX OCtx wrapper (C opaque pointer style) which does not leak any optix types into its interface.

Intersects are highly instrumented with the position of each interesect recorded into a pos buffer.


start from UseOptiXGeometryInstancedStandalone, plan:

  1. DONE: Opticks packages to reduce the amount of code
  2. DONE: adopt OCtx watertight wrapper, adding whats needed for instancing
  3. DONE: add optional switch from box to sphere
  4. DONE: generate PPM of thousands of textured Earths

jumble of thousands of spheres gradient shaded with red/green/blue border/midline/quadline


start from UseOptiXGeometryInstanced, using just OCtx

/tmp/octx.sh : normal shaded assembly of boxes and spheres /tmp/octx.sh global : global shaded assembly of boxes and spheres /tmp/octx.sh textured,tex1 : textured assembly of boxes and spheres, using tex1 green midline

/tmp/octx.sh single : normal shaded single box and sphere /tmp/octx.sh single,textured,tex1 : single box and sphere

/tmp/octx.sh textest,tex0 : vertical gradient with red border /tmp/octx.sh textest,tex1 : vertical gradient with green midlines /tmp/octx.sh textest,tex2 : vertical gradient with blue quadlines


on Linux/OptiX 6.5 the spheres are appearing as big boxes but there is no problem with the sphere implementation when used not in an assembly. Perhaps problem with the transforms/scaling/bbox ?


Returning to this issue after implementing IntersectSDF to automatically test for such problems find that the problem is no longer happening.

start from UseOptiXGeometry to investigate why getting problem with instanced spheres in OptiX 6.5 Creates PPM of a single normal-shaded sphere or box picked via argument sphere.cu or box.cu
creates a jumble of thousands of randomly oriented boxes, colorfully normal-shaded

Experimental pre7 and 7 machinery

checking FindOpticksOptiX.cmake can be made to work with 5,6 and 7

Standalone-ish OptiX 7 Examples

Basic check of CMake machinery, finding OptiX 7
Start from the SDK optixSphere example This example uses custom(aka analytic or non-triangulated) geometry. Follows the monolithic main layout of optixSphere, just adapting to use glm for viewpoint math.

Start from UseOptiX7GeometryStandalone Apply wrecking ball to the monolith, splitting into:

context, control
common types between CPU and GPU
pipeline of programs creation and updating
geometry acceleration structure building

Revisited this, tidying up the headers aiming to eliminate optix types from high levels in order to hide the version.

Attempting to switch UseOptiX7GeometryModular to use an instanced custom geometry for lots of spheres.

Started from UseOptiX7GeometryInstanced.

  1. pulled out the higher level geometry setup into Geo

2. uses a single IAS with multiple GAS 2. create big sphere containing a cube grid of two radii,

where the intersect program gets its sphere radius from Sbt record

Started from UseOptiX7GeometryInstancedGAS.

  • compound GAS, still not working
SBT mechanics worked out, using vectors of BI structs to keep count Find that have to fudge the bbox larger to get expected results ?

CMake Structure

Opticks is structured as a collection of ~20 modular CMake sub-projects organized by their dependencies. The sub-projects are hooked together into a tree using the CMake find_package mechanism which uses BCM (Boost CMake Modules) to reduce CMake boilerplate. The upshot is that you only need to worry about one level of dependencires

Bash Functions

Bash functions are used for building the tree of CMake projects, see om.bash