(chroma_env)delta:~ blyth$ chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483650
INFO:chroma: vertices: 1264049
INFO:chroma:Building new BVH using recursive grid algorithm.
Expanding 22240 parent nodes
Merging 2483650 nodes to 79179 parents
Expanding 3675 parent nodes
Merging 220475 nodes to 61739 parents
Expanding 235 parent nodes
Merging 65805 nodes to 18559 parents
Expanding 8 parent nodes
Merging 18806 nodes to 4597 parents
Merging 4605 nodes to 1196 parents
Merging 1196 nodes to 300 parents
Merging 300 nodes to 95 parents
Expanding 1 parent nodes
Merging 95 nodes to 30 parents
Merging 31 nodes to 8 parents
Merging 8 nodes to 2 parents
Merging 2 nodes to 1 parent
INFO:chroma:BVH generated in 7.2 seconds.
INFO:chroma:Saving BVH (c600c30494ebdd3c9fed0474e09f9a80:default) to cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x116824cd0>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
/usr/local/env/chroma_env/lib/python2.7/site-packages/pycuda/characterize.py:40: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(45): warning: variable "NCHILD_MASK" was declared but never referenced
kernel.cu(45): warning: variable "NCHILD_MASK" was declared but never referenced
""" % (preamble, type_name), no_extern_c=True)
Traceback (most recent call last):
File "/usr/local/env/chroma_env/bin/chroma-cam", line 8, in <module>
execfile(__file__)
File "/usr/local/env/chroma_env/src/chroma/bin/chroma-cam", line 37, in <module>
view_nofork(geometry, size)
File "/usr/local/env/chroma_env/src/chroma/chroma/camera.py", line 849, in view_nofork
camera._run()
File "/usr/local/env/chroma_env/src/chroma/chroma/camera.py", line 639, in _run
self.init_gpu()
File "/usr/local/env/chroma_env/src/chroma/chroma/camera.py", line 87, in init_gpu
self.gpu_geometry = gpu.GPUGeometry(self.geometry)
File "/usr/local/env/chroma_env/src/chroma/chroma/gpu/geometry.py", line 41, in __init__
raise Exception('one or more triangles is missing a material.')
Exception: one or more triangles is missing a material.
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
Abort trap: 6
(chroma_env)delta:~ blyth$
(chroma_env)delta:~ blyth$ chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
WARNING:env.geant4.geometry.collada.collada_to_chroma:setting parent_material to __dd__Materials__Vacuum0xaf1d298 as parent is None for node top.0
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483650
INFO:chroma: vertices: 1264049
INFO:chroma:Loading BVH "default" for geometry from cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x10db83150>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
Traceback (most recent call last):
File "/usr/local/env/chroma_env/bin/chroma-cam", line 8, in <module>
execfile(__file__)
File "/usr/local/env/chroma_env/src/chroma/bin/chroma-cam", line 37, in <module>
view_nofork(geometry, size)
File "/usr/local/env/chroma_env/src/chroma/chroma/camera.py", line 849, in view_nofork
camera._run()
File "/usr/local/env/chroma_env/src/chroma/chroma/camera.py", line 639, in _run
self.init_gpu()
File "/usr/local/env/chroma_env/src/chroma/chroma/camera.py", line 87, in init_gpu
self.gpu_geometry = gpu.GPUGeometry(self.geometry)
File "/usr/local/env/chroma_env/src/chroma/chroma/gpu/geometry.py", line 43, in __init__
refractive_index = interp_material_property(wavelengths, material.refractive_index)
File "/usr/local/env/chroma_env/src/chroma/chroma/gpu/geometry.py", line 35, in interp_material_property
return np.interp(wavelengths, property[:,0], property[:,1]).astype(np.float32)
TypeError: 'NoneType' object has no attribute '__getitem__'
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
Abort trap: 6
Succeed to visualize via Chroma, but due to hugeness of world volume navigation is near impossible.
(chroma_env)delta:~ blyth$ chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
WARNING:env.geant4.geometry.collada.collada_to_chroma:setting parent_material to __dd__Materials__Vacuum0xaf1d298 as parent is None for node top.0
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483650
INFO:chroma: vertices: 1264049
INFO:chroma:Loading BVH "default" for geometry from cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x11954f190>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
INFO:chroma:Optimization: Sufficient memory to move triangles onto GPU
INFO:chroma:Optimization: Sufficient memory to move vertices onto GPU
INFO:chroma:device usage:
----------
nodes 2.8M 44.7M
total 44.7M
----------
device total 2.1G
device used 243.2M
device free 1.9G
/usr/local/env/chroma_env/src/chroma/chroma/gpu/tools.py:32: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(198): warning: integer conversion resulted in a change of sign
kernel.cu(198): warning: integer conversion resulted in a change of sign
no_extern_c=True)
Observe a flaky but now repeatable situation where the nodes total is overstated and get a black window:
(chroma_env)delta:~ blyth$ chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483638
INFO:chroma: vertices: 1264041
INFO:chroma:Loading BVH "default" for geometry from cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x10f9cb190>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
INFO:chroma:Optimization: Sufficient memory to move triangles onto GPU
INFO:chroma:Optimization: Sufficient memory to move vertices onto GPU
INFO:chroma:device usage:
----------
nodes 3.4M 54.0M
total 54.0M
----------
device total 2.1G
device used 318.7M
device free 1.8G
(chroma_env)delta:~ blyth$
Clearing the BVH cache, fails to resolve:
(chroma_env)delta:~ blyth$ rm -rf ~/.chroma/bvh
(chroma_env)delta:~ blyth$ chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483638
INFO:chroma: vertices: 1264041
INFO:chroma:Building new BVH using recursive grid algorithm.
Expanding 20256 parent nodes
Merging 2483638 nodes to 624241 parents
Expanding 543 parent nodes
Merging 651151 nodes to 167090 parents
Expanding 223 parent nodes
Merging 167863 nodes to 50865 parents
Expanding 49 parent nodes
Merging 51088 nodes to 15739 parents
Expanding 9 parent nodes
Merging 15788 nodes to 3680 parents
Merging 3689 nodes to 897 parents
Merging 897 nodes to 205 parents
Merging 205 nodes to 51 parents
Merging 51 nodes to 15 parents
Merging 15 nodes to 4 parents
Merging 4 nodes to 1 parent
INFO:chroma:BVH generated in 3.6 seconds.
INFO:chroma:Saving BVH (ce8a94af92438d400589a5faeb5b9f37:default) to cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x1151c2190>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
INFO:chroma:Optimization: Sufficient memory to move triangles onto GPU
INFO:chroma:Optimization: Sufficient memory to move vertices onto GPU
INFO:chroma:device usage:
----------
nodes 3.4M 54.0M
total 54.0M
----------
device total 2.1G
device used 493.5M
device free 1.7G
Switch on CUDA_PROFILE in hope to learn where pycuda is caching its kernels, no joy:
(chroma_env)delta:chroma_camera blyth$ CUDA_PROFILE=1 chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
...
(chroma_env)delta:chroma_camera blyth$ head -10 cuda_profile_0.log
# CUDA_PROFILE_LOG_VERSION 2.0
# CUDA_DEVICE 0 GeForce GT 750M
# CUDA_CONTEXT 1
method,gputime,cputime,occupancy
method=[ write_size ] gputime=[ 7.424 ] cputime=[ 17.288 ] occupancy=[ 0.016 ]
method=[ memcpyDtoH ] gputime=[ 5.280 ] cputime=[ 538.104 ]
method=[ write_size ] gputime=[ 6.848 ] cputime=[ 9.459 ] occupancy=[ 0.016 ]
method=[ memcpyDtoH ] gputime=[ 4.352 ] cputime=[ 1097.478 ]
method=[ write_size ] gputime=[ 4.256 ] cputime=[ 7.456 ] occupancy=[ 0.016 ]
method=[ memcpyDtoH ] gputime=[ 3.872 ] cputime=[ 17.787 ]
(chroma_env)delta:chroma_camera blyth$ tail -10 cuda_profile_0.log
method=[ memcpyHtoD ] gputime=[ 1.344 ] cputime=[ 1.983 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.374 ]
method=[ memcpyHtoD ] gputime=[ 1.344 ] cputime=[ 1.941 ]
method=[ memcpyHtoD ] gputime=[ 1.344 ] cputime=[ 1.965 ]
method=[ memcpyHtoD ] gputime=[ 1142.528 ] cputime=[ 1021.808 ]
method=[ memcpyHtoD ] gputime=[ 1144.320 ] cputime=[ 989.674 ]
method=[ fill ] gputime=[ 51.712 ] cputime=[ 11.983 ] occupancy=[ 1.000 ]
method=[ fill ] gputime=[ 54.016 ] cputime=[ 16.531 ] occupancy=[ 1.000 ]
method=[ render ] gputime=[ 1384.512 ] cputime=[ 224.247 ] occupancy=[ 0.500 ]
method=[ memcpyDtoH ] gputime=[ 578.752 ] cputime=[ 2295.950 ]
After a restart, same problem:
delta:chroma_camera blyth$ cat chroma_camera_test.sh
#!/bin/bash -l
chroma-
which python
which chroma-cam
chroma-cam $ENV_HOME/geant4/geometry/materials/g4_00.dae
delta:chroma_camera blyth$ ./chroma_camera_test.sh
/usr/local/env/chroma_env/bin/python
/usr/local/env/chroma_env/bin/chroma-cam
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483638
INFO:chroma: vertices: 1264041
INFO:chroma:Loading BVH "default" for geometry from cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x110de9190>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
/usr/local/env/chroma_env/lib/python2.7/site-packages/pycuda/characterize.py:40: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(45): warning: variable "NCHILD_MASK" was declared but never referenced
kernel.cu(45): warning: variable "NCHILD_MASK" was declared but never referenced
""" % (preamble, type_name), no_extern_c=True)
INFO:chroma:Optimization: Sufficient memory to move triangles onto GPU
INFO:chroma:Optimization: Sufficient memory to move vertices onto GPU
INFO:chroma:device usage:
----------
nodes 3.4M 54.0M
total 54.0M
----------
device total 2.1G
device used 305.1M
device free 1.8G
/usr/local/env/chroma_env/src/chroma/chroma/gpu/tools.py:32: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(198): warning: integer conversion resulted in a change of sign
kernel.cu(198): warning: integer conversion resulted in a change of sign
no_extern_c=True)
Careful observation of output (triangle counts) reveals the cause to be the attempt to skip the overly large top.0 node, which I though I had backed out of.:
delta:chroma_camera blyth$ ./chroma_camera_test.sh
/usr/local/env/chroma_env/bin/python
/usr/local/env/chroma_env/bin/chroma-cam
WARNING:env.geant4.geometry.collada.daenode:failed to find parent for top.0 - (failure expected only for root node)
WARNING:env.geant4.geometry.collada.collada_to_chroma:setting parent_material to __dd__Materials__Vacuum0xaf1d298 as parent is None for node top.0
INFO:chroma:Flattening detector mesh...
INFO:chroma: triangles: 2483650
INFO:chroma: vertices: 1264049
INFO:chroma:Building new BVH using recursive grid algorithm.
Expanding 22240 parent nodes
Merging 2483650 nodes to 79179 parents
Expanding 3675 parent nodes
Merging 220475 nodes to 61739 parents
Expanding 235 parent nodes
Merging 65805 nodes to 18559 parents
Expanding 8 parent nodes
Merging 18806 nodes to 4597 parents
Merging 4605 nodes to 1196 parents
Merging 1196 nodes to 300 parents
Merging 300 nodes to 95 parents
Expanding 1 parent nodes
Merging 95 nodes to 30 parents
Merging 31 nodes to 8 parents
Merging 8 nodes to 2 parents
Merging 2 nodes to 1 parent
INFO:chroma:BVH generated in 6.4 seconds.
INFO:chroma:Saving BVH (c600c30494ebdd3c9fed0474e09f9a80:default) to cache.
INFO:chroma:loaded geometry <chroma.geometry.Geometry object at 0x1173d1150>
INFO:chroma:starting view [1024, 576]
INFO:chroma:create Camera
INFO:chroma:Camera.__init__
INFO:chroma:Camera.__init__ done
INFO:chroma:_run Camera
INFO:chroma:Optimization: Sufficient memory to move triangles onto GPU
INFO:chroma:Optimization: Sufficient memory to move vertices onto GPU
INFO:chroma:device usage:
----------
nodes 2.8M 44.7M
total 44.7M
----------
device total 2.1G
device used 362.0M
device free 1.8G
delta:chroma_camera blyth$
Potential causes of GPU panics:
# CUDA_PROFILE_LOG_VERSION 2.0
# CUDA_DEVICE 0 GeForce GT 750M
# CUDA_CONTEXT 1
method,gputime,cputime,occupancy
method=[ write_size ] gputime=[ 2.528 ] cputime=[ 17.956 ] occupancy=[ 0.016 ]
method=[ memcpyDtoH ] gputime=[ 2.496 ] cputime=[ 26.433 ]
method=[ write_size ] gputime=[ 1.856 ] cputime=[ 9.821 ] occupancy=[ 0.016 ]
method=[ memcpyDtoH ] gputime=[ 2.496 ] cputime=[ 20.545 ]
method=[ write_size ] gputime=[ 1.824 ] cputime=[ 9.927 ] occupancy=[ 0.016 ]
method=[ memcpyDtoH ] gputime=[ 2.496 ] cputime=[ 17.852 ]
method=[ memcpyHtoD ] gputime=[ 1.440 ] cputime=[ 4.299 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 3.469 ]
...
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 2.022 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 10.942 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 4.039 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.034 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 1.891 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 1.912 ]
method=[ memcpyHtoD ] gputime=[ 1794.976 ] cputime=[ 1993.471 ]
method=[ memcpyHtoD ] gputime=[ 1617.952 ] cputime=[ 1481.204 ]
method=[ memcpyHtoD ] gputime=[ 1601.280 ] cputime=[ 1472.250 ]
method=[ memcpyHtoD ] gputime=[ 7432.672 ] cputime=[ 7370.140 ]
method=[ memcpyHtoD ] gputime=[ 4602.432 ] cputime=[ 4620.065 ]
method=[ memcpyHtoD ] gputime=[ 2335.680 ] cputime=[ 2351.582 ]
method=[ memcpyHtoD ] gputime=[ 1.664 ] cputime=[ 5.372 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.315 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 2.037 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 1.973 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 2.185 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 2.113 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.008 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.010 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.372 ]
method=[ memcpyHtoD ] gputime=[ 1.312 ] cputime=[ 2.009 ]
method=[ memcpyHtoD ] gputime=[ 1.280 ] cputime=[ 1.959 ]
method=[ memcpyHtoD ] gputime=[ 612.832 ] cputime=[ 501.086 ]
method=[ memcpyHtoD ] gputime=[ 590.560 ] cputime=[ 449.675 ]
method=[ fill ] gputime=[ 24.544 ] cputime=[ 13.470 ] occupancy=[ 1.000 ]
method=[ fill ] gputime=[ 25.504 ] cputime=[ 7.263 ] occupancy=[ 1.000 ]
method=[ render ] gputime=[ 5259416.500 ] cputime=[ 234.175 ] occupancy=[ 0.500 ]
method=[ memcpyDtoH ] gputime=[ 194.016 ] cputime=[ 5260492.000 ]