NDFileHDF5 write failure with Stream2 + BS-LZ4 (Decompress=0)

[domitto]

I’m opening this issue to track the problem with Stream2 when using BS-LZ4 and Decompress=0.

I’m able to reproduce the failure consistently. When starting a capture with Stream2 + BS-LZ4 (single threshold), NDFileHDF5 immediately reports:

NDPluginFile::doCapture: ERROR, must collect an array to get dimensions first
NDFileHDF5Dataset::writeFile ERROR Unable to write pre-compressed data - mismatched chunk definition
NDFileHDF5::writeFile ERROR: could not write to dataset. Aborting
NDPluginFile::writeFileBase Error writing file, status=3

If I press Start Capture again, the IOC crashes with:

terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc

Compression filters (bitshuffle/LZ4) are installed, and HDF5_PLUGIN_PATH is set correctly, so it appears to be related to how Stream2 delivers the compressed data.

@MarkRivers mentioned that Stream2 may not work with pre-compressed data because plugins expect the full [NX, NY, NTHRESHOLDS] block to be compressed as a single chunk, while Dectris sends each threshold image separately.

I’ll test the regular Stream interface tomorrow as a workaround and report back with results.

[domitto]

Quick update:

I tested both Stream and Stream2, and unfortunately, I’m seeing the exact same behavior in both cases. The only way I can save HDF5 files without crashing the IOC is to set Decompress = Enabled. With decompression disabled, the NDFileHDF5 errors and subsequent IOC crash are fully reproducible.

Enabling decompression works, but it’s noticeably slower than writing the pre-compressed data directly — so ideally I’d like to avoid that extra step.

Let me know if there’s anything else I can test or any additional diagnostics you’d like.

[MarkRivers]

@domitto can you please post screenshots of the ADEiger and NDPluginHDF5 screens when you have this problem?

[domitto]

[MarkRivers]

I tested this today on our Eiger2S 9M. I cannot reproduce the problem. I am using the master branch on Github. I disabled decompression on the Stream interface, and sent the compressed data to the PVA1 and the HDF5 plugins. It worked fine with both the Stream and Stream2 interfaces.

I think I see the problem in the screenshot of your HDF5 plugin. You have set the Array port to PVA1. Did you do that intentionally, and if so, why? That would normally be set to EIG, or to some other plugin that does processing and exports NDArrays.

[domitto]

Thanks for checking this on your end. The array port being set to PVA1 was not intentional — this IOC is running under ProcServ and actively used by the beamline staff, so I’m guessing someone was experimenting with plugin configurations and the array port change slipped through. I’ll have access to the detector again tomorrow and will retest with the Array Port set back to EIG to see if that resolves the issue.

Were you able to reproduce the failure simply by switching the array port to PVA1 on your system, or did it still work correctly there? Even with a misconfigured port, the bad_alloc crash isn’t great.

I’ll report back after I rerun the test.

Thanks,
Diego

[MarkRivers]

I did not test with the HDF5 plugin getting its data from PVA1. I only saw that configuration after I did my tests, and the detector is now in use. I agree it should not crash. If you can make it crash again with PVA1 then run the IOC under gdb and post the backtrace when it crashes.

[domitto]

It is still crashing when getting data from EIG. I'm working on getting the debug target so I can get the gdb backtrace.

[MarkRivers]

I just noticed that you are using External Gate and Pump and Probe mode. I may understand why that is crashing, because that collects 2 images, not just 1 and they are compressed individually but the HDF5 plugin would expect a single 3-D array compressed as a block.

Please switch the Trigger Mode to Internal Series and see if that works OK with both Stream and Stream2. That is what I have tested.

[MarkRivers]

I'm working on getting the debug target so I can get the gdb backtrace.

You don't need a debug build to get the backtrace. Some things may be optimized out, but you will still get the line numbers in the source code where it is failing.

[domitto]

Internal series is crashing with compressed files too, here's the bt without the debug target:

sioc-hut1-ei01> auto_settings.sav: 1414 of 1414 PV's connected
[New Thread 0x7fffa2e2e700 (LWP 1312973)]
[New Thread 0x7ffef4cec700 (LWP 1313018)]
[New Thread 0x7ffe8b049700 (LWP 1313019)]
[New Thread 0x7fffa0fe4700 (LWP 1313029)]
[New Thread 0x7ffef5574700 (LWP 1313030)]
[New Thread 0x7fffa34f5700 (LWP 1313031)]
[New Thread 0x7ffef5373700 (LWP 1313032)]
[New Thread 0x7ffff174d700 (LWP 1313033)]
[New Thread 0x7ffef44eb700 (LWP 1313034)]
[New Thread 0x7fffa3dfe700 (LWP 1313035)]
[New Thread 0x7ffef42ea700 (LWP 1313036)]
[New Thread 0x7fffa01aa700 (LWP 1313037)]
[New Thread 0x7ffe8a848700 (LWP 1313038)]
[New Thread 0x7fffa2a2a700 (LWP 1313039)]
[New Thread 0x7ffe8a647700 (LWP 1313040)]
[New Thread 0x7ffff1149700 (LWP 1313041)]
[New Thread 0x7ffe8a446700 (LWP 1313042)]
[New Thread 0x7fffa3afb700 (LWP 1313043)]
[New Thread 0x7ffe8a245700 (LWP 1313044)]
[New Thread 0x7fffa37f8700 (LWP 1313045)]
[New Thread 0x7ffe8a044700 (LWP 1313046)]
2025/12/03 13:32:33.331 NDFileHDF5Dataset::writeFile ERROR Unable to write pre-compressed data - mismatched chunk definition
2025/12/03 13:32:33.331 NDFileHDF5::writeFile ERROR: could not write to dataset. Aborting
2025/12/03 13:32:33.332 NDPluginFile::writeFileBase Error writing file, status=3
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Thread 44 "FileHDF1" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff063e700 (LWP 1312696)]
0x00007ffff5e8652f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install boost-system-1.66.0-13.el8.x86_64 glibc-2.28-251.el8_10.14.x86_64 keyutils-libs-1.5.10-9.el8.x86_64 krb5-libs-1.18.2-31.el8_10.x86_64 libX11-1.6.8-9.el8_10.x86_64 libXau-1.0.9-3.el8.x86_64 libblkid-2.32.1-46.el8.x86_64 libcom_err-1.45.6-5.el8.x86_64 libgcc-8.5.0-24.el8_10.x86_64 libmount-2.32.1-46.el8.x86_64 libselinux-2.9-10.el8_10.x86_64 libsodium-1.0.18-2.el8.x86_64 libstdc++-8.5.0-24.el8_10.x86_64 libunwind-1.3.1-3.el8.x86_64 libuuid-2.32.1-46.el8.x86_64 libxcb-1.13.1-1.el8.x86_64 openpgm-5.2.122-21.el8.x86_64 openssl-libs-1.1.1k-14.el8_10.x86_64 pcre2-10.32-3.el8_6.x86_64 readline-7.0-10.el8.x86_64 systemd-libs-239-82.el8_10.4.x86_64 zeromq-4.3.4-3.el8.x86_64 zlib-1.2.11-26.el8.x86_64
(gdb) bt
#0  0x00007ffff5e8652f in raise () from /lib64/libc.so.6
#1  0x00007ffff5e59e65 in abort () from /lib64/libc.so.6
#2  0x00007ffff683809b in __gnu_cxx::__verbose_terminate_handler() [clone .cold.1] () from /lib64/libstdc++.so.6
#3  0x00007ffff683e54c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#4  0x00007ffff683e5a7 in std::terminate() () from /lib64/libstdc++.so.6
#5  0x00007ffff683e808 in __cxa_throw () from /lib64/libstdc++.so.6
#6  0x00007ffff683edc4 in operator new(unsigned long) () from /lib64/libstdc++.so.6
#7  0x00000000006b5731 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (
    __end=0x406947ecd9d0e0c9 <error: Cannot access memory at address 0x406947ecd9d0e0c9>, 
    __beg=0x202c2779726f7463 <error: Cannot access memory at address 0x202c2779726f7463>, this=0x7ffff063d1e0) at /usr/include/c++/8/bits/char_traits.h:357
#8  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char*> (
    __end=0x406947ecd9d0e0c9 <error: Cannot access memory at address 0x406947ecd9d0e0c9>, 
    __beg=0x202c2779726f7463 <error: Cannot access memory at address 0x202c2779726f7463>, this=0x7ffff063d1e0) at /usr/include/c++/8/bits/basic_string.h:240
#9  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (
    __end=0x406947ecd9d0e0c9 <error: Cannot access memory at address 0x406947ecd9d0e0c9>, 
    __beg=0x202c2779726f7463 <error: Cannot access memory at address 0x202c2779726f7463>, this=0x7ffff063d1e0) at /usr/include/c++/8/bits/basic_string.h:259
#10 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__str=..., this=0x7ffff063d1e0)
    at /usr/include/c++/8/bits/basic_string.h:444
#11 NDFileHDF5::createHardLinks (this=0x1d8dbf0, root=0x7fffa81964f0) at ../NDFileHDF5.cpp:736
#12 0x00000000006b578c in NDFileHDF5::createHardLinks (this=0x1d8dbf0, root=0x7fffa8195260) at ../NDFileHDF5.cpp:750
#13 0x00000000006b5831 in NDFileHDF5::createHardLinks (this=0x1d8dbf0, root=0x7fffa8195160) at ../NDFileHDF5.cpp:729
#14 0x00000000006c285e in NDFileHDF5::openFile (this=0x1d8dbf0, fileName=0x7ffff063d6b0 "/tmp/test_008.h5", openMode=<optimized out>, pArray=0x7fff1ccd22f0)
    at ../NDFileHDF5.cpp:320
#15 0x00000000006de32d in NDPluginFile::openFileBase (this=this@entry=0x1d8dbf0, openMode=openMode@entry=10, pArray=pArray@entry=0x7fff1ccd22f0)
    at ../NDPluginFile.cpp:65
#16 0x00000000006de925 in NDPluginFile::doCapture (this=this@entry=0x1d8dbf0, capture=capture@entry=1) at ../NDPluginFile.cpp:479
#17 0x00000000006dfd66 in NDPluginFile::writeInt32 (this=this@entry=0x1d8dbf0, pasynUser=pasynUser@entry=0x37771e8, value=value@entry=1) at ../NDPluginFile.cpp:850
#18 0x00000000006b9a98 in NDFileHDF5::writeInt32 (this=0x1d8dbf0, pasynUser=0x37771e8, value=1) at ../NDFileHDF5.cpp:2057
#19 0x00000000011b3a9b in writeInt32 (drvPvt=0x1d8dbf0, pasynUser=0x37771e8, value=1) at ../../asyn/asynPortDriver/asynPortDriver.cpp:1996
#20 0x000000000120d9c5 in processCallbackOutput (pasynUser=0x37771e8) at ../devBusyAsyn.c:315
#21 0x00000000011ac39d in portThread (pport=0x1d46380) at ../../asyn/asynDriver/asynManager.c:913
#22 0x000000000131aba5 in start_routine (arg=0x1d466c0) at ../osi/os/posix/osdThread.c:441
#23 0x00007ffff71a01ca in start_thread () from /lib64/libpthread.so.0
#24 0x00007ffff5e718d3 in clone () from /lib64/libc.so.6

[MarkRivers]

Please repost screenshots of the ADEiger screen and the HDF5 plugin screen when this happens in Internal Series mode.

What version of the HDF5 library are you using? The one in ADSupport, or a different version on your system?

[MarkRivers]

I just noticed that you set NumFramesChunk=100. Please set that to 0 or 1 and test again. I am setting it to 0 (the default) and then the plugin sets it to 1 (readback).

[domitto]

What is the easiest way to get the HDF5 version?

[domitto]

It did not crash with Frames cached per chunk set back to 0.
What does that even do - other than clearly crashing the IOC?

[MarkRivers]

What is the easiest way to get the HDF5 version?

We could add that information to the HDF5 plugin by querying the library. Right now you need to look at your areaDetector/configure/CONFIG_SITE.local file to see if HDF5_EXTERNAL is NO (uses ADSupport) or YES (uses external library).

I suspect the problem is your chunking given this error message:

2025/12/03 13:32:33.331 NDFileHDF5Dataset::writeFile ERROR Unable to write pre-compressed data - mismatched chunk definition