Python3.12 support

[orena1]

There is an change in python 3.12 in which len(view) raise an error:
https://docs.python.org/3/library/stdtypes.html#memoryview
instead of 1

This causes in issue with msgpack when it tires to calculate the size, once we update msgpack it does not break the massage transport, but it does fail in bigwarp.
My intuition is that in python3.11 when an empry array was submitted to the workers, they actually received and np.array([0]) but now they receive and np.array([]) and this raise an exception for map_coordinates as map_coordinates([], coordinates, mode='nearest') raises an exception.
This fix works, although I am not sure I fully comprehend what is going on.

[cgoina]

you have a typo here

[orena1]

fixed, thanks

[orena1]

@GFleishman what do you think?

[orena1]

@GFleishman did you try your code with python3.12 ? I currently need to keep a branch alive just for this fix, will be very happy to not need the branch and just just pip install

[GFleishman]

Hi @orena1 - thanks for the email and I'm sorry to have kept you waiting so long with this request. I'm a scatter brained scientist working on lots of projects in parallel and proper maintenance of code is often delayed behind fresh projects with new data - I'm really grateful for your contribution and support of bigstream in general - and I'm also really grateful for your patience.

I don't quite understand the issue being addressed here. I see that you're wanting to address an edge case in apply_transform_to_coordinates where a deformation was given as the transform, but that deformation is empty. I think there are more fundamental places to address this issue (such as checking input requirements at the top of the function) but I would also be fine with this solution in the meantime if it was consistent with what is currently written but simply extended to address the edge case.

But it does not seem consistent with what is currently written. The original line is:

dX = np.array([interp(transform[..., i]) for i in range(ndims)]).transpose()

The proposed change is:

for i in range(ndims):
    if transform[..., i]:
        dX.append(interp(transform[..., i]))
    else:
        dX.append(interp([0])

The transpose and conversion to a numpy array (np.array(...)) are missing.

I have to think more closely about it, but IIRC the transpose is essential for putting the displacements, that is dX in a consistent format with the input coordinates. This is because map_coordinates wants the vector components as the first axis, but the convention used across most of bigstream is to have the vector components as the last axis. If you have run the above code, I'm confused why it did not error since it seems like you would be adding a shape DxN array (displacements) to a shape NxD array (the coordinates).

The conversion to a numpy array is technically not necessary, since in the subsequent line:

coordinates = coordinates.transpose() * spacing + dX

we assume coordinates is a numpy array, so a conversion would happen automatically. I just like having things like this be explicit in the code sometimes, but I could let this part go.

So in summary, can you let me know when this edge case would ever come up? That is, why an empty deformation would ever be passed to this function? And also, can you let me know why the transpose was not included?

[orena1]

Thanks @GFleishman I am not sure why we get empty deformation, I think it happens when a block is just empty all zeros. This is not something new in python 3.12

See here, in python 3.11 len(np.array([])) was 1 but now it just raises an exception, that is why we can not send an empty array with msgpack although before we could.

I actually think that this happens a lot, probably even in your experiments, you just don't see it, did you try to create a new env with python3.12 and run your tests? I wonder if this is hidden?

Does it helps?