-
Notifications
You must be signed in to change notification settings - Fork 2
Add fixed shape tensor #18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds support for the Arrow canonical extension type arrow.fixed_shape_tensor, which enables storing fixed-shape tensors in Arrow format. The implementation follows the Apache Arrow specification for fixed shape tensor arrays.
Key Changes
- Introduces
fixed_shape_tensor_arrayclass wrappingfixed_sized_list_arraywith tensor-specific metadata - Implements JSON serialization/deserialization for tensor metadata (shape, dimension names, permutation)
- Replaces nlohmann_json dependency with simdjson for improved JSON parsing performance
- Adds comprehensive test coverage with 752 lines of tests
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| include/sparrow_extensions/fixed_shape_tensor.hpp | Defines the fixed_shape_tensor_array class and metadata structures with comprehensive documentation |
| src/fixed_shape_tensor.cpp | Implements metadata validation, JSON serialization, and array construction logic |
| tests/test_fixed_shape_tensor.cpp | Provides extensive test coverage for metadata operations, array construction, and spec compliance |
| cmake/external_dependencies.cmake | Adds simdjson dependency and updates doctest version |
| environment-dev.yml | Replaces nlohmann_json with simdjson in conda dependencies |
| CMakeLists.txt | Registers new source files and headers in build system |
| include/sparrow_extensions.hpp | Includes the new fixed_shape_tensor header in main library header |
| tests/CMakeLists.txt | Adds test file to test suite |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
No description provided.