Merge pull request #40 from isc-tdyar/062-fix-iris-vector

fix: add entity_types parameter to extract_batch_with_dspy (Feature 062)

Author: tom-dyar

.specify/memory/constitution.md

@@ -287,6 +287,18 @@ Production deployments MUST include:
 
 **Package Management**: All Python projects MUST use uv for dependency management, virtual environment creation, and package installation. Traditional pip/virtualenv workflows are deprecated in favor of uv's superior performance and reliability.
 
+**Testing Standards (NON-NEGOTIABLE)**:
+
+All tests MUST be executed using `python -m pytest` instead of standalone `pytest` command. This ensures correct Python module resolution and prevents import failures.
+
+**Critical Lesson Learned**:
+- ❌ `pytest command` → imports failed (wrong Python path)
+- ✅ `python -m pytest` → all tests pass
+
+**Always use:** `python -m pytest` for reliable module imports
+
+**Rationale**: The standalone `pytest` command may use a different Python interpreter or sys.path configuration than the active virtual environment, leading to module import errors even when packages are correctly installed. Using `python -m pytest` guarantees tests run with the same Python interpreter and import paths as the application code.
+
 Code MUST pass linting (black, isort, flake8, mypy) before commits. All public APIs MUST include comprehensive docstrings. Breaking changes MUST follow semantic versioning. Dependencies MUST be pinned and regularly updated for security.
 
 Documentation MUST include quickstart guides, API references, and integration examples. Agent-specific guidance files (CLAUDE.md) MUST be maintained for AI development assistance.

CHANGELOG.md

@@ -1,5 +1,169 @@
 # Changelog
 
+## [0.5.5] - 2025-11-16
+
+### Fixed - Entity Types Configuration Bug (Feature 062)
+- **Entity Types Configuration**: `EntityExtractionService.extract_batch_with_dspy()` now accepts and honors `entity_types` parameter from configuration
+  - **Issue**: Configured entity types were ignored and healthcare-specific defaults (USER, MODULE, VERSION) were always used
+  - **Root Cause**: Method signature lacked `entity_types` parameter, couldn't pass config to `TrakCareEntityExtractionModule`
+  - **Fix**: Added `entity_types: Optional[List[str]] = None` parameter with resolution chain: parameter > config > DEFAULT_ENTITY_TYPES
+  - **Impact**: HotpotQA Question 2 now answers correctly (F1 improved from 0.000 to >0.0)
+  - **Files Modified**: `iris_vector_rag/services/entity_extraction.py`
+    - Line 41-49: Added `DEFAULT_ENTITY_TYPES` constant with domain-neutral defaults
+    - Line 890-955: Updated `extract_batch_with_dspy()` signature and implementation
+    - Added parameter validation (ValueError for empty list)
+    - Added warning logging for unknown entity types
+    - Updated docstring with parameter documentation and examples
+
+### Added
+- `DEFAULT_ENTITY_TYPES` constant for domain-neutral entity type defaults
+  - Values: `["PERSON", "ORGANIZATION", "LOCATION", "PRODUCT", "EVENT"]`
+  - Replaces healthcare-specific defaults (USER, MODULE, VERSION) when configuration missing
+- `entity_types` parameter to `EntityExtractionService.extract_batch_with_dspy()`
+  - Backward compatible (defaults to None)
+  - Validation for empty list (raises ValueError with clear message)
+  - Warning logging for unknown entity types (supports custom types)
+- Contract tests (`tests/contract/test_entity_types_config.py`)
+  - 7 tests validating parameter acceptance, defaults, validation, typing, and backward compatibility
+  - Test Results: 6/7 passing (1 skipped due to service initialization requirements)
+
+## [0.5.4] - 2025-11-14
+
+### Fixed - Critical Bug Fixes
+- **CRITICAL (Bug 1)**: Fixed AttributeError breaking all database connections (iris_dbapi_connector.py:210)
+  - **Issue**: Non-existent `iris.connect()` method caused AttributeError in v0.5.3
+  - **Fix**: Replaced with correct `iris.createConnection()` API
+  - **Impact**: Restores database connectivity (was completely broken in v0.5.3)
+  - **Test Results**: FHIR-AI test suite now 6/6 passing (up from 3/6 in v0.5.3)
+    - ✅ ConfigurationManager (backward compatibility preserved)
+    - ✅ ConnectionManager (was failing - now fixed)
+    - ✅ IRISVectorStore (was failing - now fixed)
+    - ✅ SchemaManager (was failing - now fixed)
+    - ✅ Environment Variables (backward compatibility preserved)
+    - ✅ Document Model (backward compatibility preserved)
+  - **Files Modified**: `iris_vector_rag/common/iris_dbapi_connector.py`
+    - Line 210: `iris.connect()` → `iris.createConnection()`
+    - Enhanced error handling: AttributeError → ConnectionError with clear messages
+    - Updated docstrings and log messages
+
+- **HIGH PRIORITY (Bug 2)**: Added automatic iris-vector-graph table initialization
+  - **Issue**: Silent PPR (Personalized PageRank) failures due to missing database tables
+  - **Fix**: Automatic detection and creation of iris-vector-graph tables during pipeline initialization
+  - **Impact**: Eliminates "Table not found" errors for GraphRAG operations
+  - **Performance**: Table initialization completes in < 5 seconds (4 tables created)
+  - **Tables Created**: rdf_labels, rdf_props, rdf_edges, kg_NodeEmbeddings_optimized
+  - **Files Modified**: `iris_vector_rag/storage/schema_manager.py`
+    - Added `_detect_iris_vector_graph()` method (uses importlib.util.find_spec)
+    - Added `ensure_iris_vector_graph_tables()` public method
+    - Added `validate_graph_prerequisites()` validation method
+    - Added `InitializationResult` dataclass for table creation results
+    - Added `ValidationResult` dataclass for prerequisite validation results
+
+### Technical Details
+**Bug 1 - Connection API Fix**:
+- Root Cause: intersystems-irispython v5.3.0 provides `iris.createConnection()`, not `iris.connect()`
+- Error Messages: Clear ConnectionError with connection parameters and remediation steps
+- Backward Compatibility: No breaking changes to public APIs
+- Testing: Contract tests verify no AttributeError during connection establishment
+
+**Bug 2 - Schema Initialization**:
+- Detection: Non-invasive package detection (no import side effects)
+- Initialization: Idempotent table creation (safe to call multiple times)
+- Validation: Clear error messages listing specific missing prerequisites
+- Graceful Degradation: Skips initialization when iris-vector-graph not installed
+- Logging: INFO for success, ERROR for failures with actionable context
+
+### Migration Notes
+**From v0.5.3 to v0.5.4**:
+- No action required - bug fixes are backward compatible
+- ConnectionManager automatically uses correct API
+- SchemaManager automatically initializes iris-vector-graph tables if package detected
+- Optional: Run `SchemaManager.validate_graph_prerequisites()` to verify setup
+
+## [0.5.3] - 2025-11-12
+
+### Fixed
+- **CRITICAL**: Fixed SchemaManager bug where VECTOR_DIMENSION environment variable was ignored
+  - SchemaManager now correctly reads vector dimension from CloudConfiguration API
+  - Previous behavior: Always returned default 384 dimensions regardless of VECTOR_DIMENSION env var
+  - New behavior: Respects configuration priority (env > config > defaults) via Feature 058 CloudConfiguration
+  - Impact: Fixes FHIR-AI-Hackathon deployment issues where custom embedding dimensions were required
+- Fixed iris.dbapi import issues in connection_pool.py
+  - Replaced invalid `Connection` type hints with `Any` (iris.dbapi doesn't export Connection class)
+  - Removed incorrect `from iris.dbapi import Connection` import
+
+### Added
+- **Integration Test Coverage**: 9 comprehensive integration tests against real IRIS database
+  - `TestConnectionManagerIntegration`: 2 tests validating ConnectionManager with CloudConfiguration
+  - `TestSchemaManagerIntegration`: 3 tests validating SchemaManager dimension configuration
+  - `TestConfigurationPriorityChain`: 3 tests validating env > config > defaults priority
+  - `TestCompleteConfigurationFlow`: 1 test validating end-to-end configuration to database
+  - All tests verify real IRIS database operations (not mocked)
+  - Test Results: 9/9 passing (100%)
+
+### Technical Details
+- Files Modified:
+  - `iris_vector_rag/storage/schema_manager.py` - Lines 49-77: Changed from incorrect `config.get("embedding_model.dimension", 384)` to `cloud_config.vector.vector_dimension`
+  - `iris_vector_rag/common/connection_pool.py` - Replaced 7 Connection type hints with Any
+- Test Coverage: Added `tests/integration/test_cloud_config_integration.py` (400 lines)
+- FHIR-AI-Hackathon Compatibility: SchemaManager now properly reads VECTOR_DIMENSION=1024 and other custom dimensions
+
+## [0.5.2] - 2025-11-12
+
+### Added - Cloud Configuration Flexibility (Feature 058)
+- **Environment Variable Support**: Configure IRIS connection via environment variables
+  - `IRIS_HOST`, `IRIS_PORT`, `IRIS_USERNAME`, `IRIS_PASSWORD`, `IRIS_NAMESPACE`
+  - `VECTOR_DIMENSION` (128-8192), `TABLE_SCHEMA` for cloud deployments
+- **12-Factor App Configuration**: Priority order (env > config > defaults)
+- **Configuration Source Tracking**: Audit trail showing where each value originated
+- **Vector Dimension Flexibility**: Support 128-8192 dimensions for different embedding models
+  - 384: SentenceTransformers (default)
+  - 1024: NVIDIA NIM, Cohere
+  - 1536: OpenAI ada-002
+  - 3072: OpenAI text-embedding-3-large
+- **Table Schema Configuration**: Configurable schema prefix via `TABLE_SCHEMA` env var
+  - AWS: SQLUser schema requirement
+  - Azure: RAG schema (default)
+  - Local: RAG schema (default)
+- **Validation Framework**: Preflight validation for vector dimensions and namespaces
+  - `VectorDimensionValidator`: Prevents data corruption from dimension mismatches
+  - `NamespaceValidator`: Validates namespace permissions
+  - `PreflightValidator`: Orchestrates all validation checks
+- **Configuration Entities**: Strongly-typed configuration models
+  - `ConnectionConfiguration`, `VectorConfiguration`, `TableConfiguration`
+  - `CloudConfiguration` with `.validate()` method
+- **Deployment Examples**: Production-ready configuration templates
+  - `config/examples/aws.yaml` - AWS IRIS (%SYS namespace, SQLUser schema)
+  - `config/examples/azure.yaml` - Azure IRIS (USER namespace, Azure Key Vault)
+  - `config/examples/local.yaml` - Local development (Docker-ready)
+  - `config/examples/README.md` - Comprehensive deployment guide
+- **Password Masking**: Automatic password masking in configuration logs (`***MASKED***`)
+
+### Changed
+- `ConfigurationManager.get_cloud_config()`: New method for cloud deployment configuration
+- Configuration priority system ensures 100% backward compatibility
+- All existing APIs continue to work unchanged (v0.4.x compatible)
+
+### Fixed
+- Cloud deployment configuration now properly respects environment variables
+- Vector dimension validation prevents data corruption from configuration errors
+
+### Documentation
+- Added cloud deployment examples for AWS, Azure, and local environments
+- Security best practices for secret management (AWS Secrets Manager, Azure Key Vault)
+- Configuration troubleshooting guide with common errors and solutions
+
+### Technical Details
+- **Files Added**: 7 new files (~1,500 lines)
+  - `iris_vector_rag/config/entities.py` (380 lines)
+  - `iris_vector_rag/config/validators.py` (420 lines)
+  - Configuration examples and documentation (700 lines)
+- **Files Modified**: 2 files
+  - `iris_vector_rag/config/manager.py` - Added `get_cloud_config()` method
+  - Contract test suite - 18/22 tests passing (82%)
+- **Test Coverage**: 18 contract tests passing, all unit tests passing
+- **Zero Breaking Changes**: 100% backward compatible with v0.5.1
+
 ## [0.5.1] - 2025-11-09
 
 ### Fixed