WIP: Persist index data #38
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a working progress, but it aims to resolve #10
Problem
Index rebuilds can be slow because field values need to be calculated for every piece of content. This involves property value handlers, data collection services, and various transformations - all of which are computationally expensive when performed across thousands of documents.
Solution
Persist the calculated
IndexField[]data to a database table (umbSearchDocuments) when content is indexed. During index rebuilds, this cached data can be reused instead of recalculating fields from scratch.A
useDatabaseparameter controls this behavior:useDatabase: true(default) - Use cached fields from database if availableuseDatabase: false- Ignore cached data and recalculate fields freshWhen content types are modified (properties added/removed), the system automatically passes
useDatabase: falseto ensure stale cached data isn't used.What's Been Done
Database Layer
Documentmodel for persisted index data (Models/Persistence/Document.cs)DocumentDtofor database mappingIDocumentRepositoryandDocumentRepositorywith CRUD operationsIDocumentServiceandDocumentServiceas the service layerCustomPackageMigrationto create theumbSearchDocumentstableContent Change Strategies
DraftContentChangeStrategyto persist fields on index and use cached data on rebuildPublishedContentChangeStrategywith the same persistence patternGetManyAsyncfor batch fetching documents (optimizes descendant processing)IContentChangeStrategy.RebuildAsyncsignature to includeuseDatabaseparameterIndex Rebuild Flow
IContentIndexingService.Rebuildto acceptuseDatabaseparameterRebuildIndexesNotificationHandlerto passuseDatabase: falseon content type changes and language deletion