auditlog: switch to pghistory (for real) (#13587)

* auditlog: switch to pghistory

* ruff

* pghistory: add finding.reviewers to tracked models

* fix finding reviewers model registration

* remove more references

* add migration

* rebase migrations

* rebase migrations

* rebase

* ruff

* rebase

* cleanup

* remove obsolete test

* ruff

* move auditlog in settings.dist.py

---------

Co-authored-by: Valentijn Scholten <valentijn.scholten@iodigital.com>

Author: valentijnscholten

.github/workflows/integration-tests.yml

@@ -2,18 +2,12 @@ name: Integration tests
 
 on:
   workflow_call:
-    inputs:
-        auditlog_type:
-          type: string
-          default: "django-auditlog"
 
 jobs:
   integration_tests:
     # run tests with docker compose
     name: User Interface Tests
     runs-on: ubuntu-latest
-    env:
-      AUDITLOG_TYPE: ${{ inputs.auditlog_type }}
     strategy:
       matrix:
         test-case: [

.github/workflows/rest-framework-tests.yml

@@ -6,16 +6,11 @@ on:
         platform:
           type: string
           default: "linux/amd64"
-        auditlog_type:
-          type: string
-          default: "django-auditlog"
 
 jobs:
   unit_tests:
     name: Rest Framework Unit Tests
     runs-on: ${{ inputs.platform ==  'linux/arm64' && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
-    env:
-      AUDITLOG_TYPE: ${{ inputs.auditlog_type }}
 
     strategy:
       matrix:

.github/workflows/unit-tests.yml

@@ -25,26 +25,18 @@ jobs:
     strategy:
         matrix:
             platform: ['linux/amd64', 'linux/arm64']
-            auditlog_type: ['django-auditlog', 'django-pghistory']
         fail-fast: false
     needs: build-docker-containers
     uses: ./.github/workflows/rest-framework-tests.yml
     secrets: inherit
     with:
       platform: ${{ matrix.platform}}
-      auditlog_type: ${{ matrix.auditlog_type }}
 
   # only run integration tests for linux/amd64 (default)
   test-user-interface:
     needs: build-docker-containers
     uses: ./.github/workflows/integration-tests.yml
     secrets: inherit
-    strategy:
-      matrix:
-        auditlog_type: ['django-auditlog', 'django-pghistory']
-      fail-fast: false
-    with:
-      auditlog_type: ${{ matrix.auditlog_type }}
 
   # only run k8s tests for linux/amd64 (default)
   test-k8s:

docs/content/en/open_source/upgrading/2.53.md

@@ -1,5 +1,5 @@
 ---
-title: 'Upgrading to DefectDojo Version 2.53.x'
+title: "Upgrading to DefectDojo Version 2.53.x"
 toc_hide: true
 weight: -20251103
 description: "Helm chart: changes for initializer annotations + Replaced Redis with Valkey + HPA & PDB support"
@@ -17,9 +17,9 @@ Added Helm chart support for Celery and Django deployments for Horizontal Pod Au
 
 ### Breaking changes
 
-#### Valkey 
+#### Valkey
 
-##### Renamed values 
+##### Renamed values
 
 HELM values had been changed to the following:
 - `createRedisSecret` → `createValkeySecret`
@@ -40,7 +40,7 @@ If an external Redis instance is being used, set the parameter `valkey.enabled`
 0. As always, perform a backup of your instance
 1. If you would like to be 100% sure that you do not miss any async event (triggered deduplication, email notification, ...) it is recommended to perform the following substeps (if your system is not in production and/or you are willing to miss some notifications or postpone deduplication to a later time, feel free to skip these substeps)
    0. Perform the following steps with your previous version of HELM chart (not with the upgraded one - you might lose your data)
-   1. Downscale all producers of async tasks: 
+   1. Downscale all producers of async tasks:
       - Set `django.replicas` to 0 (if you used HPA, adjust it based on your needs)
       - Set `celery.beat.replicas` to 0 (if you used HPA, adjust it based on your needs)
       - Do not change `celery.worker.replicas` (they are responsible for processing your async tasks)

docs/content/en/open_source/upgrading/2.54.md

@@ -1,11 +1,47 @@
 ---
 title: 'Upgrading to DefectDojo Version 2.54.x'
 toc_hide: true
-weight: -20250804
-description: Dropped support for DD_PARSER_EXCLUDE
+weight: -20251201
+description: Removal of django-auditlog and exclusive use of django-pghistory for audit logging & Dropped support for DD_PARSER_EXCLUDE
 ---
 
+## Breaking Change: Removal of django-auditlog
+
+Starting with DefectDojo 2.53, `django-auditlog` support has been removed in favour of `django-pghistory`.
+This is designed to be a backwards compatible change, unless:
+- You're querying the database directly for auditlog events, or,
+- You've set the `DD_AUDITLOG_TYPE` environment variable (or `AUDITLOG_TYPE` settings field)
+
+### Required Actions
+
+If you're using `DD_AUDITLOG_TYPE`, remove it from your configuration/environment.
+
+### Existing Records Preserved
+
+Historical audit log entries stored in the `auditlog_logentry` table will continue to be displayed in the action history view for backward compatibility. No data migration is required.
+
+### Benefits of django-pghistory
+
+The switch to `django-pghistory` provides several advantages:
+
+- **Better performance**: Database-level triggers reduce overhead compared to Django signal-based auditing
+- **More features**: Enhanced context tracking and better support for complex queries
+- **Better data integrity**: PostgreSQL-native implementation ensures consistency
+
+### Migration Notes
+
+- A one-time data migration will take place to populate the `django-pghistory` tables with the initial snapshot of the tracked models.
+- The migration is designed to be fail-safe: if it fails for some reason, it will continue where it left off.
+- The migration can also be performed up front via
+  - `docker compose exec uwsgi bash -c "python manage.py pghistory_backfill_fast"`, or
+  - `docker compose exec uwsgi bash -c "python manage.py pghistory_backfill_simple"`, or
+  - `docker compose exec uwsgi bash -c "python manage.py pghistory_backfill"`
+
+The backfill migration is not mandatory to succeed. If it fails for some reason, the only side effect will be that the first auditlog diff will contain all fields of an object instead just the changed fields.
+
+## Dropped support for DD_PARSER_EXCLUDE
+
 To simplify the management of the DefectDojo application, parser exclusions are no longer controlled via the environment variable DD_PARSER_EXCLUDE or application settings. This variable is now unsupported.
 From now on, you should use the active flag in the Test_Type model to enable or disable parsers. Only parsers associated with active Test_Type entries will be available for use.
 
-There are other instructions for upgrading to 2.54.x. Check the Release Notes for the contents of the release.
+Check the [Release Notes](https://github.com/DefectDojo/django-DefectDojo/releases/tag/2.54.0) for the contents of the release.

dojo/apps.py

@@ -91,7 +91,8 @@ def ready(self):
         # Configure audit system after all models are loaded
         # This must be done in ready() to avoid "Models aren't loaded yet" errors
         # Note: pghistory models are registered here (no database access), but trigger
-        # enabling is handled via management command to avoid database access warnings
+        # enabling is handled in the entrpoint script to avoid database access warnings
+        # during startup
         register_django_pghistory_models()
         configure_audit_system()