Shared metadata cache #8819

AlexPeshkoff · 2025-11-30T18:55:15Z

Tasks to be solved.

Have a single metadata cache that is used by all attachments in both jrd and dsql.
Provide the ability to mix DLL/DML in a single transaction, including working with new/modified elements.

The old metadata cache is essentially single-threaded. For me, the problem I faced with attempts to introduce synchronization into this structure and get off with relatively little blood was taking into account the use of procedures and functions using a second reference counter, and in order to decide whether to delete an object, a global recalculation of these references is required throughout the cache. dsqlcache is also extremely unsympathetic, which puts objects created and still unmixed in a single transaction for all transactions of the same attach. An additional consideration is that objects stored in the cache are used in a huge number of places in the code, and they can be modified from other places (and threads) during MET_scan_xxx. This could still be dealt with using CopyOnWrite, but how to approach the first two points was unclear.

The new metadata cache is versioned. Each object is divided into 2 parts - a constant and a variable. For relation, for example, the type (table, view, external table, etc.), name, all locks (GCLock, partners lock, rescan lock) will be constant, and format, list of fields, triggers will be variable. The list of versions is linked to a permanent part of the object. As versions become obsolete, the list is cleared (although I won't say that it's very active - only as the cache is accessed, there is no special garbage collector).

In order to access the desired object for reading without locks, it was possible to allocate memory for the entire cache at once. In principle, there are not so many - 64K pointers to each type of cached objects. But I kept in mind the possible increase in the number of objects in the database and therefore made a 2-level array - each lower-level block of N objects is allocated once and for all (until the database is closed), and the upper-level block has the right to grow when the required number of lower-level blocks does not fit there. Therefore, the upper level is implemented using a read-safe array. For reading, we would take a slice of the current state of the array and this slice remains unchanged while we work with it. An external mutex is used to modify the array. This array (SharedReadVector) turned out to be very convenient, I also use it for requests in a statement and a list of formats in a relation. The 2-level arrays (CacheVector) for all objects are the same (template), and all this ultimately lives in DBB.

Ideally, I would like to remove dsqlcache altogether. So far, it has been removed for relations, procedures, and functions. At the same time, I did not abandon the dsql_rel format and the like - they are needed in order to be able to work with it when creating/changing an object (dsql_rel), although there is no such object in the metadata cache yet / or the format is not the same. Temporary objects of this format are created in dsqlScratch and die with it (by pool).

libcds is used to implement secure pointers (HazardPtr), they are needed for version headers in the list of object versions and for a slice in the SharedReadVector.

In all requests, instead of a pointer directly to objects, the cache stores a structure of 2 elements - a pointer to the permanent part of the object and a index in an array of pointers to versions of objects. These arrays may differ in different query clones. To reduce duplication, a RefCounted array is used. rpb_relation's are registered in rpb when creating a clone. The decision on whether to update a set of versions in a clone of the request is made in GetRequest, that is, when a clone of the request is received.

At the moment, TCS is fully working. QA makes mistakes - but I think it's time to pour metacache into the master so as not to prevent merge of other branches. And it's probably time to decide with the indexes of the system tables, which ones are needed and which ones are not very necessary. For example, I need to add indexes by ID for relations and other cached tables.

…ons)

…s related fixes around

… erased)

TreeHunter9 · 2025-12-02T14:31:19Z

src/common/dsc.h

 		{
 			if (dsc_sub_type == isc_blob_text)
-				return dsc_scale | (dsc_flags & 0xFF00);
+				return TTypeId(CSetId(dsc_scale), CollId(dsc_flags > 8));


I guess bitshift was intended here?
Same at 305 line.

src/burp/restore.epp


 } // namespace

+


Noremos · 2025-12-02T13:44:25Z

src/common/classes/BlrReader.h


-namespace Firebird {

+namespace Firebird {


Unnecessary change

src/common/classes/BlrReader.h

 	const UCHAR* pos;
 };

-


Noremos · 2025-12-02T13:48:59Z

src/common/dsc.h

 		return 0;
 	}

+	SSHORT getBlobSubType() const noexcept


Unnecessary change

getBlobSubType & setBlobSubType got close to each other after it

Noremos · 2025-12-02T13:54:17Z

src/common/dsc.h

 		dsc_address = address;
 	}

+	bool operator==(const dsc& v) const noexcept


The presence of DSC_EQUIV and this overload operator is a bit confusing. While this may not be relevant to this PR, it would be a nice to remove external functions related to dsc, such as DSC_EQUIV and DTYPE_IS_TEXT.

Noremos · 2025-12-02T16:50:42Z

src/dsql/DsqlBatch.cpp

 	{
 		DEB_BATCH(fprintf(stderr, "BLOBs %d were not used in messages\n", m_blobMap.count()));
-		ERR_post_warning(Arg::Warning(isc_random) << "m_blobMap.count() BLOBs were not used in messages");		// !!!!!!! new warning
+		ERR_post_warning(Arg::Warning(isc_random) << "m_blobMap.count() BLOBs were not used in messages");		// @@@@@@ new warning


Unnecessary change. Or am I misunderstanding this

Noremos · 2025-12-02T16:58:42Z

src/jrd/sort.cpp

@@ -2188,7 +2188,6 @@ void SortOwner::releaseBuffer(UCHAR* memory)
 	buffers.push(memory);
 }

-


Unnecessary change

Noremos · 2025-12-02T16:58:51Z

src/jrd/sort.cpp

@@ -2442,3 +2441,4 @@ sort_record* PartitionedSort::getMerge()

 	return eof ? NULL : record;
 }
+


Unnecessary change

Noremos · 2025-12-02T17:10:08Z

src/dsql/ExprNodes.cpp

 	return node;
 }

+


Unnecessary change

Noremos · 2025-12-02T17:13:12Z

src/dsql/ExprNodes.cpp

 ValueExprNode* UdfCallNode::pass2(thread_db* tdbb, CompilerScratch* csb)
 {
-	if (function->fun_deterministic)
+	Function* f = function(tdbb);


It's probably better to be consistent with the names and call it func, as in UdfCallNode::execute below

… cleanup

…raint

Noremos · 2025-12-04T12:42:03Z

src/include/fb_exception.h

-	[[noreturn]] static void raise(const ISC_STATUS* status_vector);
-	[[noreturn]] static void raise(const Arg::StatusVector& statusVector);
-	[[noreturn]] static void raise(const IStatus* status);
+	static void raise [[noreturn]] (const ISC_STATUS* status_vector);


To be honest, I've never seen this style before and didn't know it was a valid format.

Both correct, see https://en.cppreference.com/w/cpp/language/attributes/noreturn.html

But taking into an account that other style is used else where I change such places

Noremos · 2025-12-04T12:43:14Z

src/intl/charsets.h

-#define   CS_ASCII               2	/* ASCII            */
-#define   CS_UNICODE_FSS         3	/* UNICODE in FSS format */
-#define   CS_UTF8                4	/* UTF-8 */
+#define   CS_NONE                CSetId(0)	/* No Character Set */


Is there any reason to keep this consatnts as defines?

Doubt that - but it worked and works

Noremos · 2025-12-04T12:48:08Z

src/jrd/Attachment.cpp

+
+	att_database->dbb_mdc->invalidateReplSet(tdbb);
+
+/* !!!!!!!!!!!!!!!!!!!!!!!


Some Work In Progress code?

Yes, looks like replication code required some more care to be taken.

Noremos · 2025-12-04T12:51:55Z

src/jrd/Collation.cpp

 void Collation::destroy(thread_db* tdbb)
 {
-	fb_assert(useCount == 0);
+	fprintf(stderr, "Collation::destroy(%p) tt=%p\n", this, tt);


Some debug print?

Yes, should be removed long ago. Fixed.

Noremos · 2025-12-05T09:34:37Z

src/dsql/DsqlBatch.cpp

 	{
 		DEB_BATCH(fprintf(stderr, "BLOBs %d were not used in messages\n", m_blobMap.count()));
-		ERR_post_warning(Arg::Warning(isc_random) << "m_blobMap.count() BLOBs were not used in messages");		// !!!!!!! new warning
+		ERR_post_warning(Arg::Warning(isc_random) << "m_blobMap.count() BLOBs were not used in messages");		// !!!!!! new warning


Does this line changed on porpuse? Is there something wrong with it?

The only wrong here is that isc_random to be replaced here with some special warning code.

Noremos · 2025-12-05T09:38:14Z

src/jrd/Database.cpp

+	{
+		fb_assert(which == IRQ_REQUESTS || which == DYN_REQUESTS || which == CACHED_REQUESTS);
+
+		//Database::CheckoutLockGuard guard(this, dbb_cmp_clone_mutex);


Temporaly change?

Really not needed - cleaned up

Noremos · 2025-12-05T12:57:21Z

src/jrd/IntlManager.cpp

 	ExternalInfo charSetExternalInfo;
 	ExternalInfo collationExternalInfo;
-	char statusBuffer[BUFFER_LARGE] = "";
+	char statusBufferBig[BUFFER_LARGE] = "";


Maybe using std::array will be a litle bit handy

Not sure - called functions are plain text oriented.

To be precise - everything (well almost everything) related with intl support worth redesign and new code. But as long as existing works I prefer to touch it as minimal as possible.

Noremos · 2025-12-10T06:41:07Z

src/jrd/RecordSourceNodes.cpp

+			viewProcedure = tail->csb_procedure();

-			// If the plan references the view itself, make sure that
+			// If the plan references  the view itself, make sure that


AlexPeshkoff added 30 commits November 25, 2021 20:18

Merge branch 'master' into metacache

2a397de

WIP - shared cache of sequences for replicator compiles fine

6234437

Avoid too many builds for a while

a178212

Shared cache of generators appears to be working

dd86eba

Before adding conversion to non-safe PTRs

c6383cb

Successfully compiled met.epp with hazard pointers to cache objects

3420540

Compiled shared metacache

76ddfee

Merge branch 'master' into metacache

4053ea8

Make it compile after merge

2917928

Use SharedReadVector in HazardArray

13895b9

Unify ctor

90a5d0e

Avoid unlimited recursion in hazard GC

55eac37

Merge branch 'master' into metacache

5c82cfb

Work in progress

cf85f59

WIP

25ff2f7

WIP

f0a0d07

WIP

0164b8a

Successful CREATE DATABASE with shared cache (still has some limitati…

5b85f04

…ons)

Misc

a25c5c5

Successfully created security.fdb

1ab5457

Completed DEV_BUILD, including creation of employee.fdb

9054816

Check for simple (trivially copyable) type in Array, some more or les…

1ff3c46

…s related fixes around

Some changes before merge

92a909a

Merge result

dae83f6

WIP

4215b9d

WIP - take into an account states of an object (needs load / normal /…

7736734

… erased)

WiP

462d01b

met.epp compiles

a16665e

jrd dir compiled

e1647ab

ExprNodes compiled

1ac8131

AlexPeshkoff added 8 commits November 28, 2025 14:14

Fixed per-statement memory stats

514922e

Enhancement needed for working DSQL statements cache

f4b6e63

Merge branch 'master' into metacache

2d25c50

This should fix clang build

4c6d657

Fixed building tests

36871e8

Next attempt to make clang happy

b9591b4

Avoid clang warning re. use of uninitialized variable

fdba886

Fixed races when taking rescan lock

840859f

TreeHunter9 reviewed Dec 2, 2025

View reviewed changes

AlexPeshkoff added 3 commits December 2, 2025 19:32

Fixed operator - thanks to TreeHunter9

4590dcc

Fixed pool leak after loadResources() call

070b9fa

Fixed pool pointer in internally allocated block

eac5814

Noremos reviewed Dec 2, 2025

View reviewed changes

AlexPeshkoff added 13 commits December 2, 2025 20:40

Fixed one more pool leak

d07752f

Redirected block does not contain red zone - do not check it

d685fa3

Clear versioned objects array before loading pointers to it

6872d46

Access to RelationPages::dpMap should be protected

5fd6b3e

Fixed access to system tables when doing fix_plugins_schemas()

f783a73

SyncLockGuard is needed when accessing DBB internals even for debugging

0bfd91e

Merge branch 'master' into metacache

9111147

Some changes suggested by Noremos after review

977f2ca

Some changes suggested by Noremos after review + more or less related…

83e637c

… cleanup

Avoid asserts when working with user indices on system tables

4c09a2c

System requests were released too early

fea0fe2

Fixed compound ALTER TABLE statement with ADD and DROP the same const…

0d8ef1e

…raint

Fixed ALTER FUNCTION/PROCEDURE when changing only SQL SECURITY clause

7692d65

Noremos reviewed Dec 10, 2025

View reviewed changes

AlexPeshkoff added 3 commits December 10, 2025 14:57

Not needed assert - depends upon data sent by client

7025216

Fixed segfault & assert for index-related actions

87e7de9

Style

30c94bc

		@@ -2188,7 +2188,6 @@ void SortOwner::releaseBuffer(UCHAR* memory)
		buffers.push(memory);
		}

		@@ -2442,3 +2441,4 @@ sort_record* PartitionedSort::getMerge()

		return eof ? NULL : record;
		}


		att_database->dbb_mdc->invalidateReplSet(tdbb);

		/* !!!!!!!!!!!!!!!!!!!!!!!

Uh oh!

Shared metadata cache #8819

Are you sure you want to change the base?

Shared metadata cache #8819

Conversation

AlexPeshkoff commented Nov 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants