Multi-Stage Migrations

[quasiyoke]

Observed

Issue #1383 and #1402 brought to our attention that in the current approach to writing migrations:

• It might be better to abort the migration exceeding block weight altogether and demand immediate attention from the devs.
• Triggering a panic in migration would be effectively the same as exhausting the block weight — the chain won't be able to progress.

Expected

We'd like to write multi-stage migrations that limit each step of their execution to how much free weight we have at our disposal.

Suggestions

• Polkadot SDK since v1.9.0 has Multi-block-migrations — probably we could use them for such purposes. Please note:
  • MBM ensure transactional execution of migrations — the runtime code won't see the state in a partially migrated state.
  • MBM provide WeightMeter, which allows us to work with weight as we need.
• Perhaps, before we have Multi-block-migrations, it is worth designing our own mechanism for stepwise execution of migrations, which will use the unoccupied weight from the on_idle hook:
  • Issues #1383 and #1402 do not require pausing the processing of other transactions (may be designed in a way that state keys affected by migrations don't intersect with modifications in extrinsics).