After upgrading to recent versions of jupyterlab, ipympl and ipykernel I observed on different systems, either Windows 11 or Linux, that ipympl figures would not show up sometimes without an explicit plt.show() at the end of the cell.
While this looked to me at first glance like an ipympl bug because it does not appear using inline figures it turns out that downgrading ipykernel to 6.3.1 completely solves the issue.
Here is the the original report with more details: matplotlib/ipympl#609
I saw that ipykernel>=7 has quite a few changes regarding matplotlib, so I assume this is a unwanted side-effect of these changes.