There's currently no real solution here short of using analog output. You need vsync_adjust=2 (no framebuffer) for minimal lag and that's going to necessitate drops when changing cores at least.
VRR isn't fully implemented unless vsync_adjust=0 but even then it seems implemented incorrectly. (I get a lot of wobble on an LG C1 when doing this.)