Ref T5240. Currently, we calculate drag positions assuming the "ghost" element is not present (it isn't, usually), then adjust them while dragging to account for the ghost.
However, this fails after scrolling: we dirty the cache, but the ghost is present. We continue adjusting for it, but essentially double-adjust. This leads to scroll positions being about 80-ish px off from where they should be.