Since priority is a global state and a task can appear in multiple projects, rearranging the ordering on one project's workboard can change its global priority, and push it down (or up) on another project's workboard.
The ordering of cards on a workboard shouldn't have to be directly tied to the task's priority.