The current policy for moving tasks (in priority-ordered lists in Maniphest, and in general on Workboards) is that having edit permission on the task is necessary and sufficient to drag it.
However, this is not communicated in the UI: tasks which you can not move because you don't have permission to edit them have no visual distinction from tasks which you can move. Immovable tasks should be clearly marked to make this distinction obvious and help hint policy expectations to users.
Original Report
I think i have seen this reported before but I can't find it:
If you don't have permission to edit a task, and try to move it on a workboard in natural view, you get an error. This in unexpected. You shouldn't be able to edit the priority this way but you should be able to move around tasks on an editable workboard.