I always thought that the surfer furthest out has priority. As in surfing longboarders that don't share seem to love this rule. I'd agree with the upwind sailor having priority, as long as the wave in question is a predictable reef or point.
On an unpredictable beach break.... I think the first sailor on the wave should have priority, as someone could have sailed a long distance out looking for a big wave, then (on the way in) followed the peak downwind or upwind looking for the steepest section or best wall.