I think the biggest issue I have with "perfect" shots is that this is when the randomness left/right in your shot seems the greatest. I think the best examples are the two Par 3 holes. I understand that there's going to be a greater randomness because it's off the tee, but if I line up the shot to the right of the hole by 5-10 yards, hit a perfect shot, the ball winds up to the left of the hole (speaking on hole 8, which almost always has the wind blowing at you). However, if I don't hit a perfect shot, but am close to the perfect shot line, the ball flies very close to where I aimed.
I'm sure I'm missing something, though. This is my observation after numerous rounds, but there's probably something I'm not accounting for.