Where's your caddy, indeed. I have played many rounds and I haven't seen one yet.
To your point, and to the 'official' rules of golf (specifically 17-1), says you may have the flagstick attended, removed or held up prior to making a stroke from anywhere on the course. The ball better not hit the flagstick or the attendant, though, or it’ll cost you either loss of hole or two strokes.
Since it wasn't attended before you made the stroke, let's hope it doesn't become attended as your ball gets closer... or it's two strokes.
Havng said all that in fun, let me just say that it's a game and that is one of the quirks of the game that make it interesting. I have had an approach hit the pin and bounce straight back at me... quite humorous indeed.