How hot is too hot?
Barrels in the summer will heat up to a point that they are even uncomfortable to touch. It has happened to me countless times. Your question of how hot is too hot is a good one. You will start to notice your groups going south or you will get the flier from outer space. No real reason for it, the thing just happened. I think the heat interferes with the harmonics of the barrel causing them fliers or sudden expanded groups.
I have known people to take a cooler filled with ice and water and towels, and after the barrel is swabbed, pull an ice cold towel from the cooler, ring the water out of the towel and wrap the barrel with the ice towel while they check targets or what not. The ice cold towel of course will cool the barrel some.
I take three rifles to the range on a hot day. And I shoot and rest them in order. Just the sitting in the shade under the shooting bench (out of the way) for that five or ten more minutes will really help in the barrel staying cool.
But when you ask how hot is too hot? You will know the answer to that. I usually go by touch, and group size. Also, I take my time when I shoot. Having my own range, I race no one but myself. I can swab the rifle, set it down, wander around, just relax and enjoy the back yard. I know some people at public ranges can not do this (and I pity you). But you can slow down on the speed you swab, maybe run two patches or an extra dry patch. Study the target a little longer. Make better notes about what is going on with the load your developing. Just slow down on hot day. It will help your groups.