This seems to be a common topic on the various M/L sites. If you are looking for protection from the big bears, .54 is almost adequate (see the Lewis & Clark journals, their 1803 rifles were in .54 and they had plenty of troubles with grizzlies not being stopped with the first shot).
Having said that, I have had a love affair with the .45 since I got my first one in 1968, a H&A underhammer with round ball twist. I currently have a Deer Creek flintlock rifle (basically a CVA Mountain Rifle) with a Colerain barrel in that caliber which sees the most use of any of my guns. But, I have a T-C PA Hunter, a CVA Bobcat (restocked in wood) and a Traditions Shenandoah in .50 simply because that is all they were available in. I'm thinking about sending the T-C to Bobby Hoit to be bored out to .54, or maybe .58 smoothbore.
The .45 round ball will do reliably on deer, even without huge powder charges if the hunter does his part. It has the advantage of not being overkill on small game, plus you get more projectiles from a pound of lead.which equates to less weight in your shooting bag for a given number of balls. If you limit your loads to a sane 70 grains FFFg your powder will last a lot longer also.