I guess it depends on how far you are going to take the vintage Trans-Am theme. I would cross an old school big block off the list right away, unless its all aluminum. Think R.J. Gottlieb's big red Camaro

. But back on planet earth where budgets exist, I see three ways for this to play out.
1. True vintage Trans-Am which to me means Donahue in the Penske/Sunoco car with a real cross ram 302. Probably not the most potent combo, nor the easiest to tune, but would say pretty wicked and the

factor whent the hood opens would be up there.
2. Updated take on the Trans-Am there would be to take advantage of the advances in engine technology and build something along the lines of what johnjzjz mentioned. A large displacement SBC, big bore/short stroke with all the goodies and let it sing. Would still sound mean as hell and only you would know the true displacement, so 302 badges/decals would fill the bill nicely

.
3. Modern muscle, LS-based all aluminum motor. You could do 427 inches make and easy 500-550hp NA and then decide to keep the injection or go carbed.
I would do number 2, but add the cross ram intake.....hahahahaa best of all worlds.
A few pics to strengthen my case.
Chris