What makes the supernatural natural?

Sean Carroll (the Caltech physicist) has a blog at Discover magazine. Yesterday, in a readworthy post by the title What Questions Can Science Answer?, he wrote

Here is my favorite example question. Alpha Centauri A is a G-type star a little over four light years away. Now pick some very particular moment one billion years ago, and zoom in to the precise center of the star. Protons and electrons are colliding with each other all the time. Consider the collision of two electrons nearest to that exact time and that precise point in space. Now let’s ask: was momentum conserved in that collision? Or, to make it slightly more empirical, was the magnitude of the total momentum after the collision within one percent of the magnitude of the total momentum before the collision?

This isn’t supposed to be a trick question; I don’t have any special knowledge or theories about the interior of Alpha Centauri that you don’t have. The scientific answer to this question is: of course, the momentum was conserved. Conservation of momentum is a principle of science that has been tested to very high accuracy by all sorts of experiments, we have every reason to believe it held true in that particular collision, and absolutely no reason to doubt it; therefore, it’s perfectly reasonable to say that momentum was conserved.

A stickler might argue, well, you shouldn’t be so sure. You didn’t observe that particular event, after all, and more importantly there’s no conceivable way that you could collect data at the present time that would answer the question one way or the other. Science is an empirical endeavor, and should remain silent about things for which no empirical adjudication is possible.

But that’s completely crazy. That’s not how science works. Of course we can say that momentum was conserved. Indeed, if anyone were to take the logic of the previous paragraph seriously, science would be a completely worthless endeavor, because we could never make any statements about the future. Predictions would be impossible, because they haven’t happened yet, so we don’t have any data about them, so science would have to be silent.
All that is completely mixed-up, because science does not proceed phenomenon by phenomenon. Science constructs theories, and then compares them to empirically-collected data, and decides which theories provide better fits to the data. The definition of “better” is notoriously slippery in this case, but one thing is clear: if two theories make the same kinds of predictions for observable phenomena, but one is much simpler, we’re always going to prefer the simpler one. The definition of theory is also occasionally troublesome, but the humble language shouldn’t obscure the potential reach of the idea: whether we call them theories, models, hypotheses, or what have you, science passes judgment on ideas about how the world works.

And that's precisely right. I like to say that science is about building models; build models to explain a collection of facts, and the most parsimonious model wins.

But then Carroll says something that I think is slightly wrong:

Could science, through its strategy of judging hypotheses on the basis of comparison with empirical data, ever move beyond naturalism to conclude that some sort of supernatural influence was a necessary feature of explaining what happens in the world? Sure; why not? If supernatural phenomena really did exist, and really did influence things that happened in the world, science would do its best to figure that out.

What I think is wrong with that statement is not a problem of what science can handle, but rather what is meant by 'supernatural.' I agree that too much blood has been spilled over definitions, but being a swashbuckler aficionado, I like blood. If a phenomenon can be studied by science, then it is by definition a natural phenomenon. If supernatural phenomena really did exist, we would not call them supernatural. In fact, it is the job of science to make all things supernatural natural. Carroll gives an example of angels, with scientists struggling to come up with models that explain them, but all fail.

Eventually, they agree that the most compelling and economical theory is one with two parts: a natural part, based on unyielding rules, with a certain well-defined range of applicability, and a supernatural one, for which no rules can be found.

I find this hard to swallow. Not the angels (well, that too), but that scientists would ever give up. Michael Shermer likes to ask what evidence would convince a hardened atheist of God's existence. Suppose "God" personified appeared before you in a flash, cut off your right arm, and then made it grow back in a second. Would that convince you that God existed? Well, not really. Appearing in a flash could very well happen by some trickery that we have not yet invented. Who knows what we will have learned a thousand or a million years from now. It is not hard to imagine that a technology that could make someone appear suddenly could be invented that far down the line. Same thing for regrowing an arm. We know other species who can do exactly that (albeit not that fast), so perhaps with medical advances it will one day become possible. And such theories would be more parsimonious that positing a supernatural being, which would take much more explaining than limb regeneration on steroids:

Mammals like us can regenerate skin or fuse broken bones back together, but salamanders can replace a lost limb in a few weeks, regrow damaged lungs, mend a severed spinal cord and even replenish lost chunks of brain.

Then what evidence would you need? I have thought about this at length, and can also not find anything that would convince me for sure that God exists. If truly supernatural-looking strange things happen, it can always be posited that they are illusions of the mind (think The Matrix). I know some people will argue that this is a cop-out, but the truth is that the real cop-out is the position that there is a supernatural being that made it all.

If anyone has a good example of conclusive evidence for a supernatural being, please share it!