Inspired in part by Guess who's smarter
Under the banner of the "nature/nurture" debate, we discuss how things like language acquisition and perhaps common sense can happen so fast with so little apparent training.
Perhaps the answer is that in one small, isolated training scenario, we are actually training our entire brain lightly through an extensive, slow internal training process. Every time we learn one little thing about the world, it ripples through everything we ever learned and makes teeny tiny little adjustments.
This would mean two things: that a computer could spend a few years on a relatively small training set gaining common sense or language abilities, assuming we could find a suitable "internal training" algorithm. And second, that training operations are expensive. The Nth training OP takes at least O(N) if not O(N^K) where K is the average number of connections. Even with years to train, I'm not sure that's tractable.
One relevant question is: what kind of pre-existing structures need to be in place for such learning to occur. Can anything be learned without such structures?
Update: The first paragraph of this paper briefly talks about "knowledge transfer" and "inductive bias shifts" which I think are related. Some papers are cited.