The concept of growing so slowly that one could not see this was a little too subtle for me at that age. But there is something a little deeper going on. I can leave my spectacles overnight close to the edge of the table without fearing that they are really moving at 3mm/hour - too slow for me to see yet fast enough that I will wake up in the morning to see them smashed on the floor. The nature of friction in the world we live in ensures that objects which seem to be still are very rarelyin reality moving too slowly for us to see.
Contrary to what we like to think, our brains are designed to live in the real world - or any rate what was the real world 40,000 years ago, rather than to discover ultimate truths. So it is natural to feel that something that appears to be still is really still, and not moving too slowly for us to notice. Perhaps the limits of what we can see moving are by design such that objects on the scale we deal with are rarely moving below the threshold of our congnition. And growth is presumably understood as movement.