Most interface elements are fixed in place. This makes the interface stable, makes them easy to find. It seems generally accepted that having interface elements appear and disappear is a bad thing. It is connected with modality.
However, ecological psychology suggests that we perceive the world in terms of context sensitive affordances. It's natural that there is no grasping functionality when you are looking at a wall.