The brain is constantly making inferences about environmental variables in order to interpret stimuli and plan actions. In order to do so, it needs to maintain a model of the environment in which inferences on not directly observable quantities can be accomplished. We argue that the noisy and incomplete data received through the senses motivates a probabilistic representation. This approach enables us to build ideal observer models to establish optimal performance. We will discuss why sensory cortex cares little about what the senses communicate, how a noisy nervous system can be a design that seeks to achieve optimal performance, and why the processing pipeline is two-directional in a hierarchical model of the environment?