The objects of perception, the objects of consensus reality, and the objects of the world are not the same sets. Perception is shaped by our priors, intentions, and the limits of our senses and cognition, as well as by the objective world we're looking at. Consensus reality is not just a median of different people's perceptions, but (for instance) carries extra information for shaping social behavior: "property (mine/yours)" exists in consensus reality even though it doesn't exist in physics or sense data. And even chunking the world into "separate things" is a mental judgment that often masquerades as a direct perception of reality.