The modeling of subsurface geometry and properties is a key element to understand Earth processes and manage natural hazards and resources. In this paper, we suggest this field should evolve beyond pure data fitting approaches by integrating geological concepts to constrain interpretations or test their consistency. This process necessarily calls for adding the time dimension to 3D modeling, both at the geological and human time scales. Also, instead of striving for one single best model, it is appropriate to generate several possible subsurface models in order to convey a quantitative sense of uncertainty. Depending on the modeling objective (e.g., quantification of natural resources, production forecast), this population of models can be ranked. Inverse theory then provides a framework to validate (or rather invalidate) models which are not compatible with certain types of observations. We review recent methods to better achieve both stochastic and time-varying geomodeling and advocate that the application of inversion should rely not only on random field models, but also on geological concepts and parameters.