Hunting for new physics from cosmic initial conditions to cosmic acceleration: Fundamental theories in light of Stage IV cosmological surveys and cutting-edge inference techniques