Probes of cosmic expansion constitute the main basis for arguments to support or refute a possible apparent acceleration due to different expansion rates in the Universe as described by inhomogeneous cosmological models. We present in this Letter a separate argument based on results from an analysis of the growth rate of large-scale structure in the Universe as modeled by the inhomogeneous cosmological models of Szekeres. We use the models with no assumptions of spherical or axial symmetries. We find that while the Szekeres models can fit very well the observed expansion history without a Λ, they fail to produce the observed late-time suppression in the growth unless Λ is added to the dynamics. A simultaneous fit to the supernova and growth factor data shows that the cold dark matter model with a cosmological constant (ΛCDM) provides consistency with the data at a confidence level of 99.65%, while the Szekeres model without Λ achieves only a 60.46% level. When the data sets are considered separately, the Szekeres with no Λ fits the supernova data as well as the ΛCDM does, but provides a very poor fit to the growth data with only 31.31% consistency level compared to 99.99% for the ΛCDM. This absence of late-time growth suppression in inhomogeneous models without a Λ is consolidated by a physical explanation.