We found that by assuming that the density of dislocations in an irradiated material varies as a function of the distance to grain boundaries and that mobile interstitial defect clusters perform three-dimensional diffusional motion it is possible to achieve significantly better agreement with experimental observations of profiles of heterogeneous void swelling than in the model where defects diffuse purely one-dimensionally. This approach explains the origin of several distinct features characterising the effect of heterogeneous void swelling, including the variation of the shape of swelling profiles as a function of irradiation dose, the formation of peaks of swelling and void denuded zones, and the occurrence of anomalously large voids in the regions adjacent to grain boundaries.