This work introduces the Fractal Diffusion Model, which builds on Fractal Generative Models and adapts the fractal framework by replacing the autoregressive modules with conditional diffusion models. Previous research has demonstrated that recursive modelling of text and images can yield consistently good results. We explore using DiT as a recursive atomic module in this paradigm to enable the parallel denoising of image patches to achieve sampling acceleration while preserving the self-similarity structure and fidelity of the generated content. In addition, we find that modulating shared weights of recursive networks can be done efficiently using small hypernetworks conditioned on diffusion timestep and the fractal depth in combination with unshared readout blocks to avoid representation collapse.