To understand the necessary condition for the observation of all-optical helicity-dependent switching (AO-HDS) of magnetization in thin films, we investigated ferromagnetic Co/Pt and Co/Ni multilayers as well as ferrimagnetic TbCo alloys as a function of magnetic layer compositions and thicknesses. We show that both ferro-and ferrimagnets with high saturation magnetization show AO-HDS if their magnetic thickness is strongly reduced below a material-dependent threshold thickness. By taking into account the demagnetizing energy and the domain wall energy, we are able to define a criterion to predict whether AO-HDS or thermal demagnetization (TD) will be observed. This criterion for the observation of AO-HDS is that the equilibrium size of magnetic domains forming during the cooling process should be larger than the laser spot size. From these results we anticipate that more magnetic materials are expected to show AO-HDS. However, the effect of the optical pulses' helicity is hidden by the formation of small magnetic domains during the cooling process.