Dislocation and grain boundary properties of Mg and its alloys were assessed by atomistic simulations using various semi-empirical potentials. The results were compared with available experimental data and first-principles calculations. It was observed that embedded-atom method (EAM) potentials notably deviated from experimental observations in dislocation properties, especially for pyramidal dislocations. For the coincident site lattice (CSL) grain boundary, the choice of interatomic potential significantly influences the minimum energy structures, namely, the T-type structure is stable when using modified embedded-atom method (MEAM) potentials, whereas the A-type structure demonstrates stability when EAM potentials are applied. Furthermore, a strong correlation between the per-site segregation energies of alloying elements and the excess free volumes at grain boundaries was identified across all tested potentials. The outcomes of this work pave the way for potential selection in future large-scale modeling of small-scale plasticity in Mg and its alloys.