Electrical current pulses can be used to manipulate magnetization efficiently via spin-orbit torques (SOTs). Pulse durations as short as a few picoseconds have been used to switch the magnetization of ferromagnetic films, reaching the THz regime. However, little is known about the reversal mechanisms and energy requirements in the ultrafast switching regime. In this work, we quantify the energy cost for magnetization reversal over 7 orders of magnitude in pulse duration, in both ferromagnetic and ferrimagnetic samples, bridging quasi-static spintronics and femtomagnetism. To this end, we develop a method to stretch picosecond pulses generated by a photoconductive switch by an order of magnitude. Thereby, we can create current pulses from picoseconds to durations approaching pulse width available with commercial instruments. We show that the energy cost for SOT switching decreases by more than an order of magnitude in all samples when the pulse duration enters the picosecond range. We project an energy cost of 9 fJ for a 100 x 100 nm 2 ferrimagnetic device. Micromagnetic and macrospin simulations unveil a transition from a non-coherent to a coherent magnetization reversal with a strong modification of the magnetization dynamical trajectories as pulse duration is reduced. Our results cement the potential for high-speed magnetic spin-orbit torque memories and highlights alternative magnetization reversal pathways at fast time scales.