Since babies and toddlers are not old enough to understand the idea of "danger", it is very important to protect them at all times.
The fact that they are very curious, means that it is better to keep the electrical outlets dangerous (either empty or in use) well out of their reach.
These small holes in the walls stimulate the curiosity and thirst for exploration of all the toddlers. All the means are good: the fingers, a piece of toy, a metallic object and everything that will pass under their hand. The safest way is to protect all your unused outlets with safety outlet plates. Cover used outlets with a safety box. Use Power Strip Covers to avoid prong exposure.