Disabling voice lock mode

I am trying to disable voice lock mode which I find distracting. I can successfully disable it in a local buffer with C-e d v, but it is enabled again as soon as I switch to another buffer. I have also tried C-u -1 C-e d v to turn it off globally, which yielded the same result. I also tried putting the following in my ~/.emacs file:

(voice-lock-mode -1)

I also tried (global-voice-lock-mode -1) and I tried setting the mode’s variables to 0 or nil; none of these things worked. What I would like is to have a way to disable voice-lock-mode and not have it be enabled whenever I switch to another buffer.

So my question is, how do I permanently disable voice-lock-mode in all buffers and major modes?

Thanks