For matrices drawn from the standard Gaussian orthogonal ensemble (GOE) and Gaussian unitary ensemble (GUE), we prove that the normalized zero counting measure of the permanental characteristic polynomial $Per(zI_N-H_N)$ converges almost surely to the standard Wigner semicircle law on $[-2,2]$, rotated by $π/2$ onto the imaginary axis. This proves the conjecture proposed by Fyodorov in \cite{Fyodorov2006}.