Whittle-Matérn (WM) Gaussian random fields (GRFs) are defined as solutions of stochastic partial differential equations (SPDEs) and provide a natural analog of Matérn GRFs on non-Euclidean geometry where the Matérn covariance function is not valid. In particular, WM GRFs on metric graphs have been an active area of research motivated by road and river networks where spatial dependence is more naturally described by intrinsic distances in the network than by Euclidean distances. This family of GRFs is controlled by three parameters relating to marginal variance, spatial range, and smoothness, but can be extended to so-called generalized WM GRFs through spatially varying coefficients in the SPDE. Recent work has considered the use of spatially varying covariates, but the full possibilities of flexibility have not been considered. In this work, we introduce latent GRFs that describe the spatially varying coefficients of the SPDE. This flexible model is compared to less flexible models in a simulation study evaluating both the ability to estimate the covariance structure and predictive ability. An important focus is the number of observations and replications necessary to reliably recover the covariance structure. We find that the flexible model improves over less flexible models in the presence of sufficient data. We also demonstrate practical applicability on traffic counts in a part of Madrid, and observe major differences between in-sample and out-of-sample predictive abilities of the models compared.