Well, for the sake of conversation, I'm willing to hold that he was fluent but with a small vocabulary.
I think, like a lot of things, we try to overload one term with meanings that are best described with other terms. We say fluent when we mean fluent and large vocabulary and good grammar and perhaps "thinking in the language" and maybe other things. But fluidity of speech (which, again, I am holding the side of the conversation that says it's mostly the physical skill with the language), is an independent quality. The quantity of those other variables does not directly affect the quantity of it.
I have a large vocabulary in English, but in person I tend to stammer, hesitate, search for words, and, as in my writing, I often get lost in parenthetical remarks. But some folks with a tenth of my vocabulary can talk circles around me. I'm going to say that they are more fluent.