Summoning Void Spectres

[Loren]

Is there some limit on the number of them that can be summoned? I've got some, I want more and nothing happens. Gobs of gems.

Do I have to get starspawn killed in order to be able to summon them??