Construction speed and population size

[Nitram Draw]

What I don't like about the population modifiers is that they are almost exclusively positive. There isn't much penalty for to few or to many people.
46 modifiers seems to be a lot and could provide the needed variety. Maybe a planet at max population gets a production increase and a happiness decrease for example.
The population modifiers should be tied to a planets max population capacity tho I don't think this is the case right now.