It is very imprtant that both the transport ship and the actual fleet have orders to capture planets. I knew this, yet still forgot to make sure the troop ship had the orders, the result was the whole planet got glassed and my small fleet wasted fire on the planet while fighters destroyed part of it.