Yes, you are correct. If you are only selecting out a small subset the difference becomes much smaller.
When you say this could be slow, doesn't the SQL engine first evaluate the ID = 1 to limit the subset of records before processing the 30-40 selected records ?
The optimizer can simply better utilize it's data structure if it is simply grouping values over keyed fields. Our business groups like to try and summarize 96M active service records from 135M all service file down to a single record for each of the 5M active customers from 15M all customer file. So it is important for us to remove all excessive processing cycles.