Because the cost to things that allow the whale to survive is greater than the benefit of being larger, on the whole. If they get bigger, they die and are survived by slightly smaller versions, and over a long period of time, you get a range of slightly bigger and slightly larger whales, centered around whatever average size best fits the niche for that particular time span.

It's usually a whole suite of interrelated reasons that are affecting any given feature - size would include hearts, nervous system performance, food intake, protection from predation, and so on, with dozens of factors playing a part.