Khao Lak has 3 seasons : low, high, and peak season.
Some people like low season as it is quieter and traffic is greatly reduced - and hotel/resort prices are better. Some people may be put off by the rougher seas and and it is not always safe to swim as the sea reacts to the weather. Swimming for children in particular is probably not safe during this season. Be safe, ask hotel staff if you are not sure about swimming in the sea.
November to April is considered high season in Khao Lak as western people escape their harsh winters and flock to nicer climates. If you are a people watcher, it is also a fun time to just sip a beer and watch people walk by. This period of the year is also when the sea is calmer and the water much more clear - so perhaps better for scuba diving, snorkeling and swimming in the sea.
Peak season (26 Dec - 10 Jan) is normally when Khao Lak is busiest, as it is long holiday for most people and the weather is so nice.
In truth, there is no bad time to visit Khao Lak - you'll not likely be disappointed.