How do you know that Ham was the youngest son?
"So Noah awoke from his wine, and knew what his younger son had done to him" (Genesis 9:24).
The word "younger," or "youngest" depending on your translation, comes from the Hebrew word qatan. It is an adjective which can be translated as "small," "insignificant," or "young." Typically it is used to compare two people's ages and identifies which one is younger. When used in a comparison of more than two items, it functions as a superlative to identify the youngest (or smallest, or least significant) of all being compared.
Since there are three sons, the word qatan means Ham is the youngest of the three. A similar example is found in I Samuel 16:11 and 17:14 where David is identified as the youngest of Jesse's sons.
This does cause some a problem because it is typically understood that children are listed in birth order, yet the three sons are always listed as "Shem, Ham, and Japheth." One possibility is that the three sons were triplets since they were born when Noah was 500 years old (Genesis 5:32).
What is also interesting is that the Bible states that Japheth was the eldest brother. "And children were born also to Shem, the father of all the children of Eber, the brother of Japheth the elder" (Genesis 10:21). In both Genesis 10:1-2 and I Chronicles 1:4-5, after listing the three sons, it is Japheth who is broken down next to the following generations; another indication that he is the eldest. But in those lists, Ham's descendants are listed second and Shem's third.
Therefore, we must conclude that while the Bible tends to list children in birth order, it is not a hard-fast rule which can always be counted upon.