This study investigates the consequences of growing up with a single parent on children’s well-being in Japan. We focused on not only the father’s absence but also the mother’s. Moreover, we used a natural experimental approach; that is, we focused on single parenthood due to death and divorce, which enabled us to make causal inferences on the effect of single parents on children. Using data from the Comprehensive Survey of Living Conditions conducted in Japan in 2010 and 2013, we found that the father’s absence had a causal effect on children’s poor health, stress levels, mental illness, and not attending high school, while the mother’s absence had a causal effect on children’s smoking behavior. Overall, the former had a stronger causal impact on children’s well-being in Japan. Most of the observed effects of the mother’s absence were not causal but reflected the selection process of the single fatherhood pathway. In Japan and other countries, the numbers of single-mother families far exceed the numbers of single-father families, which reflects social norms that pressure mothers, but not fathers, to directly raise their children. Therefore, single fatherhood caused by divorce reflects a worst-case scenario of child-mother relationships because it suggests the mother’s lack of interest in parenting and/or the child, which is associated with children’s negative outcomes. We also found that the relationships between parental absence and children’s well-being were fundamentally homogeneous across child gender.