Immune responses are highly variable between individuals and populations, with this variance mediated by many factors including age, sex, and genetics. To dissect how each of these factors contributes to differential immune responses we recruited 1,000 healthy donors as part of the Milieu Intereiur cohort that were equally stratified by age (20-70 years old) and sex (50:50). Whole blood from each donor was stimulated in a standardized approach with Escherichia coli, BCG, Stapholocus aureus, SEB, Candida albicans and Influenza virus and the transcriptomic response was analyzed by Nanostring gene expression arrays. We show that age affects gene expression in a stimuli specific manner, while the effect of sex is more common across conditions. Both age and sex contribute to the expression of many immune-related genes but at a low level, while in contrast, the genetic effects are much stronger but relevant to fewer genes. Genetic analysis identified hundreds of eQTLs, regulated in both cis and trans, for all stimuli induced responses. We confirm the presence of a TLR1 master regulator recently shown to be an important factor controlling variable immunity in European populations. Finally we integrated transcriptomic and cellular data sets in an integrative model to describe how age and sex mediated their effects through different immune cell populations. These results lay the foundations for new patient stratification strategies that consider age, sex, and genetics as impacting variable immune response outcomes.