Caffeine, a widely consumed adenosine A(1) and A(2A) receptor antagonist, is valued as a psychostimulant, but it is also anxiogenic. An association between a variant within the ADORA2A gene (rs5751876) and caffeine-induced anxiety has been reported for individuals who habitually consume little caffeine. This study investigated whether this single nucleotide polymorphism (SNP) might also affect habitual caffeine intake, and whether habitual intake might moderate the anxiogenic effect of caffeine. Participants were 162 non-/low (NL) and 217 medium/high (MH) caffeine consumers. In a randomized, double-blind, parallel groups design they rated anxiety, alertness, and headache before and after 100 mg caffeine and again after another 150 mg caffeine given 90 min later, or after placebo on both occasions. Caffeine intake was prohibited for 16 h before the first dose of caffeine/placebo. Results showed greater susceptibility to caffeine-induced anxiety, but not lower habitual caffeine intake (indeed coffee intake was higher), in the rs5751876 TT genotype group, and a reduced anxiety response in MH vs NL participants irrespective of genotype. Apart from the almost completely linked ADORA2A SNP rs3761422, no other of eight ADORA2A and seven ADORA1 SNPs studied were found to be clearly associated with effects of caffeine on anxiety, alertness, or headache. Placebo administration in MH participants decreased alertness and increased headache. Caffeine did not increase alertness in NL participants. With frequent consumption, substantial tolerance develops to the anxiogenic effect of caffeine, even in genetically susceptible individuals, but no net benefit for alertness is gained, as caffeine abstinence reduces alertness and consumption merely returns it to baseline.