Understanding how environmental variation drives phenotypic diversification within species is a major objective in evolutionary biology. The seaweed fly Coelopa frigida provides an excellent model for the study of genetically driven phenotypes because it carries an α/β inversion polymorphism that affects body size. Coelopa frigida inhabits highly variable beds of decomposing seaweed on the coast in Scandinavia thus providing a suitable test ground to investigate the genetic effects of substrate on both the frequency of the inversion (directional selection) and on the phenotype (genotype × environment effects). Here we use a reciprocal transplant experiment to test the effect of the α/β inversion on body size traits and development time across four suitable natural breeding substrates from the clinal distribution. We show that while development time is unaffected by G × E effects, both the frequency of the inversion and the relative phenotypic effects of the inversion on body size differ between population × substrate combinations. This indicates that the environment modulates the fitness as well as the phenotypic effects of the inversion karyotypes. It further suggests that the inversion may have accumulated qualitatively different mutations in different populations that interact with the environment. Together our results are consistent with the idea that the inversion in C. frigida likely evolves via a combination of local mutation, G × E effects, and differential fitness of inversion karyotypes in heterogeneous environments.