Over the past two decades, the dendroclimate community has produced various annually resolved, warm season temperature reconstructions for the extratropical Northern Hemisphere. Here we compare these tree-ring based reconstructions back to 831 CE and present a set of basic metrics to provide guidance for non-specialists on their interpretation and use. We specifically draw attention to (i) the imbalance between (numerous) short and (few) long site chronologies incorporated into the hemispheric means, (ii) the beneficial effects of including maximum latewood density chronologies in the recently published reconstructions, (iii) a decrease in reconstruction covariance prior to 1400 CE, and (iv) the varying amplitudes and trends of reconstructed temperatures over the past 1100 years. Whereas the reconstructions agree on several important features, such as warmth during medieval times and cooler temperatures in the 17th and 19th centuries, they still exhibit substantial differences during 13th and 14th centuries. We caution users who might consider combining the reconstructions through simple averaging that all reconstructions share some of the same underlying tree-ring data, and provide four recommendations to guide future efforts to better understand past millennium temperature variability.