Today, the sea ice cover is rapidly declining in the Arctic Ocean. While major changes in the Arctic marine ecosystems are starting to appear, the full implication of a changing Arctic ecosystem is not well understood. Here we show, using sedimentary ancient DNA metabarcoding to document marine eukaryotic communities, that higher diversity is recorded in a warmer, sea ice free Labrador Sea compared to when this region is sea ice covered. Changes in diversity and communities happen slower than the rapid climate warming of Greenland Interstadial 8 (ca. 38,000 years ago) and the Bølling period (ca. 14,700 years ago). Surface ocean conditions, rather than atmospheric warming, exert control on the marine eukaryote community and diversity. Our observations from the paleoclimate record suggest that when sea ice further decreases under global warming, the (sub-)polar marine ecosystems will change drastically, and feedbacks to the carbon cycle and climate system become very likely.