According to this article, the number of date palms in Morocco has declined from 15 million at the end of the 19th century to 4.5 million now, mainly due to desertification. That has to have had some effect on genetic diversity, and I’m willing to bet there are data out there on the numbers of varieties at different times in the past.
Not just desertification; disease has played a part too.
There are numbers, and an IPGRI-led project was gathering them, but I can’t find a publication. The project’s home page is here, but it does not seem to be very up-to-date (hahaha).