it's a never ending argument. The fact is that you will never know when Fed's prime really was. You say it was 04-07 but that is very convenient because this is when he was at his most dominant, you don't really take any other factors into account. As soon as he started losing in 2008 (11 years ago at the ripe age of 26/27) he was past his prime. Isn't it utterly ridiculous to claim Federer was past his prime by 08? in the prime of of his physical prowess? Now why is it that Fed fans said this? because he started losing in slams. This requires a more in depth analysis though. Federer's best years were indeed 04-07 but we absolutely need to take into account his competition, whether you like it or not. Nadal came into the scene in 2005 but he initially was very poor on grass and hardcourts, he was getting wasted by nobodies at USO and AO. He was improving his grass court game every year, making finals of W in 06, then 07 and then 08, each time improving his results. Nadal started to play well on hards after 09, when he made his first AO final. He won all of his 4 HC slams after 09, which makes sense as by 09 he was in early 20s... so was reaching his prime as he approached mid 20s. Djokovic started making his mark at 19, in 2007. So what happened in 2008 exactly? when Roger, all of sudden, got OLD at 26? Djokovic beat him at AO, Nadal at FO and Wimbledon and then Roger won USO. Now go back to 2005, extract Roddick and insert Andy in AO semis (instead of Djokovic). Also go back and extract 05 Nadal (who was 19) and put him in 08 Wimbledon. Guess what? 99% chance Federer would've won 08 AO, 08 Wimbledon, made FO finals and won USO. You think not? you know it. There is no way Federer suddenly went old in his mid 20s especially considering that at 36 he won a slam. It is true that around 2012-2017 Federer was past his prime but still playing at a high level in spurts. It is true that Djokovic's prime 11-18 didn't coincide with Federer's prime (i would argue 04-10, not just the convenient 04-07 which is preposterous) but just as past his prime fed beat prime djoker on occasion, baby joker was beating prime fed in 07-08. It goes both ways, we can both find examples that suit our arguments. What is clear though is that, without a doubt, having to face a slightly past his prime Fed and prime Nadal is HARDER than facing the likes of Hewitt, Roddick and little baby Nadal between 04-07. I would argue Djokovic has had it toughest between the 3 of them and Djoker got the added bonus of Andy Murray too.
I do think there is a compelling argument to be made that Djokovic is best ever. He has 15 slams and i think has the most complete game of the 3. Several people have called his highest level, highest level ever achieved, including Nick Bolletieri (there are many others i forget at the moment). He has a winning record against Fed and Nadal (no-one else has) and only one that could win all masters titles and 4 slams in a row, probably most impressive feats after # of slams. He will inch closer to Fed but i think having most # slams is not the only factor when assessing best ever.