I've said this before, but I think you could argue that Novak is (arguably) the GOAT because of Roger and Rafa, not despite them. He played third-fiddle for a bunch of years, which drove him to work even harder. Meaning, finishing #3 three years in a row probably makes one very hungry to reach #1.
Similarly, part of Roger's problem later on is that he wasn't really challenged during his peak years, except for Rafa on clay and the odd match here and there. He was, in a way, like someone who grew up wealthy or naturally smart, so didn't have to work to catch up. So it created a kind of "coasting" quality that didn't withstand the rising tides of Rafa and Novak.
Furthermore, something happened in his game starting in 2007 and more evident in 2008 that had nothing to do with playing against Rafa (or Novak). In 2007, while he still won three Slams and had a great year, he started losing to lesser players - it is the least dominant, on a match-by-match basis, of those four peak years (2004-07).
For instance, here are his losses in 2006 vs. 2007:
2006: Nadal x4, Murray
2007: Canas x2, Nadal x2, Volandri, Djokovic, Nalbandian x2, Gonzalez
Presumably he wouldn't have lost those matches to Volandri, Canas and Gonzalez in 2006, and probably not Djokovic, and maybe not Nalbandian. I wonder if something diminished in his game, or if it was more psychological.