Let's see...
*Rafa is the greatest player in tennis history on a specific surface type.
*Off that surface, he's still great but more of a "lesser great." In other words, if you look at Rafa as a non-clay player, he's probably about as good as someone like Becker or Edberg were overall.
I think the sum total of that still makes him one of the greatest players in history, and in the "herd of GOATs," if not one of the front-runners. As you say, every great player has their kryptonite, and every great record has one or two holes. As far as GOAT candidacy goes, the holes I see are:
*Roger: None really, except for maybe the H2H vs Rafa and, if we want to be picky, Olympic gold.
*Novak: Slam count, and that's pretty much it. Olympic gold, I guess - but it seems like less of a lack for Novak, perhaps because with Roger it is the only thing missing, so stands out more.
*Rafa: More weeks at #1, World Tour Finals, maybe another non-clay Slam or two.
Pete had his lack of a clay Slam and generally weak record on clay, Borg retired early and never won USO. Laver's was quite complete, except for the fact that most of it was before the Open Era and difficult to compare.
Rafa's holes are similar to Pete's, and perhaps more difficult to fill than Novak's - although Novak would need to resurge to fill his, obviously. For Rafa to be on equal footing with Roger as far as career records go, assuming Roger is done with major accomplishments, I think Rafa needs to get to 200+ weeks AND win at least one WTF AND another 2-3 Slams. If he does all of that, and Roger doesn't do anything more than win another Masters or two and a handful of minor titles, then I think their records will be comparable.