Out of curiosity, I looked up who actually beat Roger during 2004-07. Roger lost only 24 matches during that span, compared to 315 wins - an overall 93 winning %.
Those 24 losses were from:
Tomas Berdych: Athens 04 Olympics
Dominik Hrbaty: Cincinnati 04
Gustavo Kuerten: Roland Garros 04
Albert Costa: Rome 04
Rafael Nadal (8): Miami 04, Roland Garros 05, Roland Garros 06, Rome 06, Monte Carlo 06, Dubai 06, Roland Garros 07, Monte Carlo 07
Tim Henman: Rotterdam 04
David Nalbandian (3): Tennis Masters 05, Paris 07, Madrid 07
Richard Gasquet: Monte Carlo 05
Marat Safin: Australian Open 05
Andy Murray: Cincinnati 06
Novak Djokovic: Rogers 07
Filippo Volandri: Rome 07
Guillermo Canas (2): Miami 07, Indian Wells 07
So yeah, Nalbandian is one of only three players to beat Roger more than once during his best four years - Rafa did it 8 times and Canas twice.