Roger's 2004-07 record is pretty astonishing:
Overall W-L: 315-24 (92.9%)
Hards: 199-13 (93.9%)
Grass: 42-0 (100%)
Clay: 63-10 (86.3%)
Carpet: 11-1 (91.7%)
His 24 losses were to:
8 Nadal
3 Nalbandian
2 Canas
1 Djokovic, Gonzales, Volandri, Murray, Gasquet, Safin, Berdych, Hrbaty, Kuerten, Costa, Henman
9 of the 24 losses came in 2007, so he lost only 15 matches in 2004-06, with a 94.3%. 2007 was also when he lost to some of the worst players on that list: Gonzalez, Volandri, twice to Canas. But in 2004-06, he lost only five matches to players who were never #1.
The weird thing about Roger's 2006 season wasn't the 4 losses to Rafa, but the loss to Andy Murray. Rafa was understandable: he was easily the 2nd best player on tour, and had Roger's number - especially on clay, and three of their matches were in clay finals. But Andy was 19 and unseeded at Cincinnati, ranked #31 at the time, and beat Roger in two sets. I guess that's the fun of best-of-threes, and Roger must have had an off day and/or struggled with Andy's defense. Probably overly cocky, too.