The 46-year-old American started the fight brightly, moving around the ring to neutralise Valuev's long reach advantage. The 2.14m Russian, the tallest and heaviest champion ever, struggled to close down Holyfield but began asserting his jab as the fight wore on.
One judged scored the bout a draw, while the others had Valuev winning 116-112 and 115-114.
"Of course I am disappointed," Holyfield said. "I thought I had done enough to get the win. Now I have to go home and think about my future."
Holyfield (42-10-2) was attempting to become the oldest heavyweight champion. He had not fought since losing a one-sided decision to then-WBO champ Sultan Ibragimov more than a year ago.
Holyfield was criticised for this latest comeback attempt, but the "Real Deal" looked in great shape and was never seriously hurt by Valuev (51-1), who has avoided the top heavyweights and did little to improve his standing in the division. He was vulnerable to Holyfield's right hooks, even if the Russian also was never stunned.
Holyfield was the busier fighter from the start, and his footwork did much to reduce Valuev's few punches to little effect at Hallenstadion.
The Russian carried a weight advantage of almost 100 pounds (45 kilograms), and planted himself in the center of the ring, failing often to cut off Holyfield or unleash punches of power.
For Valuev, it was the biggest win on a resume that includes no fights against the Klitschko brothers, who hold the other major heavyweight belts.
Holyfield had refused to say if he would retire should he lose or draw against Valuev, who was the bookmakers' overwhelming favorite. Struggling with financial problems, Holyfield may try to use the better-than-expected result to score a big payday against another champion or Riddick Bowe, who has made a comeback.
Holyfield won one of three epic battles against Bowe in the 1990s, and beat James "Buster" Douglas, George Foreman and Mike Tyson twice to become a dominating figure in boxing.
The fight was in front of 12,500 largely pro-Holyfield fans, who gave his entrance a standing ovation. He wore a red robe fit for a king. Valuev was booed, and there were even scattered whistles while the Russian national anthem played.
The fans similarly showed their displeasure when the judges' scorecards were read out.
