The 33-year-old American, who announced on Monday this would be his farewell event, had fallen to 100th in the rankings and was far from the level that saw him reach the last eight on the Flushing Meadows hardcourts in 2005 and 2006.
But having defied the odds by overcoming scoliosis as a child and Zoster, a condition that attacked his hearing and eyesight and nearly paralyzed his face, Blake called it quits on his own terms before homeland fans who inspired him.
"I don't know when it's going to hit me," Blake told the crowd at Louis Armstrong Stadium after the 79th-ranked Croatian beat him 6-7 (2/7), 3-6, 6-4, 7-6 (7/2), 7-6 (7/2).
"It's after midnight and you stayed to watch. I'm never going to have this again in my life and I need to appreicate every single one of you for being here. It will never be forgotten. I'll take in every memory I've ever had here."
The former Harlem youth who snuck into such US Open night sessions became an inspiration for younger African-American players.
"It has been a long road. I'm at the same venue but I'm not the same person," Blake said. "I got up to four in the world. I did everything I could."
Blake began his match Wednesday on the 50th anniversary of the famed "I Have a Dream" speech by Martin Luther Knig Jr., a landmark US civil rights moment.
"I thought about that a lot," Blake said. "I'm proud to be playing on such a historic day and it's a great reminder than we still have a ways to go."
"To have something that was a demonstration 50 years ago and still resonates today and was a beacon of what progressed the civil rights moment, I'm proud. But we're definitely not at the finish line."
Blake took advantage of his fourth break-point chance in the seventh game of the fifth set with a forehand winner and held to 4-4 as chants of "Here we go, James, here we go," erupted and both men then held into the tie-breaker.
The end came after three hours and 24 minutes when Karlovic blasted his 38th ace on match point, tying the third-most aces in any match in US Open history.
"Hopefully that's not going to be my lasting memory, up two sets to love and lost. That's frustrating. It's not a fun way to go out," Blake said.
"I'm definitely going to think about this match and how I could have won it but I'm going to think more about how I have the rest of my life to spend with the people I love the most."