It's not clear if other smartphones and tablet computers are logging such information on their users. And this week's revelation that the Apple devices do wasn't even new - some security experts began warning about the issue a year ago.
But the worry prompted by a report from researchers Alasdair Allan and Pete Warden at a technology conference in Santa Clara, California, raises questions about how much privacy you implicitly surrender by carrying around a smartphone and the responsibility of the smartphone makers to protect sensitive data that flows through their devices.
Much of the concern about the iPhone and iPad tracking stems from the fact the computers are logging users' physical coordinates without users knowing it - and that that information is then stored in an unencrypted form that would be easy for a hacker or a suspicious spouse or a law enforcement officer to find without a warrant.
Researchers emphasize that there's no evidence that Apple itself has access to this data. The data apparently stays on the device itself, and computers the data is backed up to.
Apple didn't immediately respond to a request for comment by The Associated Press.
Tracking is a normal part of owning a cellphone. What's done with that data, though, is where the controversy lies.
A central question in this controversy is whether a smartphone should act merely as a conduit of location data to service providers and approved applications - or as a more active participant by storing the data itself, to make location-based applications run more smoothly or help better target mobile ads or any number of other uses.
Location data is some of the most valuable information a mobile phone can provide, since it can tell advertisers not only where someone's been, but also where they might be going - and what they might be inclined to buy when they get there.
Allan and Warden said the location coordinates and time stamps in the Apple devices aren't always exact, but appear in a file that typically contains about a year's worth of data that when taken together provide a detailed view of users' travels.
"We're not sure why Apple is gathering this data, but it's clearly intentional, as the database is being restored across backups, and even device migrations," they wrote in a blog posting announcing the research.
Allan said in an email to the AP that he and Warden haven't looked at how other smartphones behave in this regard, but added there's suspicion that phones that run Google Inc's Android software might behave in a similar way and is being investigated.
Google did not immediately respond to a request for comment.
Alex Levinson, a security expert, said the tracking Apple's devices do isn't new - or a surprise to those in the computer forensics community.
The Apple devices have been retaining the information for some time, but it was kept in a different form until the release of the iOS 4 operating software last year, Levinson, technical lead for the Katana Forensics firm, wrote on his blog.
Through his work with law enforcement agencies, Levinson said he was able to access the location data in older iPhones and warned about the issue over a year ago.
The location data is now easier to find because of a change in the way iPhone applications access the data, he said.
"Either way, it is not secret, malicious, or hidden," Levinson wrote.
"Users still have to approve location access to any application and have the ability to instantly turn off location services to applications inside the settings menu on their device."
The existence of the location-data file on the phone is alarming because it's unencrypted, the researchers said, which means that anyone with access to the device can see it.
Charlie Miller, a prominent iPhone hacker, said a security change that Apple made last month would make extracting the file from the phone in a remote attack very difficult.
Even if an attacker were to break into someone's phone looking for the file, he wouldn't have the right privileges to access the file.
The data is "pretty well-protected on the phone," Miller, principal security analyst with Independent Security Evaluators, said in an interview.
"On the phone, they take a lot of precautions." He said.
"It's sort of frightening in the sense that it's there, and it's full of information about where you've been, but the good news is it's not easy to get to."
But it's a different matter when the data is transferred to another computer in a backup. If the backup computer is infected with malicious software, the file could easily be located and sent to the hacker.
A way to protect against that is to encrypt the iPhone backup through iTunes, the researchers said.
The issue has prompted several members of Congress to write letters to Apple, based in Cupertino, California, to answer questions about the practice.
Rep Edward Markey, a Massachusetts Democrat, questioned whether the practice may be illegal under a federal law governing the use of location information for commercial purposes, if consumers weren't properly informed.
"Apple needs to safeguard the personal location information of its users to ensure that an iPhone doesn't become an iTrack," he said in a statement.
"Collecting, storing and disclosing a consumer's location for commercial purposes without their express permission is unacceptable and would violate current law."
Apple shares rose $9.20, or 2.7%, to $351.71 on the strength of the company's latest quarterly financial results, which showed Apple's net income nearly doubled, in large part on strength of iPhone sales.