When it comes to iPhone X privacy, we’ve all been facing the wrong way. The advanced face scanner set to debut when the smartphone launches on Friday promises to ditch the fingerprint sensor in favor of something more advanced. But while some have cautioned about handing over scans of your face to your Apple device, the biggest issue around the scanner may come not from the phone’s maker but from a less expected source.

“The privacy issues around of the use of very sophisticated facial recognition technology for unlocking the phone have been overblown,” Jay Stanley, a senior policy analyst with the American Civil Liberties Union, told Reuters in a Thursday story. “The real privacy issues have to do with the access by third-party developers.”

For unlocking the phone, Apple stores a user’s face scans inside the phone on a secure enclave. That means it doesn’t leave the device, in the same way previous phones’ fingerprint scanner worked. But while the high-resolution scans will stay device-bound, there’s a possibility that app developers can use a more limited set of data.

Apple’s privacy guidelines restrict what app makers can and can’t do. They can capture a rough map of your face, along with 50 different kinds of facial expressions. They can then move this data onto an external server. It’s this aspect that has privacy advocates worried. While iOS will request a user’s permissions to use the front-facing camera, it’s the fact that developers can capture some face data and store it elsewhere that’s concerning.

The iPhone 8, iPhone X and iPhone 8 Plus.
The iPhone 8, iPhone X and iPhone 8 Plus.

The above also depends on Apple kicking out bad apps. When the App Store first launched in 2008, one of the highlighted benefits was the fact that Apple has sole control over what a user installs on their device, meaning the company can kick out rogue developers that don’t respect its rules. Bad apps have slipped through the cracks before, though, with Apple only removing fake virus scanning apps earlier this year.

As a method of unlocking a phone, Apple claims Face ID is secure, touting internal data that shows there’s a one in a million chance the phone unlocks when presented with the wrong person’s face. But other uses have technology commentators worried, and it may be some time before it becomes clear how app makers plan to use Face ID during apps.

“It doesn’t take much of a leap to imagine how the technology used to anthropomorphize emoji might appeal to advertisers that want to know where (and whether) you’re looking at the screen during their messages,” said Geoffrey Fowler, in his review for the Washington Post. “Apple’s terms for app developers require permission before tapping into the camera, and forbid using face data for advertising — but we’re just at the beginning for this technology.”

Photos via Getty Images / Justin Sullivan, Apple