The fact that Apple is working on a mixed reality device (AR/VR) is probably the most 'widely known' secret in the tech industry, and every once in a while, news about it always leaks out.
It's believed that this will be the next generation of personal devices that Apple will use to replace the iPhone, so instead of rubbing your fingers on the glass, people will be "pointing" at the air.
Yet all these years later, the iPhone has evolved from the iPhone 8 to the iPhone 14, and this next-generation terminal is still "under the radar", always living in rumors.
Just when people were about to lose their sense of freshness about mixed reality, news about the mysterious device has been picking up again recently.
More and more news suggests that Apple's mixed reality product could be hiding somewhere in a secretive lab in Cupertino and is making preparations to meet the public.
Parker Ortolani, who guessed the macOS Monterey name based on trademark registration information last year, recently discovered that Apple has recently been using shell companies to register trademark applications for RealityOS around the world.
This is a common secrecy tactic before Apple launches, as Apple has previously used a shell company called "Yosemite Research LLC" to register trademarks for Yosemite, BigSur, and Montery (all version names for macOS).
Parker Ortolani discovered that "Realityo Systems LLC", a company with the same registered address as "Yosemite Research LLC", has filed trademarks for RealityOS in Brazil, Uruguay and other countries.
Combined with the information previously found by developers in the dev logs related to RealityOS, Parker Ortolani believes there's reason to believe that this is the operating system that will soon be used on Apple's mixed reality devices, and while it's unlikely to be unveiled at WWDC in a few days, it at least means we're one step closer to unveiling it.
Just from the RealityOS naming, the OS should focus on connecting virtual and real interactions, and one might be able to apply the logic of real-world interactions to the virtual world, meaning Apple could potentially bring something new to the VR headset form factor we're used to.
So what exactly does this mysterious device look like? We may have some clues from a recent report published by The Information about an internal Apple development debate.
Moving forward in the mire
According to The Information, there's been a lot of debate within Apple about how to break down the barrier between virtual and reality.
Back in 2016, Apple's VR/AR team had enlisted the support of Apple's board members with immersive VR experiences to begin work on a related product.
It was an innovation that started from scratch and was described by some insiders as "Apple's boldest new product innovation since the iPhone," but it was a much bumpier ride than the iPhone in terms of progress.
▲ Image from: Time
Some people familiar with the matter have revealed that while Apple's current CEO Cook has been supporting the project, he hasn't paid quite enough attention to the new product compared to Steve Jobs' deep involvement with the iPhone project in the past, making it difficult for the project team to fight for development resources with the Mac and iPhone teams.
Apple's investment in the AR/VR space began with the acquisition of German startup Metaio in 2015 and the hiring of executive Mike Rockwell around the same time.
Metaio was developed from a project within Volkswagen, which Apple acquired with the intention of using Metaio's AR technology to develop autonomous driving systems, and then Facebook made a major acquisition of Oculus VR, which started a virtual reality boom in Silicon Valley, before Apple started using Metaio to build an AR/VR project team.
▲ Mike Rockwell
Leading the hardware team is newcomer Mike Rockwell, previously EVP of Dolby Laboratories, who is an unqualified expert in the field of visual and hi-fi audio.
To build a piece of hardware from scratch, Mike Rockwell needed to enlist the support of other hardware departments, such as the touch sensor, camera, and Apple chip and display teams.
In fact, the development of Apple products, including the iPhone and iPad, is done collaboratively by these departments responsible for each function, rather than being divided into "iPhone groups" and "Mac groups," and it's easy to see why Apple's different products are linked together in a way that surpasses their rivals.
Mike Rockwell's biggest challenge was getting support from two of the strongest internal departments - Apple's industrial design team and the human interface team, led by Jony Ive, the most vocal person at Apple in the post-Jobs era.
Mike Rockwell's team's vision of a virtual reality headset was quickly rejected by Jony Ive's team, who argued that a helmet-shaped product form cut people off from their external environment, alienated them from each other, and didn't look 'hip' from the outside.
▲ The figure shows Facebook's extranet technology, for reference only
To get the industrial designer team to buy in, Mike Rockwell's team had thought of a number of improvements, such as adding an external camera to allow the wearer to see the outside world as well, but in the end Jony Ive's team opted for another option - installing an externally facing screen on the device that would allow someone from the outside world The screen allows the wearer's eyes and expressions to be seen from the outside.
This may go some way to reducing the wearer's detachment from the environment, or maybe it will just make the wearer look even weirder, but either way, the Mike Rockwell team has the support of the industrial design team to keep this product from being stillborn.
Push back and start again
With the development hurdles cleared, Mike Rockwell's team still needed to answer the next question - what should the product look like?
Mike Rockwell believes that Apple should develop a powerful headset that differentiates itself from rivals like Oculus, with superior longevity and display performance, so it would need a powerful external "base station" device, much like a powerful external PC was needed in the early days of VR.
According to Bloomberg, the chip used in this 'base station' device is most likely the same M1 Ultra that was used in Mac Studio a while back, and its processing power is indeed enough to deliver a high-quality virtual experience that would dwarf any kind of VR all-in-one.
▲ Designer's own concept artwork Image from: AntonioDeRosa
To Mike Rockwell's surprise, however, Apple's board of directors preferred Jony Ive's solution: an all-in-one product that doesn't require a base station, but is smaller and more portable, although the quality of the content displayed is much less than the former.
The team, led by Jony Ive, has been pushing for a standalone version of the headset since the beginning of the project, and Jony Ive felt that getting users to want to wear the product for a long time was the primary problem to solve with its industrial design, so he was adamant about getting the device out of the way of the base station.
▲ The difference in the picture is probably so
At the executive presentation, executives, including Cook, compared the lifelike virtual graphics of Mike Rockwell's team's base station kit to the cartoon-like animations of Jony Ive's team, and voted for the lighter version of the latter.
This hit Mike Rockwell's team hard, not only because the product prototype they had spent so much time on was rejected, but also because the department, including the chip team, had already prepared the custom parts they needed, and it was clear that this could not be pushed back and started again.
▲ Apple's chip star Johny Srouji
According to information leaked to The Information by people familiar with the matter, Apple's chip team, led by Johny Srouji, has an image processing chip codenamed Bora and a wireless chip codenamed Golay ready for the headset to drive the internal display and the data transfer between the device and the base station.
The other chip, codenamed Jade, is a computing chip designed specifically for the base station, which is now known as the M1 Ultra.
Now with the abandonment of the base station solution, it means that all sorts of complex components have to be condensed into the body of the headset, which can be a huge challenge for the device's battery life, computing performance, and heat dissipation.
If the product overheats, then the previously designed algorithms for hand tracking, computer vision, and 3D modeling will be much less effective, so both hardware and software will have to undergo a major overhaul.
This is one of the major reasons why Apple's headset has been delayed time and time again.
After Jony Ive's departure, he continued to be involved in the design and development of this product as a consultant, and some of Jony Ive's insistence on the product has made it increasingly complex.
▲ Magic Leap
For example, Jony Ive wasn't happy with the prototype's battery-mounted design on the device, preferring a structure like Magic Leap's where the battery is mounted on the "legs of the glasses"; to give the user enough contact with the outside world, the glasses are equipped with 14 cameras to track the user's facial expressions, and the Bora chip, made by the chip team, is designed to process this complex image information.
These designs brought with them a number of problems, and because of the latency between the Bora chip and the main processor, codenamed Staten, the wearer could easily become nauseous and uncomfortable, and eventually the chip team needed to develop a streaming encoder to alleviate this problem to some extent.
▲ Designer's own concept artwork Image from: AntonioDeRosa
Overall, Mike Rockwell's team still has a lot to work out, both in terms of software and hardware, but according to the latest news from Bloomberg, the mystery device has been shown to executives at an Apple board meeting and the product as a whole is in accelerated development.
After going around in circles, Apple's internal development footprint is finally close to unanimous, with "Apple's leaks professionals" including Bloomberg and Ming-Chi Kuo suggesting the product will launch as soon as late this year or in 2023.
Give a little more patience
Although the mystery device couldn't land for a long time, we've actually been able to see the fruit from this great tree in recent years with new Apple products.
In addition to the aforementioned M1 Ultra chip, The Information the Staten processor on this device will likely debut on devices like the new Macbook Air and iPad, the M2 processor that was previously speculated.
For now, combining virtual and reality is still the future Apple believes in, and Ming-Chi Kuo believes Apple will release an AR device based on the smart glasses form factor in 2025 after releasing the headset device it is currently working on, and will release an AR device in the form of contact lenses in 2030, when smart computing will enter the "age of invisibility".
▲ Designer's own concept artwork Image from: AntonioDeRosa
Of course, all of this will only make sense once people wear this 'iPhone on their face', otherwise it will still be the next generation iPhone that will beat the iPhone.