Extended reality: the successor to Zoom and Teams

TNO’s Medialab in The Hague is buzzing with activity, with screens, desks, cameras, and cables in abundance. But put your VR glasses on, and you suddenly find yourself in a calm and ordered meeting room. A telephone is on the long table in front of you, around which are empty chairs. The ceiling tiles house attractive recessed lights, while to your left is a colourful wall that was not there just a few seconds ago. Suddenly, TNO’s Hans Stokking appears out of nowhere. How strange! A few minutes ago, he collected you at reception and was standing right next to you. Now, he is sitting at the virtual meeting table. ‘Can you see and hear me properly?’ he asks. Welcome to the world of extended reality!

Cutting an image

‘The drawback of Zoom and Teams is that everyone is in their own environment’, says Stokking, who is a senior scientist and closely involved with the technology of virtual reality. ‘It means you never feel as though you really are sitting alongside each other.’ Wearing VR glasses, you can ‘genuinely’ come together in a digital meeting room in which everyone appears to be sitting at the same table, even if they are not. This is known as ‘extended reality’.

‘With VR you can see the whole person and any related non-verbal communication’

Stokking explains: ‘We use a camera to take a picture of every person taking part in the discussion. We then cut out that image and put it in the digital meeting room. You can see each other, on the other side of the table or next to you. With Zoom and Teams, you see only other people’s heads and part of their upper body. With VR glasses, you can see the whole person at their actual size. That adds important non-verbal communication.’ The fact that we are conducting our conversation wearing VR glasses in a meeting room that actually only exists inside a computer proves how realistic extended reality is.

Watching football

Jan van Erp is principal scientist at TNO and one day a week professor in Twente. He wants to initiate new research programmes. ‘It is not our aim to imitate reality as effectively as possible’, he says. ‘TNO would like to carry out research into how people experience extended reality. What makes the biggest impression?’ Hans Stokking: ‘We just showed the example of a meeting room, but you can recreate any environment. Perhaps you’d prefer to have a conversation on the beach. In that case, we would put the people in that particular setting. And what would the sound of the waves add to the participants’ experience? For technologies of this kind, you have to look for the best experience at the lowest cost. Perfection is possible, but it is far too costly.’

‘The coronavirus crisis has moved extended reality right up the agenda’

Experiment in the care sector

The applications of extended reality are countless, such as in the care sector, for example. TNO will shortly be starting an experiment in a care home where, because of the coronavirus, visits are difficult. Jan van Erp: ‘The disadvantage of Zoom, Skype, and Facetime is that older people do not feel as though their family members are close by. We believe that extended reality can evoke that feeling much more. However, we do not want to make older people wear VR glasses. So therefore, we are putting an iPad on the table. We then project the image on the screen of a family member who is at home in front of the camera. We cut out this image and ‘paste’ it in the environment where the older relative is. The family member then appears to be sitting on a chair opposite him or her. The iPad functions as a type of window that you look through.’

Commercial applications

The coronavirus crisis has moved extended reality right up the agenda. ‘In the past, people would tell us what we were working on was far too futuristic’, says Sri Ganesan. ‘Since the coronavirus came on the scene, no-one is saying that anymore. Everyone now understands the opportunities that extended reality brings.’ At TNO, Ganesan focuses primarily on transport and logistics. The impact of extended reality on this area will be far-reaching. ‘I expect the first application to appear in the commercial world’, he says. ‘This technology will make digital meetings more realistic, thereby reducing the need for physical meetings to take place.’

‘Even concluding business deals, which often necessitates travel, is now possible thanks to extended reality. It’s important when doing business that you trust each other. Being able to look each other in the eye, for example, making genuine contact. Extended reality allows you to simulate this. We are moving towards a world in which the need to travel will diminish. Extended reality could result in huge savings in travel times, costs, and reductions in CO₂ emissions.’

‘We are experimenting with a glove with sensors’

Unwell

There is just one downside to the technology. Hans Stokking: ‘A small percentage of people feel unwell when using VR glasses. That is why we are monitoring the experiment in the care home with great interest. We are using iPads there, not VR glasses. However, we have to say that up to now, wearing VR glasses does produce the best effect.’

Mimicking real life as much as possible – that is the aim. Jan van Erp: ‘In communications, physical contact is very important. You shake each other’s hand, or give someone a pat on the shoulder. We are experimenting with a glove with sensors. This can be used to imitate the feeling of actual contact.

Virtual holidays

What does this mean for the distant future? Hans Stokking envisages a world where people no longer have to leave their front door. ‘In the future, I think we’ll be going on virtual holidays’, he says. This vision is shared by Sri Ganesan, although he does not believe this will happen for another sixty years – unlike Stokking, who expects it within about thirty years. Jan van Erp has even stronger reservations: ‘I think people will always have the need for genuine contact. It is clear that young people and students in particular – the generation that has grown up in a digital age more than anyone else – have been keen to maintain contact with each other, precisely during the period of the coronavirus.’