Augmented reality (AR) is a promising new technology that has applications in domains ranging from ecommerce to education and gaming. However, AR also comes with relatively complex interactions as it is trying to realistically combine a virtual scene or artifact with the user’s physical space.

Our Research

To understand the user experience of mobile apps that use augmented reality, we ran a usability-testing study involving 11 participants; 7 sessions were held in person and 4 were remote. This study included a diverse group of applications from areas such as education, health, fitness, art, history, ecommerce, and entertainment. The goal of the study was to see if any progress had been made since our last AR study involving ecommerce apps and to investigate how AR is used in other domains beyond ecommerce.

AR Still Unfamiliar to Most Users

Many people still have no idea what augmented reality is:  most of our participants did not know much about AR or were not at all familiar with the term. When asked “How would you describe the AR technology to someone who might not know anything about it?” eight participants confused AR with virtual reality. For example, one participant stated: “I have seen AR in demos at the store, where you put the head set on, but I don’t have anything like that in my possession. So, it [my experience with AR] has been pretty limited.” Three participants had used AR before once and just one participant had used it twice.

AR Onboarding: Tips and Walkthroughs

While extensive help and tutorials are generally not needed for mobile apps, some onboarding can increase users’ awareness of new, unfamiliar technologies such as AR. AR presents users with novel interaction patterns, many of which are still lacking in both usability and functionality. Hence, many AR experiences will benefit from some form of education or onboarding.

There are two types of instructional content commonly used in mobile apps:

Contextually relevant tips are brief instructions or hints that are related to the user’s goal. They are generally the more effective method of providing help — usually, because users are motivated to pay attention to them. AR-related tips can highlight key parts of the UI, tell people how to interact with the AR object, alert them of additional options, or give them feedback about how to correct certain actions (e.g., during calibration).

Instructional walkthroughs are generally more complex and teach users about how to interact with the application. While their use in mobile apps is not generally recommended (since most mobile apps are fairly simple and don’t need explanations of the UI), instructional walkthroughs do have a place in AR-based applications due to the novelty of AR and AR-related patterns. Thus, AR walkthroughs can provide step-by-step onboarding and teach users the key components and functionality for completing the task by guiding them through a series of actions in an AR experience.

While both tips and walkthroughs can successfully assist novice users throughout an AR experience, this article focuses on onboarding walkthroughs.

AR-Related Walkthroughs

Walkthroughs are not a substitute for the poor interface design. AR-related instructional walkthroughs should generally have two goals:

• Increase awareness of the AR feature and show its potential.
• Guide users through the novel interactions of a simple AR experience so they can learn how to use the AR feature.

If the primary functionality of the app is AR-related, the app may combine both these goals into a single walkthrough that is presented upon launch, If, however, AR is just a supporting feature for your app, don’t flood the user with AR instructions when the app is first launched. Often, users tend to dismiss lengthy or complex walkthroughs that are shown when they first use an app, especially if they consider them irrelevant for their immediate goals. Instead, show an interactive, instructional walkthrough when the AR feature is first used. Awareness of the AR technology can be created through a tip on those pages that include the AR feature.

In our study, the most used type of walkthrough was the deck of cards. While this onboarding style provided an overview of the AR features and set users’ expectations for the feature, it often caused cognitive overload and strained the user’s working memory. Participants couldn’t fully remember what they had read and had to refer to the help menu or relaunch the experience to review the deck of cards one more time.

Interactive walkthroughs were generally more effective than static walkthroughs, as they gave users a chance to practice a small-scale AR activity. Participants were generally appreciative when they could see detailed instructions about how to work with the AR feature and felt delighted when they were able to easily complete a first AR task within the app.

One participant using the JigSpace app was pleased with the detailed instructions that guided her through her first AR interaction within the app: “It’s easier than most apps, because it actually gives you  [instructions] at the bottom of the screen that actually tell you what you can do and what you can’t do. Tells you ‘just swipe it,’ ‘spin it’ — pretty straight forward.”

AR-Walkthrough Components

In addition to highlighting the core functionality of the UI, being aware of some of the main challenges that users face in their first interactions with AR features can help app designers set the stage for a successful AR experience. In particular, an AR walkthrough should be brief and touch upon the following key considerations for getting started with the AR feature:

• What to expect from the AR experience
• How to handle the device
• How to prep the environment.

What to Expect from the AR Experience?

When landing on the homepage of an app, many users had no idea about how AR was used. Sharing an overview of the AR experience and its goal clearly and in a descriptive way will help the users prepare for the next steps.

Focus first on the high-level purpose of the AR feature (e.g., see how a piece of furniture will look in your room, create interactive 3D presentations, drive a Mars Rover in your own environment) and, once that is clear, present users with low-level instructions such as how to use the specific controls in the interface.

A participant trying a game in the ARLOOPA app could not understand the goal of the game and what she should expect from the AR experience. She stated: “I don’t know how to play it. I’m just going to try a different game. I did not understand that one. […] So, I push Help because I don’t know how to do it.”

Unfortunately, using the Help menu did not resolve the issue, as it provided low-granularity information about interacting with the AR object and did not mention the goal of the game. The participant said: I don’t understand how to play this game at all. This is where I get frustrated.” She added: “It would have been helpful to get a better description of ideas or directions, I guess. This seems really weird. Even though I don’t understand how to play the game, I am inside the game because it’s so zoomed in, but it doesn’t help because you don’t even know what you’re doing. So, this needs to be described better.”

The Mission to Mars app showed an initial walkthrough with some basic instructions about interacting with the AR object. However, the objective of the AR experience was unclear. Hence, the participant later struggled to understand the hints and tips as she was using the app. Clarifying the purpose of the AR experience ahead of time could have helped her see the bigger picture and resolved potential confusion. She said:  “I don’t know what ‘follow the arrow to the nearest circle’ means. It seems like, it’s supposed to be a game. Cause it has a green thing that says like health, which I associate with game and staying alive. […] Oh, what’s that? Oh, I can drive it. […] So, there’s really no direction as to what I’m doing. I didn’t know I could drive it, but I sort of figured it out with this white circle thing. I’m backing it up. I don’t know. It seems sort of awkward, but actually it’s cool. […] I didn’t know what the Mars Rover looked like. Oh, I guess I was supposed to navigate this guy to this green area.”

How to Hold the Mobile Device

Due to the participants’ unfamiliarity with the AR technology, they often were unaware that the device’s camera would be used to take advantage of the AR feature. For instance, upon launching BBC’s Civilizations AR app, a participant was confused when she received instructions in text format but could not see any other visuals on the screen. Initially, she assumed that the AR feature had failed; it took her a couple of minutes to realize that the problem occurred because she had placed the phone on the table and the desk was blocking the camera view. She said: “Well, ‘cause I had my phone on the table, the camera was on already. So, it was already going. I didn’t really realize that. So, I thought it was just black and I was waiting for the tutorial to tell me what to do. So now I want to go back in the tutorial and find out what it was telling me. I don’t want to skip the tutorial.”

Let users know how they should hold their phone: whether they should keep it in their hands, at a specific height, angle, or orientation (portrait or landscape). Inform users that the camera should not be covered and should be directed towards an empty area. This information will prevent mistakes such as placing the phone on the table or leaning it against the wall.

Furthermore, the handling of the device should take into account physical limitations and should not conflict with accomplishing the primary task. For instance, participants struggled with drawing in the SketchAR app. The app “projected” an image onto a piece of paper and the user could trace the lines in that image to make their own drawing. To draw the selected image, the user had to hold the phone very still with one hand above the paper and draw on the paper with the other hand, while looking at the screen so they could replicate the image on the paper. Moving the phone could cause the user to lose their work.

A participant who attempted this task commented: “I have to hold the phone […] in place without moving. So, in this case it seems pretty difficult to keep it up, [it] keeps losing the paper. […]. [It’s frustrating] that it keep losing the paper, but, more than that, it is that I’m holding my phone with my left hand, looking at it without trying to move it. Because I feel like if I move it, then it’s going to lose track of the paper. […] I’ve lost it. […] Is it because the table is white as well and it’s losing it or is it because I’m just moving around so much?”

How to Prep the Environment

Preparing the physical environment can be another hurdle that users need to overcome as they interact with an AR feature. Users should receive timely and contextually relevant step-by-step instructions regarding requirements such as lighting, surface quality, and space size. For example, some applications will perform better when viewed outdoors or in a particular location.

One participant who was using the Smartify: Museum and Art Guide app received no clear instructions regarding the preferred size of the space where she should place an AR artifact; hence she had difficulty fitting and adjusting it in her room. She said, “So, it should have told me, you know, ‘the object is going to be three-by-three-feet size;’ then, I would have scanned or chosen the location accordingly so that there wouldn’t have been this confusion of, you know, ‘more light, move your phone, and all that.’ […]  If they would tell me ‘this particular art item would need this much space. So, scan accordingly’ — that would be helpful.”

The Augmented Berlin app did give some instructions regarding the requirements of AR experience, but the instructions were not detailed enough. A study participant said, “It was good that it gave me a tip like that it would require more space and lighting, but, you know, it didn’t tell me how much. More — could be any amount. So having the exact space that it would require, like saying it would require a 10-by-10-feet wall, […] would have been helpful for me to choose the right space, because now I can hear the audio and the audio says ‘here, it happened something, something happened here’ [but I don’t see anything].”

Additionally, the instructions should not be overwhelming or too complicated. The Best Buy app asked the user to attach four stickers to the environment to calibrate the AR feature. One study participant felt it was too much and skipped this part: “It’s getting difficult for me to, you know, project it because it needs particular things, but I think I [will] just try to place and I tap it. […] It says your TV was placed but I can’t see the TV. Yeah, [..]’ it’s a little difficult, it’s not very […]’’ user friendly. It’s asking so many specifications to be, like, perfect.” 

Many users won’t bother to go through a complicated process to view an item in the room, especially if their primary goal is not interaction with the AR object. In such cases, the AR feature is useful only if it adds extra value with minimum effort.

Another critical aspect of the AR experience is users’ safety. Many AR interactions involve moving around the physical environment while looking at the phone screen. Hence, users might disregard their environment or miss potential hazards such as slippery surfaces, sharp objects, or obstacles. Tell users to find a safe place away from all these potential risks before beginning the AR experience.

Conclusion

AR is an exciting new technology that has plenty of potential. Many of our study participants were delighted when they were able to go through an experience successfully. For example, after using the MauAR — Berlin Wall app, one participant said, “I think it’s cool! I will probably hang on to this [app] after this [session]! I would probably like [to] run through it and finish it out. I’ve never seen anything like it! […] It’s like an education application that is a little more immersive…”

However, AR-related patterns (as well as the meaning of AR) are unfamiliar to most users. To help people take advantage of AR in your apps, consider developing contextual tips and instructional walkthroughs that set the stage and help users prepare for the interaction.

Make sure that your walkthroughs describe the purpose of your app and how the AR functionality fits in. Then clearly explain to users how they are supposed to handle their device throughout their experience and how their space should be chosen for an optimal, safe interaction.

With the advent of Web 2.0, authenticating users became a crucial task for developers.

Before Web 2.0, website visitors could only view the content of a web page – there was no interaction. This era of the internet was called Web 1.0.

But after Web 2.0, people gained the ability to post their own content on a website. And then content moderation became a never-ending task for website owners.

To reduce spam on these websites, developers introduced user authentication systems. Now website moderators can easily know the source of spam and can prevent those spammers from accessing the website further.

If you are want to know how to implement content moderation on your website, you can read my article on How to detect and blur faces in your web applications.

Now let’s see what we’ll be getting into in this tutorial.

What You’ll Learn in This Tutorial

In this tutorial, we will discuss different authentication techniques you can use to authenticate users. These include email-password authentication, phone auth, OAuth, passwordless magic links, and at last facial authentication.

Our primary focus will be on authentication via face recognition techniques in this article.

We’ll also build a project that teaches you how to integrate facial recognition-based authentication in your React web application.

In this project, we’ll use the FaceIO SaaS (software as a service) platform to integrate facial recognition-based authentication. So, make sure you set up a free FaceIO account to follow along.

And finally, we’ll take a look at the user privacy aspect and discuss how face recognition doesn’t harm your privacy. We’ll also talk about whether it’s a reliable choice for developers in the future.

This article is packed with information, hands-on projects, and discussions. Grab a cup of coffee and a slice of pizza 🍕 and let’s get started.

The final version of this project looks like this. Looks interesting? Let’s do it then.

Different Types of User Authentication Systems

There are many user authentication systems out there right now that you can choose to implement in your websites. There are no real superior or inferior auth techniques. All of these auth systems depend on using the right tool for the job.

For example, if you are making a simple landing page to collect emails from users, there is no need to use OAuth. But if you are building a social platform, then using OAuth makes more sense than traditional authentication. You can pull the user’s details and profile images directly from OAuth.

If your web application is built around any investment-related content or legally binding services, then using phone auth makes more sense. A user can create unlimited email accounts but they’ll have limited phone numbers to use.

Let’s take a look at some popular authentication systems so we can see their pros and cons.

Email-password based authentication

Email-password-based authentication is the oldest technique for verifying a user. The implementation is also very simple and easy to use.

The pro of this system is you don’t need to have a third-party account to log in. If you have an email, whether it is self-hosted or from a service (like Gmail, Outlook, and so on), you are good to go.

The primary con of this system is you need to remember all of your passwords. As the number of websites is constantly growing and we need to log in to most sites to access our profiles, remembering passwords for every site becomes a daunting task for us humans.

Coming up with a unique and strong password is also a huge task. Our brains aren’t typically capable of memorizing many random strings of letters and numbers. This is the biggest drawback of email-password-based authentication systems.

Phone authentication

Phone authentication is generally a very reliable auth technique to verify a user’s identity. As a user typically doesn’t have more than one phone number, this can be best suited for assets-related websites where user identity is very important.

But the drawback of this system is people don’t want to reveal their phone numbers if they don’t trust you. A phone number is much more personal than an email.

One more important factor of phone authentication is its cost. The cost of sending a text message to a user with an OTP is high compared to email. So website owners and developers often prefer to stick with email auth.

OAuth-based authentication

OAuth is a relatively new technique compared to the previous two. In this technique, OAuth providers user authentication and useful information on behalf of the user.

For example, if the user has an account with Google (for example), they can log in to other sites directly using their Google account. The website gets the user details details from Google itself. This means that there’s no need to create multiple accounts and remember every password for those accounts.

The major drawback of this system is that you as a developer have to trust the OAuth providers and many people don’t want to link all their accounts for privacy reasons. So you’ll often see an email-password field in addition to OAuth on most websites.

Magic link authentication

Magic links solve most of the problems you face in email password-based authentication. Here you have to provide only your password and you will receive an email with an auth link. Then you have to open this link in your browser and you are done. No need to remember any passwords.

This type of authentication has gained in popularity these days. It saves a lot of time for the user, and it’s also very cheap. And you don’t have to trust a 3rd-party like in the case of OAuth.

Facial recognition authentication

Facial recognition is one of the latest authentication techniques, and many developers are adopting it these days. Facial recognition reduces the hassle of entering your email-password or any other user credentials to log in to a web application.

The most important thing is that this authentication system is fast and doesn’t need any special hardware. You just need a webcam, which almost all devices have nowadays.

Facial recognition technology uses artificial intelligence to map out the unique facial details of a user and store them as a hash (some random numbers and text with no meaning) to reduce privacy-related issues.

Building and deploying an artificial intelligence-based face recognition model from scratch is not easy and can be very costly for indie developers and small startups. So you can use SaaS platforms to do all this heavy-lifting for you. FaceIO and AWS recognition are examples of these type of services you can use in your projects.

In this hands-on project, we are going to use FaceIO APIs to authenticate a user via facial recognition in a React web application. FaceIO gives you an easy way to integrate the authentication system with their fio.js JavaScript library.

Project Setup

Before starting, make sure to create a FaceIO account and create a new project. Save the public ID of your FaceIO project. We need this ID later in our project.

To make a React.js project, we will use Vite. To start a Vite project, navigate to your desired folder and execute the following command:

npm create vite@latest

Then follow the instructions and create a React app using Vite. Navigate inside the folder and run npm insall to install all the dependencies for your project.

After following all these steps, your project structure should look like this:

.
├── index.html
├── package.json
├── package-lock.json
├── public
│   └── vite.svg
├── src
│   ├── App.css
│   ├── App.jsx
│   ├── assets
│   │   └── react.svg
│   └── main.jsx
└── vite.config.js

How to Integrate FaceIO into Our React Rroject

To integrate FaceIO into our project, we need to add their CDN in the index.html file. Open the index.html file and add the faceIO CDN before the root component. To learn more, check out FaceIO’s integration guide.

<body>    
    <script src="https://cdn.faceio.net/fio.js"></script>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
</body>

Now remove all the code from the App.jsx file to start from scratch. I’ve kept this tutorial as minimal as possible. So I’ve only added a heading and two buttons in the UI to demonstrate how the FaceIO facial authentication process works.

Here, one button works as a sign-in button, and the other one works as a log-in button.

The code inside the App.jsx file looks like this:

import "./App.css";
function App() {
  return (
    <section>
      <h1>Face Authentication by FaceIO</h1>
      <button>Sign-in</button>
      <button>Log-in</button>
    </section>
  );
}

export default App;

How to Register a User’s Face using FaceIO

Working with FaceIO is very fast and easy. As we are using the fio.js library, we have to execute only one helper function to authenticate a user. This fio.js library will do most of the work for us.

To register a user, we initialize our FaceIO object inside a useEffect hook. Otherwise, every time a state changes, it re-runs the components and reinitializes the faceIO object.

let faceio;
useEffect(() => {
    faceio = new faceIO("Your Public ID goes here");
}, []);

Your FaceIO public ID is located on your FaceIO console. Copy the public ID and paste it here to initialize your FaceIO object.

Now, define a function named handleSignIn(). This function contains our user registration logic.

Inside the function call the enroll method of the faceIO object. This enroll method is equivalent to the sign-up function in a standard password backed registration system and accepts a payload argument. You can add any user-specific information (for example their name or email address) to this payload.

This payload information will be stored along with the facial authentication data for future reference. To learn about other optional arguments, check out their API docs.

In our sign-in button, on user click we invoke this handleSignIn() function. The code snippets for user sign-in look like this:

const handleSignIn = async () => {
    try {
      let response = await faceio.enroll({
        locale: "auto",
        payload: {
          email: "example@gmail.com",
          pin: "12345",
        },
      });

      console.log(` Unique Facial ID: ${response.facialId}
      Enrollment Date: ${response.timestamp}
      Gender: ${response.details.gender}
      Age Approximation: ${response.details.age}`);
    } catch (error) {
      console.log(error);
    }
  };

<button onClick={handleSignIn}>Sign-in</button>

How to Sign In using Face Recognition

After registering the user, then you’ll need to get the user into the authentication or log-in/sign-in flow. Using the fio.js library also makes it very easy for us to set up a log-in flow using face authentication.

We have to invoke the authenticate method of the faceIO object which is equivalent to the sign-in function in a standard password backed registration system and all the critical work will be done by the fio.js package.

At first, define a new function named handleLogIn() to handle all the log-in logic in our React app. Inside this function, we invoke the authenticate method of the faceIO object as I mentioned earlier.

This method accepts a locale argument. This is the default language of the interaction of users with FaceIO widgets. If you are not sure, you can assign auto in this field.

The authenticate method also take more optional arguments like permissionTimeout, idleTimeout, replyTimeout and so on. You can check out their API documentation to know more about optional arguments.

We invoke this handleLogIn() function when someone clicks on the Log-in button:

const handleLogIn = async () => {
    try {
      let response = await faceio.authenticate({
        locale: "auto",
      });

      console.log(` Unique Facial ID: ${response.facialId}
          PayLoad: ${response.payload}
          `);
    } catch (error) {
      console.log(error);
    }
  };

<button onClick={handleLogIn}>Log-in</button>

Our user authentication project using FaceIO and React is now complete! You learned how to register and login a user. You can see the process is fairly simple compared to implementing an email-password based or some other authentication method we discussed earlier in this article.

Now you can style all the jsx elements using CSS. I didn’t add CSS here to reduce complexity in this project. If you are curious, you can take a look at my GitHub gist.

If you want to host this React FaceIO project for free, you can check out this article on how to deploy your React and Nextjs project in Cloudflare pages.

How to Use the FaceIO REST API

Besides providing widgets via the fio.js library, FaceIO also provides REST APIs to streamline the authentication process.

Every application in the FaceIO console has an API key. You can use this API key to access the FaceIO REST API endpoints. The base URL for the REST API is https://api.faceio.net/.

The URL schema accepts URL parameters like this https://api.faceio.net/cmd?param=val&param2=val2. Here cmd is an API endpoint and param is an endpoint parameter if it accepts any.

Using the REST API endpoints, you can automate various tasks in your backend.

1. You can delete a face ID on a user’s request.
2. You can attach a payload with a face ID.
3. You can change the PIN associated with a face ID.

This REST API is intended to be used purely on the server side. Make sure you don’t expose it to clients. It’s important that you read the following Privacy and Security sections to learn more about this.

How to Use FaceIO WebHooks

Webhooks are event-driven communication systems among servers. You can use this webhook feature of FaceIO to update and sync your backend with new events happening in your front-end web application.

The event of this webhook fires on new user enrollment, facial authentication success, facial ID deletion, and so on.

You can set up FaceIO webhooks in your project console. A typical webhook call from FaceIO lasts for 6 seconds. This contains all the information about a specific event in a JSON format and looks like this:

{
  "eventName":"String - Event Name",
  "facialId": "String - Unique Facial ID of the Target User",
  "appId":    "String - Application Public ID",
  "clientIp": "String - Public IP Address",
  "details": {
     "timestamp": "Optional String - Event Timestamp",
     "gender":    "Optional String - Gender of the Enrolled User",
     "age":       "Optional String - Age of the Enrolled User"
   }
}

Privacy and FaceIO

Privacy is the most important thing for a user nowadays. As big corporations use your data for their good, questions arise on whether the privacy of these face recognition techniques is valid and legitimate.

FaceIO as a service follows all the privacy guidelines and gets user consent before requesting their camera access. Even if the developer wanted, FaceIO doesn’t scan faces without getting consent. Users can easily opt-out of the system and can delete their facial data from the server.

FaceIO is CCP and GDPR compliant. As a developer, you can release this facial authentication system anywhere in the world without facing privacy issues. You can read this article to know more about FaceIO privacy best practices.

FaceIO Security

The security of a web application is an important topic to discuss and consider. As a developer,  you are responsible for the security of a site or application’s users.

FaceIO follows some important and serious security guidelines for user data protection. FaceIO hashes all the unique facial data of the user along with the payload we specified earlier. So the stored information is nothing but some random strings which can’t be reverse engineered.

FaceIO outlines some very important security guidelines for developers. Their security guide focuses on adding a strong pin code to protect user data. FaceIO also rejects covered faces so that no one can impersonate someone else.

Conclusion

If you’ve read this far, thank you for your time and effort. Make sure to follow along with the hands-on tutorial so you can fully grasp the topic.

The project should be approachable if you follow all the steps. If you make something out of it, show me on Twitter. If you have any questions, please ask. I will happy to help you. Till then, have a good day.