c[AR]d experience

Team Members: Yuxuan Liu, Yuanchu Si

c[AR]d is a mixed reality game that uses both technologies of AR and VR to create a tangible yet life-like board game experience. Players play board games with tangible paper cards, but also can see monsters and characters wandering around or explosions happen nearby.

Project Link: https://drive.google.com/open?id=1B28_sWhimnyP8GITVkvXLmE_Y5o-5I0n

c[AR]d trailer
cARd gameplay


The motivation of this game came from the exploration of new input system for current VR headset. The initial thought is that we think the current VR controller sometimes can cause motion sickness problem because what player see don’t match with players’ real action. Since the VR headset provide a more immersive and engagement environment, we want to find a way to let player interaction with the virtual environment more naturally.

With this idea in mind, we realized that the board game could be a good example of exploring other formats of the input system. Besides, the virtual environment also could potentially increase the original board game experience. So we started to look for a game that simple enough for implementation and has the potential to add more visual effects in the VR environment. At first, we wanted to recreate a chess game in the virtual environment, the idea is that we use an additional camera to capture the real chess movement and render it the VR environment. However, it turns out that it could be extremely difficult to do real-time object recognization so switch the idea to the card game since the image target is easier to track.

Design Inspiration and Prototype:

When I was a child, I always like to collect the cards from my favorite anime. One of my favorite anime is called Digimon. My friends and I play a lot of simulation fight games with those cards we have. Sometimes I think it could be cool if we can see the Digimon fight against each other besides me. Nowadays with the rapid development of mixed reality technology, it has become possible to play to the game in real life. I think using AR is a possible way to achieve it, but the VR headset could provide a much better immersive experience and visual effects compared to the AR display. That brought the initial idea of creating this mixed reality card game.

To demonstrate this idea, we found a game called Concentration perfectly matches the requirement. This is a memory game with very simple and straightforward rules. The players will start with a set of unrevealed cards. Each player can flip two cards one time and players will play in turn. If those two cards front pattern matches, the player who flips it will win the pair, and if not the players need to flip the cards back.

The poetic moment in our game is depicted as the moment that virtual characters like Digimons come into life from printed cards.

Guilmon Comes to Life

To create such experience, in the first prototype we built a PC version of the game and did the first round playtest. In the first prototype, we collected the user’s feedback and refined the game based on that.

User Testing and Evaluation:

Once we had the VR build of the game, we did an experiment to compare the player experience of the traditional board game and VR environment. There are three things we want to measure in this experiment, it includes (1) the general player experience in both environment (2) the social engagement in both environment, and (3) whether the player likes the tangible interaction.

We conducted a survey the explore these three problems and the following graph is the result.

All the survey’s questions are a 7-point Likert scale which 1 indicates total disagree and 7 means totally agree.

The Median of General Game Experience Questions

From the result, it is obvious that the VR version of this game provides a more immersive and engaging experience compared to the traditional version. Players could get more clear feedback and generally more enjoy the VR version of the game. However, the VR environment also increases the difficulty and limited the freedom of how the player plays this game.

The Median of Social Engagement Questions

In terms of questions about social engagement, I found that the traditional set up is outperformed than the VR environment. Although both setup players are able to talk to each other, players still feel less connected to the VR headset. From the chart, we can see that most of the players hold a slightly disagree attitude towards interacting with other players in the VR environment while strongly agreed that they have some sort of interaction and connect with the other player in a traditional manner.

The Median of Tangible Interaction Questions

From our result of tangible interaction questions, I find that most of the players hold a neutral attitude towards the tangible interaction and the median result even shows that sometimes they even disagree that tangible interaction would improve their player experience. But when they switched to the VR environment, most of the players started to miss the feeling of touching cards and those game pieces. This result shows that although the players do not think to add those tangible interactions will improve their general experience, they still prefer the original cards and pieces compared to the controller.

Based on the result, I think the idea of enhancing the board game experience with VR headset is proved to be possible and beneficial. The current VR environment is lacking the content that encourages social communication but I think we can optimize it by adding a virtual avatar, motion synchronization, and even facial expression synchronization in future development. For the tangible interaction, this game is limited to interact with cards so the result might not typical enough to prove that it significantly contributes to the player experience. It is still worth to be explored in the future since most of the players still want it in the VR environment.


Player (you) is a board game hobbyist. Last year, you started playing a fantasy setting board game and you love it. You always play with friends, but much more than that. Sometimes you dreamed of the dragons and monsters in the game. You commanded the army of the magic creatures to fight your ways in the fantasy world.

Recently the dreams get more intense. The fights are more real to you and sometimes when you wake up, your body feels the pain when the magic missile from an enemy mage that hurt you. You are afraid that the dream consumes your sleep too much, but you just can’t help playing it for fun.

Yesterday, when you played with your friend, you played your favorite black dragon on the board again. Something wrong happened. Immediately you see huge shadow casting on your window. When you look out of the window, you see the real dragon flying in the sky and prepares to breathe fire to your friend. You were so shocked and woke up.

c[AR]d is a mixed reality game that uses both technologies of AR and VR to create a tangible yet life-like board game experience. Players play board games with tangible paper cards, but also can see monsters and characters wandering around or explosions happen nearby.

Dragon Fly

By Yuxuan Liu, Yuanchu Si

Circuit Playground Express

Concept: The board we used has many cool functions including motion sensor, temperature sensor, light sensor, and even sound sensor. We have tried almost all kinds of sensor and finally decided to implement the motion sensor. The benefit of the motion sensor is that can give us more synched feedback with reality and without much delay. We use data from the motion sensor and make the circuit board into a flight controller. Users should expect the flight vehicle they control maneuvers as the controller tilts. They should wear the controller as a watch and move their arms to maneuver. The game is basically a fly simulator, we presented it as a dragon controller but it also has a potential implementation in real-world like a drone controller.

Motivation: The motivation of this project is that we feel that traditional controllers don’t provide immersive enough experience in flight simulation. Typically players use joysticks to control the posture of the vehicle, which requires a long learning curve. To increase the immersion, we want our user’s input to match the real movement as much as possible. Also, we feel that directly control the rotation with the real hand rotation could be more accurate and easier to learn.

Application and future work: Drone control, any other flight simulators, racing games. Optimization based on implementation.

Digimon into life

In the c[AR]d project, we dedicated to building an experience to bring board games into life. Monsters, magic spells, battlefields are not paintings printed on cards; instead, they come into life thanks to AR/VR technology. Still, players use real tangible cards to play board games. The mixture of reality and virtuality could create the ultimate board game experience.

Now we are building a demo with characters in anime Digimon. Digimons come out or evolve when players flip the same cards in a turn. They stand by the player and are presented in scale so that players could see their favorite Digimons fighting by their sides.

The poetic moment in our game is depicted as the moment that virtual characters like Digimons come into life from printed cards.

ECD Assignment1

  1. Face Platformer
  2. Kinect Cosplay
  3. VR + Kinect

Face Platformer

Want to use eyebrows to play a 2D platformer? You can do it!

Squeeze your eyebrows to charge, and release them to jump! The game uses computer vision to recognize facial movements of such as eyebrow movements. We could add more controls triggered by other registered facial movements or expressions, such as smile and sad face.

You play this game using your eyebrows

Technology: computer vision

Kinect Cosplay

Kinect cosplay could let players cosplay their favorite characters without being physically dressed up with costumes. Kinect recognizes human skeletons and software could “dress” them with preset 3D models. Museums, exhibitions, or theme parks could setup this for visitors to perform as their favorite characters.

Imagine the face is yours at DisneyLand

Technology: Kinect

VR + Kinect to increase social presence in VR

In today’s VR systems, the virtual avatar are depicted according to the positions of the player’s head and hands positions. It’s not accurate because it lacks the body posture so sometimes the animated avatar might seems weird, especially in multiplayer applications that require high fidelity social presence. If combined with Kinect device set up in front of the player, the application could animate the avatar with the skeleton data captured by Kinect to dramatically increase the fidelity of body posture.

“Real man fighting” in VR?

Technology: Kinect, VR

A simple Kinect game made with Unity

Yuxuan Liu, Yuanchu Si

We built a simple game that players interact with in-game objects through Kinect body recognition. Some random food objects fall from the top of the screen. The player gains score by moving their body and touch the object in the game scene. The rules are:

  • +10 by touching with head, hand
  • +5 by touching with arm, shoulder

Dota 2 Solo Mid Heart Rate Analysis

Yuxuan Liu, Yuanchu Si

Dota 2 is usually considered one of the most competitive digital games ever because it requires complex real-time decision making and super quick reactions. In this analysis, we try to investigate how the heart rate variation changes as the gaming phases shift in Dota 2.

Our hypothesis is that when a player is attempting to kill an enemy hero, the player’s heart rate will rise significantly, compared to laning and last-hitting. 

We collected the data by setting up 2 players playing Solo Mid mode with Shimmer HR sensor installed on them to record ECG during the game session. The game lasted about 5 minutes and we got synced data of heartbeat records and screen recording video, by which we can inspect the heart rate variation with what’s happening in the game. 

The data from the Shimmer sensor is in the unit of mVolts, and about 200 records compose a single heartbeat. We detected all the peaks, calculated the time difference between peaks, and calculated the heart rate variation among different game phases.

These results are in the table below. 

Game phaseHeart beat variation
Before game starts (Phase 1)93.22
Before the first fight (Phase 2)208.78
During the first fight (Phase 3)206.32
During the second fight (Phase 4)115.11
During the third fight (Phase 5)81.41

We can see in Phase 1 and Phase 5, the heart rate remains steady, while the heart rate is significantly higher in Phase 5 than Phase 1. In Phase 2 and 3, we can see the heart rate gradually increase towards a high-frequency level. It might be because right after the game starts, the player gradually gets more intense and focused on gameplay. 

The result is that once the game heats up, the player’s heart rate would hold steady at a high level, rather than jump between high and low depending on different scenarios. We can see that once the game begins, the player’s heart rate begins to rise and hold steady throughout the entire game session. It probably means that the player is constantly stimulated and ready to tackle any challenges that could happen in milliseconds. Fighting with an enemy hero is only one of the scenarios in Dota 2 and it’s nothing too different than other in-game behaviors in terms of the degree of stimulation to players. 

Brain State of Watching different Videos Using OpenBCI

Yuanchu Si, Yuxuan Liu, Yunbo Yang

In this study, we want to analyze how different types of content stimulate different brain activities. We collected data by OpenBCI with six channels on the front head and observed the variation of power bands by time series. We showed the participant three pieces of video to see the difference.

The OpenBCI software extracted five power bands: delta, theta, alpha, beta, and gamma. The reference of what power bands indicate is shown below. In summary, delta and theta represent the level of drowsiness, alpha represents the level of relaxing, beta represents the range of focus and gamma is high when you recognize something you have seen recently.

Provocative music video

The participant listened to a K-pop song and its provocative music video by order. When he listened to music only, his brainwave doesn’t show much variation among different bands, showing he might not be paying attention. When he was presented the music video, we can see more up-and-downs, especially beta and gamma bands. Also, he paid more attention when the music video is shown. 

We found that the gamma band would rise significantly each time when the refrain repeats. It shows the participant recognized the refrain and enjoyed it. Also when it reached the part that the singers danced with provocative postures, beta band rises significantly. It might show the participant was stimulated and more interested in these parts of the music video.

Dota Watafak

This Dota 2 compilation video consists of some hilarious clips of gameplay. The bands were normal before the hilarious part begins. When it reached the funny part, all bands rise significantly. He must be amused by the stupid plays. 


In this study, we compared different power bands of EEG between listening to a piece of music and watching its music video. We found that the beta signal would rise when watching the video against only listening to music, which confirms our hypothesis that both vision and hearing could influence the degree of concentration. Moreover, different scenes of a video significantly determine the focus level of the participant. 

Facial recognition of human and cat faces + interactions

Yuanchu Si, Yuxuan Liu

Screen recording of this program

We used Processing and OpenCV to recognize both human and cat face in a single program, added with some simple interactions.

  • Different text on different types of faces
  • Changes the text beside cats by size (proximity)
  • Changes the text beside cats by the distance between the cat and human faces

Varjo VR Eye-tracking Data Record

Presented by: Yuxuan Liu, Yuanchu Si

We imported Varjo SDK into a Unity Environment asset called Adam Exterior Environment. To enable movement, we create an invisible plane in the sky on which players can walk around and explore the world with their controllers. We want to see what the participants looks at in this game world. The video recorded the VR scene and the eye-tracking data of the participants. 

Group Non-Existing (Digital Game Blog Post)

Since no one in class wants to change their group, we just started to think about this project in our original group, and we had a couple of ideas.

On the last Thursday, Liu said he want to make a multiplayer 3D game, and he has two ideas about the game. The first idea is a two player cooperative game and one of the player plays in 2D scene while the other plays in 3D scene. The other idea is a competitive game and we give players some building materials and they build their own defence and try to destroy others. However Fey said he wants to make some games more abstract and more focus on experience and feelings. These are the games we wouldn’t have chance to make when we go the industry.

On the last Tuesday, we (Liu, Haowei, Fey, Dan’s group ) met again, we shared ideas from each other, while ended with discussing within the original group.

Haowei asked Fey if he wanted to continue the idea of experience game with emotions as the main theme, but Fey might move to other ideas.

She came up with a gameplay mechanic for horror game that since fear is from people themselves, so the game can set a scale of fear, the character would be difficult to control when it is higher, and more horror stuff would appear in the game.

Another idea was a setting for game.

Sketchbook Game (name TBD)

Narrative: It’s about person’s journey inside his brain. Trying to find an idea while he wander through his deep subconscious. Game starts at a cliff, with a sign on it: memento mori. Meaning ‘’remember, you are a mortal’’. Player has to jump into void to start his/her journey. (jump will mean ‘’leap of faith’’)

While player fighting with his/her enemies which is: depression, anxiety, boredom, self-doubt, he will find an idea. And idea will turn into a boss fight. Because sometimes you have to kill your ‘idea’ even though you fall in love to it. When player defeats the idea(boss), monstrosity will turn into a baby in a crib. And player has to strike to that innocent being. – the meaning behind this: it’s hard to kill your ideas, even more painful but has to be done. –

Player will start as a child and growing up inside game world till he is strong enough to fight the boss -idea-. The player start like 4 or 5 years old, and he has superpowers that he will lose in the future

Disrespect, boredom, fear

Self-abased, self-doubt, anxiety, depression

Level TBD: Rescue a butterfly

Instead of a journey throuth of finding an idea, we wanted to tell a story about how education system takes children ‘’superpowers’’. We wanted give players superpowers such as flying at the start and when they get almost get used to the mechanic, we take it away from them.

We demonstrate taking superpowers away in a way that show player she is not the only one. Education system reduces everyone into the same level and student compete on that unfair playground. They make it look like it’s fair by taking away children superpowers, in this case, you can see in the box there are magical wands, rubrics cube and soon to be added: our characters flying cape.

Flying level

(players reach the ‘teacher’ level by passing this flying level)

Main theme was of this levels inspired from a saying that ‘’Will they kick us out from the game if don’t look like them?’’
There are people watching the flying character, they all look empty inside. Like their joy was taken from them. And the birdhouse at the end. It’s metaphorical classroom/school. Since all birds pushed away by their parents to fly alone.

We add some inspirational moments from history shows that being different is not a shame.

Since we had a limited time and limited resources, we had to skip the parental protection phase of the character and focus more on criticize the educational system.

We sketch some ideas about our main character. Even though we had something we like, we decided to use more abstract character rather than real ‘girl’.

At next levels, we will focus on a parental aspect of the education system. Children’s seen as racing horse and have to clog their ‘friends’ way to climb the ladder of success.

In this part of the game, we tried to visualize the ‘’exams’’. It’s a race and all parents are watching their children. Of course, some kids don’t succeed as other do, so we put them as a obstacles for player to jump over. It’s key to feedback players that when they jump over ‘’other kids’’ they have to receive a success sound as a feedback. Cause that’s what happens in real-life.

Our character has to chase a butterfly and tries to catch before the ‘’exam’’ ends. But players never able to catch it. And they fall in to the ‘’line level’’ which is demonstrates the success and failure in education system. In Success part, there is kid who already catch the butterfly and taking the elevator to ‘’success’’ but our character has to fall down to the ‘’failure’’ since she couldn’t catch the butterfly.

We tried to visualize a year in a kid’s life. A year that she starts school. We try to achieve: visualize the story, criticize the system and raise empathy on players.

Lately, Fey been listening the Muse’s new album The Simulation Theory. And he get inspired from that so much. In the album, Matt Bellamy mentioned that people who rise and question the systems ending up failing since ‘’we are caged in simulations’’ and ‘’It’s too late for a revolutionBrace for the final solution’’. That’s why we added a newgameplus option. When players fail(game doesn’t allow them to succeed) they will have an option to ‘’try again’’ or ‘’out of system’’

When they enter ‘’try again’’ they will go back to the beginning of the game but the interactions text will differ and those interactions will point out to the player ‘’you are caged in simulation’’.

When players choose out of system option, they will see a warning ‘’this means war, with your creator’’ and the ‘’ending screen’’ will appear. Which we haven’t made the decision yet but it will be ready for the Thursday.