This interactive immersive installation explores the invisible processes through which artificial intelligence observes, records, and transforms human behaviour into data. By placing the participant at the centre of the experience, the project visualises the journey from a physical presence to a machine-readable profile, revealing how AI systems perceive people as collections of measurable information.
The experience begins with a simple act of person detection. As the participant enters the space, the system identifies their presence and establishes them as a subject within its field of observation. This minimal interface reflects the first stage of computer vision, where an individual is recognised not as a person but as an object to be tracked.
The second stage introduces real-time line graphs generated through hand and pose tracking. Every movement is translated into numerical values and live visualisations, exposing how gestures and body language become quantifiable parameters that can be measured and stored. Next, a heatmap and motion feedback system accumulates the participant’s movements over time. Rather than representing a single action, the visualisation builds a behavioural footprint, revealing patterns and frequencies that are invisible to the human eye but valuable to machine learning systems.
The final processing stage shifts from data collection to computation. The immersive space fills with floating numbers, calculations, and fragmented datasets that surround the participant, visualising the hidden analytical processes performed by AI. This stage represents the organisation and processing of collected information before it is transformed into a usable model.
Once the computation is complete, the interface displays “100% Complete”, followed by “Subject Categorized.” After an extensive process of observation and analysis, the participant is reduced to a simplified numerical identity and assigned a predefined category.
Rather than criticising artificial intelligence itself, the project invites audiences to reflect on the mechanisms of contemporary data-driven technologies. It questions how everyday interactions are continuously converted into datasets for machine learning and training, and how these systems inevitably reshape human experience by treating people as collections of measurable variables rather than complex individuals.
Charlotte Juliens – Ekin Ayca Demirli – Inioluwa Adeyiga
During the final week, we focused on editing all of our footage together in After Effects. We brought the different visual stages of the project into one complete sequence, making sure the transitions between the detection, data visualisation, heatmap, calculation process and final classification felt clear and connected.
We also wrote the final project statement, which helped us explain the concept, aim and meaning behind the work more clearly. The statement allowed us to reflect on what the project is about and how it questions the way technology can collect, analyse and categorise human behaviour.
We also added a voice-over to explain the project throughout the video. The voice-over helped guide the viewer through each stage and made the process easier to understand. It explained how the system detects the subject, collects movement data, transforms it into graphs and heatmaps, and finally categorises the person based on the information it has gathered.
This final editing stage helped bring the whole project together. By combining the visuals, text, effects, project statement and voice-over, the project felt more complete and communicated our idea more clearly. The final outcome shows how data can be used to analyse human movement, while also questioning how technology can turn people into measurable information.
In this week, we reserved the 270° room because our project was designed as an installation. We wanted to test how the visuals would work in a larger immersive space, rather than only viewing them on a computer screen. This was an important step because the scale of the room changed how the project felt and made the data visualisation appear more like a real system surrounding the viewer.
Throughout the project, one of the biggest challenges was working with different versions of TouchDesigner. Some computers were using the 2023 version, while others had the 2025 version. This caused problems because certain operators and functions were different or missing depending on the version. For example, some TOPs and setups that worked in the newer version did not work properly in the older one.
When we arrived in the 270° room, we realised that the computer there did not have the newest version of TouchDesigner. This created a problem at first because our project was built using tools from the newer version. To solve this, we rendered the project from a different computer that had the correct version installed. This allowed us to continue testing the installation in the room without having to rebuild the whole project.
Even though this was stressful, it helped us understand how important it is to check software versions before presenting or installing a project. In the end, we were still able to test the work in the 270° room and see how our visuals, heatmaps, data, and completion screen functioned in an immersive environment.
This week, we continued developing the next stage of our project by adding heatmaps to our MediaPipe setup. After successfully connecting the hand and body detection, we wanted to create a stronger visual representation of the subject’s movement and activity.
The heatmap allowed us to show where the subject was moving the most on screen. This added another layer to the project, because the body was no longer only being tracked through points and lines, but also through areas of intensity. The more movement happened in one area, the stronger the visual effect became.
After this, we focused on how to connect our data and numbers into a clear and effective visualisation. We already had tracking values and movement data, but we needed to find a way to make this information look visually interesting and understandable. To help with this, we watched multiple videos and references about data visualisation, motion tracking, and digital interface design.
These references helped us think about how numbers, graphs, heatmaps, and labels could work together on screen. We wanted the final result to feel like an advanced system analysing the subject in real time. This stage was important because it helped us move from a basic technical setup into a more designed and believable visual experience.
We especially liked the final video we looked at, as it showed a strong and clear representation of numerical data. Inspired by this, we recreated a similar visual style in our own project and then replaced the example values with our own tracked data. This helped us turn the raw numbers from our system into a more polished and readable visual output that fit the overall concept of data extraction.
To move towards the final stage of our project, we started working on the completion screen. At this point, we wanted to create a strong visual ending that would clearly communicate that the system had finished analysing the subject. For this, we needed a visually effective way to show messages such as “100% Complete” and “Subject Categorized.”
We looked at different video references to explore how this kind of final screen could be presented in a clear and engaging way. One of the videos stood out to us because of its strong visual style, especially the glowing text and repeated screen-based effect. We all liked this reference, so we decided to base our own completion screen on it.
Using this as inspiration, we recreated a similar visual approach in TouchDesigner and adapted it to fit our project. This helped us design a completion screen that felt more polished and believable, while also matching the overall aesthetic of our data extraction concept. The result created a clear transition into the final stage of the system, making the project feel more complete and visually cohesive.
At the beginning of the session, we looked at different FMP projects. This was helpful because it showed how other students developed their ideas from a concept into a finished project. It also helped me understand how an experimental project can still have a clear structure, visual direction, and critical meaning behind it.
This week, we continued developing the next stage of our project by adding heatmaps to our MediaPipe setup. After successfully connecting the hand and body detection, we wanted to create a stronger visual representation of the subject’s movement and activity.
The heatmap allowed us to show where the subject was moving the most on screen. This added another layer to the project, because the body was no longer only being tracked through points and lines, but also through areas of intensity. The more movement happened in one area, the stronger the visual effect became
At the end of the session, we wrote a project statement to clarify the overall concept. This helped us define the purpose of the work more clearly. The project is not only about making an interactive visual system, but also about questioning how digital systems measure and simplify human behaviour. The system may be able to detect movement, but it cannot understand emotion, intention, or context.
This week, we worked with RealityScan, which is a photogrammetry tool from Unreal Engine. The aim of the session was to scan a real object using photos and turn it into a 3D model.
I first tried to scan my nose spray. I took photos from different angles and imported them into RealityScan. However, the scan did not work very well. The software was able to detect some of the images and camera positions, but the final result failed and did not create a clear 3D model of the object.
After that, I tried again using my Ice Tea Peach bottle. This worked slightly better, and the software was able to create more visible geometry. However, the scan still was not good enough. Because the bottle was already half empty, some parts of it became slightly transparent, which made it harder for RealityScan to read the object properly. The reflective plastic and liquid inside also made the scan more difficult.
Once the scan was generated, I used a selection tool, similar to a lasso tool, to delete the unwanted geometry around the object. There were a lot of extra pieces and messy shapes around the scan, so cleaning it up was an important part of the process
From this process, I learned that photogrammetry works better with objects that have a clear surface, strong details, and no transparency or reflection. Smooth, shiny, or see-through objects can confuse the software because it struggles to match the same points across different photos.
Overall, this session showed me how real-life objects can be turned into digital 3D models, but also how sensitive the process is. Even small issues, such as transparency, reflections, bad lighting, or not enough angles, can affect the final result.
After this, we started working on our own TouchDesigner project. Our project idea was called Data Extraction. The concept was about how human behaviour can be transformed into data. We wanted to create an interactive system where movement would be detected and converted into visual information, such as numbers, labels, graphs, and digital feedback. The idea was that the participant would appear as if they were being analysed by a machine.
We began by exploring how to use tracking inside TouchDesigner. We worked with the MediaPipe plugin because we wanted to use body or pose tracking to detect movement. We also looked at using the NDI 6 tools, as the plan was originally to bring in a camera feed through NDI. However, we had several technical issues with connecting the camera input correctly. Some nodes were not showing the image as expected, and it was confusing to understand how to connect the different operator types together.
One issue we came across was the difference between TOPs, CHOPs, and DATs in TouchDesigner. Some nodes were video/image based, while others were data based, which meant they could not always be connected directly. This made the setup more complicated, especially when trying to connect the camera feed into the MediaPipe system. We also had problems with the image segmentation and pose tracking not producing the live output we expected.
Because the NDI/camera setup was taking too much time to troubleshoot, we decided to simplify the workflow and use the webcam directly instead. This allowed us to keep moving forward with the project rather than getting stuck on the technical setup. Using the webcam still worked for the concept because the main idea was about detecting movement and translating it into data.
After switching to the webcam, we focused more on how the project could function visually and conceptually. The system would take the participant’s movement and turn it into a kind of data interface. Visually, we imagined the screen showing surveillance-style graphics, labels, metrics, and fake analysis such as “Subject Detected,” “Movement Data Extracted,”. These labels helped communicate the idea that the system is reducing a real person into simplified categories.
After completing the MediaPipe setup, we moved on to the next step: adding line graphs to visualise the subject’s movement data. Since MediaPipe was already detecting the body and hands, we wanted to translate that tracking information into a more data-driven visual layer.
We focused mainly on the hand detection, using the position values from the tracked points to create live line graphs. These graphs respond to the subject’s movement in real time, showing how the hand position changes across the screen. This helped us make the project feel more like a system that is actively analysing and collecting behavioural data.
By adding these graphs, the visualisation became more dynamic. Instead of only showing the detected skeleton or tracking points, we could now show the movement being converted into data. This connects well with our concept of data extraction, where the subject is not just being observed, but also measured and categorised through visual information.
This week, we explored how to create a 360° environment using an existing 3D scene. We started by downloading an environment asset from Fab. I chose a western-style map, with desert surroundings, wooden buildings, saloons, and mountains in the background.
After adding the map into Unreal, we worked with Off World Live. We added a 360 camera to the scene so that the environment could be captured in a 360° view instead of a normal camera view.
We then created a material for the 360 image. In the material graph, we used the 360 texture and connected it so it could be displayed properly. This helped us understand how a 360 image can be used inside Unreal as a material.
After that, we created a 360 HUD widget. In the blueprint, we used Event BeginPlay, Create 360 HUD Widget, and Add to Viewport, so the 360 view would appear when the project starts.
Once the HUD widget was working, we used the Sequencer to create a small animation. We added movement/keyframes to test how the 360 camera and environment could be animated over time. This allowed us to turn the setup into a short video sequence instead of only a still 360 image.
At the end, we imported/exported the sequence through Adobe Media Encoder. This allowed us to render the Unreal animation as a video file.
The final render had a visible black line in the centre, which was most likely caused by the 360 camera or export settings. This showed me that 360 workflows can be quite sensitive, especially when working with materials, widgets, and video export.
Alongside, our group worked on developing the visual direction of the project further. We looked at different references connected to artificial intelligence, data visuals, glitch effects and digital installation work. Through these references, we discussed what kind of visual style would best support our concept and make the project feel clear and engaging. This helped us make stronger decisions about the overall look of the piece and create a more unified visual identity.
This week, we received feedback for our TouchDesigner project idea, Data Extraction. The main concept suggested was that everything you do is turned into data. This gave us a clearer direction for the project, because it connects human movement, sound, and behaviour with systems of measurement and digital interpretation.
We were advised to research Rosa Menkman, especially her work around data, glitches, errors, and digital visualisation. Her practice could help us think about how data can be represented visually, not only in a clean or scientific way, but also through distortion, abstraction, and broken digital forms. We need to study how she gathers data and how she visualises it, so we can apply some of these ideas to our own TouchDesigner experiments.
Other possible research areas include neuroscience, Kinect, and Leap Motion. Kinect could allow us to track full-body movement, while Leap Motion could focus more specifically on hand gestures. These tools would help us make the interaction more physical and responsive. We also want to explore sound analysis, where music or voice is turned into visual data inside TouchDesigner.
During the class, we used NDisplay to recreate a billboard-style screen setup. We first created the setup in Unreal Engine and worked with two screens, a left screen and a right screen, inside the 3D environment. We also made a small animation in Unreal Engine to test how movement would work within the scene. After this, we rendered the result from Unreal Engine.
We then brought the render into Premiere Pro, where we edited and exported the video so that the left and right screens could be shown correctly. This helped us understand how a virtual production workflow can move between Unreal Engine and video editing software.
This session helped me understand how virtual production can combine 3D environments, animation, video rendering, and editing workflows. It also showed me how NDisplay can be used to create multi-screen setups, such as billboards, LED walls, or other large screen-based environments.
In our second week, we tested VCam for virtual production in Unreal Engine. The aim was to explore how a phone can be used as a virtual camera to move through and frame a digital environment in a more physical and intuitive way.
At first, the setup did not work properly because everyone was connected to the same Wi-Fi network, which made the connection very slow and laggy. We also tried using a hotspot, but it still didn’t working.
After around an hour, VCam finally connected on my phone. However, the app was still glitching and not running smoothly. When I left the app and went back in, I was logged out, so I could not continue testing it properly.
Even though the technical side was frustrating, the session was useful because it showed how virtual production depends not only on the software, but also on a stable network and reliable setup. It also helped me understand how VCam could be useful for animation and previsualisation when it works correctly.
During this week, we also had to choose which brief we wanted to develop for the submission at the end of the term. I decided to choose the first brief, Expanded Animation / Context and Practice.
I chose this brief because I want to explore TouchDesigner more deeply. I have not properly learned TouchDesigner before, so this project feels like a good opportunity to experiment with it and understand how it can be used in an animation or interactive context.
