This week, we worked with RealityScan, which is a photogrammetry tool from Unreal Engine. The aim of the session was to scan a real object using photos and turn it into a 3D model.
I first tried to scan my nose spray. I took photos from different angles and imported them into RealityScan. However, the scan did not work very well. The software was able to detect some of the images and camera positions, but the final result failed and did not create a clear 3D model of the object.
After that, I tried again using my Ice Tea Peach bottle. This worked slightly better, and the software was able to create more visible geometry. However, the scan still was not good enough. Because the bottle was already half empty, some parts of it became slightly transparent, which made it harder for RealityScan to read the object properly. The reflective plastic and liquid inside also made the scan more difficult.
Once the scan was generated, I used a selection tool, similar to a lasso tool, to delete the unwanted geometry around the object. There were a lot of extra pieces and messy shapes around the scan, so cleaning it up was an important part of the process
From this process, I learned that photogrammetry works better with objects that have a clear surface, strong details, and no transparency or reflection. Smooth, shiny, or see-through objects can confuse the software because it struggles to match the same points across different photos.
Overall, this session showed me how real-life objects can be turned into digital 3D models, but also how sensitive the process is. Even small issues, such as transparency, reflections, bad lighting, or not enough angles, can affect the final result.
After this, we started working on our own TouchDesigner project. Our project idea was called Data Extraction. The concept was about how human behaviour can be transformed into data. We wanted to create an interactive system where movement would be detected and converted into visual information, such as numbers, labels, graphs, and digital feedback. The idea was that the participant would appear as if they were being analysed by a machine.
We began by exploring how to use tracking inside TouchDesigner. We worked with the MediaPipe plugin because we wanted to use body or pose tracking to detect movement. We also looked at using the NDI 6 tools, as the plan was originally to bring in a camera feed through NDI. However, we had several technical issues with connecting the camera input correctly. Some nodes were not showing the image as expected, and it was confusing to understand how to connect the different operator types together.
One issue we came across was the difference between TOPs, CHOPs, and DATs in TouchDesigner. Some nodes were video/image based, while others were data based, which meant they could not always be connected directly. This made the setup more complicated, especially when trying to connect the camera feed into the MediaPipe system. We also had problems with the image segmentation and pose tracking not producing the live output we expected.
Because the NDI/camera setup was taking too much time to troubleshoot, we decided to simplify the workflow and use the webcam directly instead. This allowed us to keep moving forward with the project rather than getting stuck on the technical setup. Using the webcam still worked for the concept because the main idea was about detecting movement and translating it into data.
After switching to the webcam, we focused more on how the project could function visually and conceptually. The system would take the participant’s movement and turn it into a kind of data interface. Visually, we imagined the screen showing surveillance-style graphics, labels, metrics, and fake analysis such as “Subject Detected,” “Movement Data Extracted,”. These labels helped communicate the idea that the system is reducing a real person into simplified categories.
After completing the MediaPipe setup, we moved on to the next step: adding line graphs to visualise the subject’s movement data. Since MediaPipe was already detecting the body and hands, we wanted to translate that tracking information into a more data-driven visual layer.
We focused mainly on the hand detection, using the position values from the tracked points to create live line graphs. These graphs respond to the subject’s movement in real time, showing how the hand position changes across the screen. This helped us make the project feel more like a system that is actively analysing and collecting behavioural data.
By adding these graphs, the visualisation became more dynamic. Instead of only showing the detected skeleton or tracking points, we could now show the movement being converted into data. This connects well with our concept of data extraction, where the subject is not just being observed, but also measured and categorised through visual information.