To open and close this bar, use optionalt+/

Projects

Containerized Digital Twin Platform for Indoor Environmental Quality Monitoring

This project develops a containerized Digital Twin platform that integrates IoT sensors, 3D visualization, and advanced analytics for comprehensive Indoor Environmental Quality (IEQ) monitoring and Indoor Air Quality (IAQ) prediction. Its modular, Docker-based architecture enhances scalability, simplifies deployment, and enables cost-effective operation across diverse environments.

A proof-of-concept in a university campus office demonstrates real-time monitoring with descriptive analytics visualizing thermal, acoustic, and lighting comfort, as well as air quality, offering remote stakeholders holistic IEQ assessments. Predictive analytics enable 72-hour PM₂.₅ forecasting with an RMSE of 4.884, providing actionable insights for proactive environmental control.

The research project is led by Borja García de Soto, with Zihao Zheng overseeing system design and Juan Diego Castaño Molina contributing to platform development and predictive analytics. With the collaboration and support from SHORES and CITIES.

On-Device Large Language Models for Construction Robotics

This project investigates the deployment of Large Language Models (LLMs) directly on construction robots, enabling natural and intuitive human–robot interaction without reliance on cloud-based processing. In construction, where automation adoption remains limited and workforce training in robotics is scarce, locally executed LLMs offer a practical and secure alternative to remote systems.

The proposed framework performs multimodal processing, including speech and vision, entirely on-device, reducing latency, connectivity issues, and privacy risks associated with cloud dependence. Through optimized model efficiency and hardware integration, the system achieves real-time responsiveness and greater autonomy in bandwidth-constrained environments.

A case study demonstrates the Local Intelligent Safety Assistant (LISA), which uses few-shot prompting to inspect work areas, interact with workers, and promote on-site safety compliance. Results show that on-device LLMs can deliver a robust, responsive, and accessible interface for human–robot collaboration in real-world construction settings.

The research project is led by Borja García de Soto, Nas Gopee, and Samuel Prieto. With the collaboration and support from SHORES.

AI-Driven Platform for Cyber Risk Management in Construction Projects

This project develops an end-to-end cybersecurity management platform tailored for large, interconnected construction projects. Powered by a patent-pending 16-factor Mixture-of-Experts risk engine and a Parallel Group-Factor Greedy optimizer, the system analyzes BIM models, IoT data streams, supply-chain records, and project-management data to quantify cyber risk in real time.

Construction teams can upload design and scheduling artifacts to automatically generate a dynamic risk scorecard ranking vulnerabilities by probability, impact, and mitigation cost. The integrated optimizer identifies the most cost-effective control set to meet target residual-risk thresholds, while a conversational LLM assistant explains trade-offs in plain language.

Interactive dashboards monitor risk at project, portfolio, and enterprise levels, and a secure compliance vault centralizes evidence for ISO 27001 and NIST submissions. By unifying fragmented cyber tasks into a transparent, data-driven workflow, the platform enables faster, safer, and more cost-efficient decision-making.

The research project is led by Borja García de Soto and Dongchi Yao, and the support of Begad Elfackrany. With the collaboration and support from CCS-AD and SHORES.

A Crowdsourcing Approach to Construction Cybersecurity

This project tackles the growing cybersecurity risks in the construction sector through a crowdsourcing methodology powered by an AI-driven data analysis pipeline. The approach converts collective human knowledge into actionable cybersecurity insights for management and decision-making.

Two case studies support the data collection process. The first, Hack My Robot, is a competition where students attempt to compromise the data and operations of a robotic system under realistic construction-site conditions. The second, the Crowdsourcing Platform, is an online tool that aggregates technology-specific cybersecurity expertise from industry professionals.

Collected data undergoes semantic segmentation, LLM-based topic modeling, labeling, and summarization, producing structured knowledge on threats, vulnerabilities, and mitigations. Results demonstrate that crowdsourced intelligence can yield coherent and meaningful cybersecurity insights tailored to construction contexts.

The research project is led by Borja García de Soto and Semih Sonkor. With the collaboration and support from CCS-AD and SHORES.

Personal Autonomous Construction Assistant (PACA)

This project introduces the Personal Autonomous Construction Assistant (PACA), a multimodal natural Large Language Model (mmLLM)-based interface designed to enable seamless and intuitive communication between construction workers and robots. Integrating robots into construction workflows is challenging due to the dynamic nature of sites and the diversity of the workforce, many of whom lack formal robotics training.

PACA incorporates an Automatic Speech Recognition (ASR) engine to process spoken instructions in multiple languages and uses a Large Language Model (LLM) to interpret user intent and generate context-aware responses. When appropriate, the LLM translates commands into robot actions, using image-based reasoning to enhance situational awareness and decision-making.

A Text-to-Speech (TTS) engine provides natural spoken feedback, ensuring fully bidirectional and accessible interaction for on-site operators. PACA represents a major step toward human-centered, intelligent, and language-driven robotic systems for construction environments.

The research project is led by Borja García de Soto and Samuel Prieto. With the collaboration of the KINESIS CTP and the support from SHORES.

Autonomous Robotic Platform for Construction Site Data Collection

This project develops an autonomous robotic platform designed to automate and enhance data collection on construction sites, addressing the limitations of traditional manual inspections that are time-consuming and prone to human error.

The platform features holonomic wheels for precise maneuverability in cluttered environments and integrates multiple sensors for autonomous navigation, including a 120 m LiDAR, Inertial Measurement Unit (IMU), depth camera, and wheel encoders. It carries a long-range 3D scanner capable of capturing point clouds enriched with RGB, thermal, and reflectance information.

All components are managed through the Robot Operating System (ROS), which synchronizes data streams and ensures efficient coordination across sensor SDKs. This configuration enables reliable, repeatable, and high-resolution spatial data capture, supporting advanced analysis for progress tracking, safety assessment, and quality assurance.

The research project is led by Borja García de Soto, Nas Gopee, and Samuel Prieto. With the collaboration of the KINESIS CTP and the support from SHORES.

Autonomous AI-powered inspection of buildings (ABECIS)

This project presents a methodology to use UAVs and deep learning technology to conduct automated inspections for cracks on high-rise buildings to improve process efficiency and safety of workers. The proposed methodology is divided into four components: 1) Developing a UAV system to capture the exterior wall images of the building in an autonomous way, 2) Collecting data, 3) Processing and analyzing the images captured for cracks using deep learning, and 4) Rendering the identified locations of the cracks on a 3D model of the building, constructed using photogrammetry, for clear visualization. Data Collection, Image-Analysis, and Photogrammetry are done using publicly available open-source deep learning and simulation technologies. The generated code for the UAV simulation and the crack detection algorithm with the pre-trained data model is released on GitHub. Prelimiray work can be found in the paper titled "ABECIS: an Automated Building Exterior Crack Inspection System using UAVs, Open-Source Deep Learning and Photogrammetry" presented at ISARC 2021 (10.22260/ISARC2021/0086). This research project is led by Borja and Pi

Spatio-temporal modeling and 4D simulation

During the construction of high-rises, construction workers and small and medium-sized materials are typically transported using temporary elevators. Temporary elevator planning plays a positive role in the successful completion of a construction project. However, conventional temporary elevator planning has limitations. On the one hand, the optimization of temporary elevator planning is rarely considered from a spatio-temporal perspective. On the other hand, the display of temporary elevator planning mainly adopts abstract (e.g., text), low-dimensional (e.g., 2D drawing), and static (e.g., 3D model) ways, but such practices are not easy to be fully understood by the project team. To address these limitations, this research proposes a framework for the spatio-temporal planning of temporary elevators considering a virtual reality environment, making temporary elevator planning more effective and intuitive. This research project is led by Borja and Keyi.

Two-step projection of multi-label image classification and masking of point clouds

This project uses an extensive image database with more than 11,000 images taken from various construction sites covering different stages of construction processes. Photos are multi-labeled into 13 different coarse categories. The developed model is an integral part of a point cloud segmentation pipeline where 2D imagery is extracted from the 3D point cloud environment using a set of estimated camera poses. The extracted 2D images are segmented and masked using the developed model into the pre-defined coarse categories. The semantically segmented images annotated with different color masking are projected back to the point cloud carrying additional segment attributes. The proposed course labels in this project includes HVAC ducts, pipes, conduits and wiring systems, formwork, drywall components, etc. These specific course segments will be refined to recognize detailed construction elements and detect defects, delays, safety hazards, and construction site working space limitations. This research project is led by Borja and Eyob.

Game engines and reinforcement learning multi-agent-based for bricklaying construction

This research proposes a framework consisting of a communication interface between Unity and ROS for real-world distributed robotic construction. It deploys reinforcement learning in a game engine-based simulation. The Unity scene provides the environment in which agents observe, act, learn and get feedback. We draw from state-of-the-art reinforcement learning techniques for multi-agent-based execution plan generation by establishing connections between the python API and python trainer with the environment. All the learning algorithms have been set up with the TensorFlow platform to communicate with the Unity model. Then Unity passes all the information collected to ROS, namely, the poses of the robot, target object, target location, and the motion plan. In turn, ROS returns a trajectory message to Unity corresponding to the real robot feedback for further simulation of the remaining task. This research project is led by Borja and Xinghui.

IFC Extensions for Inspection of Formwork Systems Using Robots

Quality inspection of formwork systems is a tedious and, in some cases, difficult process and is easily affected by the knowledge, skill, and even the worker’s mood during the inspection, making the collection of data subjective and inconsistent. To address these weakness, the application of robots equipped with sensors for the quality inspection of formwork systems is a promising solution. However, the current Industry Foundation Classes (IFC) schema does not include relevant entities and attributes, which limits its digitalization and automation transformation and upgrading. With this background, this paper aims to extend the IFC schema for accommodating the needs of the quality inspection of formwork systems using robots equipped with sensors. A prototype system based on the extended IFC schema is developed to verify the feasibility and effectiveness of the concept. This research project is led by Borja and Keyi.

Cybersecurity aspects of Operational Technology on construction sites

Digitization in the construction industry is increasing with the integration of information technologies (IT) and operational technologies (OT) into different activities. The levels of OT to control and monitor site activities utilizing (semi)autonomous and remotely controlled machines raise cybersecurity concerns. Safety issues are magnified given the collaborative work of humans and machines/equipment. This study's motivation is to understand the current state of the art and identify gaps to suggest future directions regarding OT in construction from the cybersecurity perspective. To achieve that, a bibliometric analysis is conducted. The analysis utilizes the Scopus database to retrieve related publications and VOSviewer software to visualize bibliometric networks. Main research themes are identified, and each theme is reviewed from a cybersecurity perspective. The gaps in the existing literature and suggestions for further research related to the cybersecurity of OT in construction are provided. This research project is led by Borja and Semih and funded by the Center for Cyber Security at NYUAD (CCS-AD).