My research explores how computation can broaden the ways we create, simulate, understand, and interact with visual media.
Selected research · Most recent first
012026
MemeMatch: A Large-Scale Dual-Context Multimodal Dataset and Retrieval System for Internet Memes
Do Tri An Le · Donát Ákos Köller · Qixin Deng* · Roland Molontay*
Abstract
We introduce MemeMatch, a large-scale multimodal meme dataset and retrieval system that bridges meme collection, annotation, and analysis in a unified pipeline. The dataset contains nearly one million image-with-text memes from Reddit’s r/Memes (2018–2023) and ImgFlip, with rich metadata. Each meme is decomposed into two semantic contexts: local context, capturing the editable text payload (overlay text and title), and global context, capturing the underlying visual substrate or template semantics. Both are enriched with transformer-based annotations, including 14-dimensional sentiment and emotion vectors, BERTopic-derived topics, and zero-shot usage-intent labels. This structured representation supports exploratory analysis and context-aware retrieval by natural language or image query.
We present a structure-informed framework for simplifying hex-dominant meshes by reducing non-hexahedral cells. The method extracts and analyzes mesh sub-structures, decomposes complex configurations, ranks candidate collapses through a neighborhood relation graph, and applies quality-aware smoothing. Evaluations across meshes produced by several state-of-the-art techniques show consistent reductions in non-hex cells while preserving or improving mesh quality.
We present a secure desktop application for monitoring school lanyard-policy compliance while protecting student data and maintaining reliable operation. The Python and Tkinter system uses authenticated, encrypted Google Sheets synchronization, layered validation, administrative access controls, and local caching for offline continuity. Deployment in a middle-school environment demonstrates rapid synchronization, dependable recovery, and reduced administrative workload.
Robust Differentiable Sketch Rendering for Single-View 3D Reconstruction
Aobo Jin · Qixin Deng · Lei Si · Zhigang Deng
Abstract
We propose an end-to-end approach for constructing detailed 3D objects from a single-view sketch. A learned differentiable sketch renderer connects normal-map geometry with 2D strokes, enabling sketch-based losses to propagate through the reconstruction pipeline. Silhouette-derived confidence and regression-similarity losses improve frontal geometric detail, and comparisons demonstrate stronger reconstruction of plausible shapes without requiring semantic stroke annotations.
Advanced Facial Emotion Classification with 135 Classes for Enhanced Cybersecurity Applications
Gregory Powers · Aobo Jin · Abdul Basit Tonmoy · Haowei Cao · Hardik Gohel · Qixin Deng*
Abstract
We study fine-grained facial emotion recognition across 135 expression classes, moving beyond the seven or eight categories used by many existing systems. The proposed framework combines U-Net and ResNet architectures to preserve spatial cues while learning deep discriminative features. Experiments show improved accuracy over current baselines and demonstrate the value of high-dimensional emotion classification for behavior analysis, human-computer interaction, and cybersecurity applications.
Dense Crowd Motion Prediction through Density and Trend Maps
Tingting Wang · Qiang Fu* · Minggang Wang · Huikun Bi · Qixin Deng · Zhigang Deng
Abstract
We introduce density and trend maps as high-level representations for predicting group behavior and individual pedestrian motion from video. A density-map network estimates crowd state, while a prediction network forecasts future density and motion trends. These crowd-level predictions also improve trajectory-based forecasts for individual pedestrians, with experiments demonstrating robust performance against state-of-the-art motion-prediction methods.
We propose a conditional generative adversarial network architecture, called S2M-Net, to holistically synthesize realistic three-party conversational animations from acoustic speech input and speaker marking. A generator encodes speech features, transforms the latent representation into gesture-kinematics space, and decodes three-party conversational motion, while a discriminator evaluates whether generated motion sequences are real or synthetic. Quantitative and qualitative evaluations and paired-comparison user studies compare the system with the state of the art.
End-to-End 3D Face Reconstruction with Expressions and Specular Albedos from Single In-the-wild Images
Qixin Deng · Binh H. Le · Aobo Jin · Zhigang Deng*
Abstract
Recovering 3D face models from in-the-wild face images has numerous potential applications, but complex lighting effects—including specular lighting, shadows, and occlusions—remain challenging. We propose a convolutional-neural-network framework that regresses dense 3D shape, head pose, expression, diffuse albedo, specular albedo, and corresponding lighting conditions from a single image. Novel hybrid loss functions disentangle identity, expression, pose, albedo, and lighting, and experiments demonstrate quantitative and qualitative improvements over prior methods.
A Live Speech Driven Avatar-mediated Three-party Telepresence System: Design and Evaluation
Aobo Jin* · Qixin Deng* · Zhigang Deng†
Abstract
We present a live speech-driven, avatar-mediated, three-party telepresence system through which three distant users, embodied as avatars in a shared 3D virtual world, can perform natural three-party telepresence without tracking devices. Live speech drives head, eye, lip, torso, and hand motion in real time, while a cloud server transmits and synchronizes motion and speech. A formal user study reports a measurably better telepresence experience than Second Life and Skype.
Plausible 3D Face Wrinkle Generation Using Variational Autoencoders
Qixin Deng · Luming Ma · Aobo Jin · Huikun Bi · Binh Huy Le · Zhigang Deng*
Abstract
We propose an end-to-end system that augments coarse-scale 3D faces with synthesized fine-scale geometric wrinkles. By formulating wrinkle generation as a supervised generation task, the method models a continuous wrinkle space with a compact generative model and produces plausible wrinkles through sampling and interpolation. A complete transfer pipeline applies synthesized wrinkles across faces with different shapes and topologies while preserving plausible detail across expressions and animation.
A Deep Learning-Based Model for Head and Eye Motion Generation in Three-party Conversations
Aobo Jin · Qixin Deng · Yuting Zhang · Zhigang Deng
Abstract
We propose a deep-learning approach for generating realistic three-party head and eye motion from acoustic speech input and speaker marking. Using a high-quality three-party conversational-motion dataset, the model predicts dynamic eye and head directions for all interlocutors. Combined with lip-sync and speech-driven hand and body gesture generation, the approach produces realistic three-party conversational animations, supported by experiments and comparative user studies.