United States

Manipulating human poses based on natural language is an emerging research field that has traditionally focused on coarse commands such as “walking” or “dancing.” However, fine-grained pose manipulation, like instructing “put both hands in front of the stomach,” remains underexplored. In this paper, we introduce PoseLLaVA, a pioneering model that integrates SMPL-based pose representations into the multimodal LLaVA framework. Through a novel pose encoder decoder mechanism, PoseLLaVA achieves precise alignment between pose, textual, and visual modalities, enabling detailed control over pose manipulation tasks. PoseLLaVA excels in three key tasks: pose estimation, generation, and adjustment, all driven by detailed language instructions. We further introduce a fine-grained pose adjustment dataset PosePart, where each sample contains an initial pose and a target pose, along with specific instructions for adjustments, mimicking the guidance a human instructor might provide. Extensive evaluations across these tasks demonstrate significant improvements over existing methods, including metrics such as MPJPE and PA-MPJPE, which measure SMPL reconstruction errors, and Recall rates, which assess feature alignment across modalities. Specifically, PoseLLaVA reduces MPJPE errors by more than 20% compared to state-of-the-art methods in pose adjustment and generation tasks. Additionally, we demonstrate the feasibility of combining PoseLLaVA with generative models, such as diffusion, for pose image editing, highlighting its potential applications in language-controlled pose manipulation.

AAAI 2025

PoseLLaVA: Pose Centric Multimodal LLM for Fine-Grained 3D Pose Manipulation

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



poster

Human pose estimation in videos remains a challenge, largely due to the reliance on extensive manual annotation of large datasets, which is expensive and labor-intensive. Furthermore, existing approaches often struggle to capture long-range temporal dependencies and overlook the complementary relationship between temporal pose heatmaps and visual features. To address these limitations, we introduce STDPose, a novel framework that enhances human pose estimation by learning spatiotemporal dynamics in sparsely-labeled videos. STDPose incorporates two key innovations: 1) A novel Dynamic-Aware Mask to capture long-range motion context, allowing for a nuanced understanding of pose changes. 2) A system for encoding and aggregating spatiotemporal representations and motion dynamics to effectively model spatiotemporal relationships, improving the accuracy and robustness of pose estimation. STDPose establishes a new performance benchmark for both video pose propagation (i.e., propagating pose annotations from labeled frames to unlabeled frames) and pose estimation tasks, across three large-scale evaluation datasets. Additionally, utilizing pseudo-labels generated by pose propagation, STDPose achieves competitive performance with only 26.7\% labeled data.

SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos

Tabular data, despite its prevalence in various industries, has been under-explored in deep learning research. Self-supervised learning (SSL) techniques have shown promise for pre-training deep neural networks (DNNs) on tabular data. However, their full potential is yet to be realized due to the challenge of designing appropriate data augmentations for this type of data. Unlike image and language domains, where the success of SSL heavily relies on the inherent structure of the data, such as spatial relationships in images or semantic relationships in text, tabular data lacks such explicit structure. This lack of clear structure makes traditional input-level augmentations, like modifying or removing features, less effective, as they struggle to balance preserving critical information with introducing useful variability. In response to these challenges, we propose RaTab, a novel method that shifts the focus from input-level to representation-level augmentation using matrix factorization, specifically truncated SVD. This shift allows for the preservation of essential data structures while generating a richer diversity of representations with dropout technique. RaTab enhances the effectiveness of SSL for tabular data by focusing on the representation space and utilizing truncated SVD, resulting in significant improvements.

Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data

In social networks, people influence each other through social links, which can be represented as propagation among nodes in graphs. Influence minimization (IM) is the problem of manipulating the structures of an input graph (e.g., removing edges) to reduce the propagation among nodes. IM can represent time-critical real-world applications, such as rumor blocking, but IM is theoretically difficult and computationally expensive. Moreover, the discrete nature of IM hinders the usage of powerful machine learning techniques, which requires differentiable computation. In this work, we propose DiffIM, a novel method for IM with two differentiable schemes for acceleration: (1) surrogate modeling for efficient influence estimation, which avoids time-consuming simulations (e.g., Monte Carlo), and (2) the continuous relaxation of decisions, which avoids the evaluation of individual discrete decisions (e.g., removing an edge). We further propose a third accelerating scheme, gradient-driven selection, that chooses edges instantly based on gradients without optimization (spec., gradient descent iterations) on each test instance. Through extensive experiments on real-world graphs, we show that each proposed scheme significantly improves speed with little (or even no) IM performance degradation. Our method is Pareto-optimal (i.e., no baseline is faster and more effective than it) and typically  several orders of magnitude (spec., up to 15,160X) faster than the most effective baseline, while being more effective.

DiffIM: Differentiable Influence Minimization with Surrogate Modeling and Continuous Relaxation

With the emerging of huge amount of unlabeled data, unsupervised out-of-distribution (OOD) detection is vital for ensuring the reliability of graph neural networks (GNNs) by identifying OOD samples from in-distribution (ID) ones during testing, where encountering novel or unknown data is inevitable. Existing methods often suffer from compromised performance due to redundant information in graph structures, which impairs their ability to effectively differentiate between ID and OOD data. To address this challenge, we propose SEGO, an unsupervised framework that integrates structural entropy into OOD detection regarding graph classification. Specifically, within the architecture of contrastive learning, SEGO introduces an anchor view in the form of coding tree by minimizing structural entropy. The obtained coding tree effectively removes redundant information from graphs while preserving essential structural information, enabling the capture of distinct graph patterns between ID and OOD samples. Furthermore, we present a multi-grained contrastive learning scheme at local, global, and tree levels using triplet views, where coding trees with essential information serve as the anchor view. Extensive experiments on real-world datasets validate the effectiveness of SEGO, demonstrating superior performance over state-of-the-art baselines in OOD detection. Specifically, our method achieves the best performance on 9 out of 10 dataset pairs, with an average improvement of 3.7\% on OOD detection datasets, significantly surpassing the best competitor by 10.8\% on the FreeSolv/ToxCast dataset pair.

Structural Entropy Guided Unsupervised Graph Out-Of-Distribution Detection

Data augmentation methods, especially SoTA interpolation-based methods such as Fair Mixup, have been widely shown to increase model fairness. However, this fairness is evaluated on metrics that do not capture model uncertainty and on datasets with only one, relatively large, minority group. As a remedy, multicalibration has been introduced to measure fairness while accommodating uncertainty and accounting for multiple minority groups. However, existing methods of improving multicalibration involve reducing initial training data to create a holdout set for post-processing, which is not ideal when minority training data is already sparse. This paper uses multicalibration to more rigorously examine data augmentation for classification fairness. We stress-test four versions of Fair Mixup on two structured data classification problems with up to 81 marginalized groups, evaluating multicalibration violations and balanced accuracy. We find that on nearly every experiment, Fair Mixup $\textit{worsens}$ baseline performance and fairness, but the simple vanilla Mixup $\textit{outperforms}$ both Fair Mixup and the baseline, especially when calibrating on small groups. $\textit{Combining}$ vanilla Mixup with multicalibration post-processing, which enforces multicalibration through post-processing on a holdout set, further increases fairness.

Who’s the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration

The unwavering disparity in labeled resources between resource-rich languages and those considered low-resource remains a significant impediment for Large Language Models (LLMs). Recent strides in cross-lingual in-context learning (X-ICL), mainly through semantically aligned examples retrieved from multilingual pre-trained transformers, have shown promise in mitigating this issue. However, our investigation reveals that LLMs intrinsically reward in-language semantically aligned cross-lingual instances over direct cross-lingual semantic alignments, with a pronounced disparity in handling time-sensitive queries in X-ICL setup. Such queries demand sound temporal reasoning ability from LLMs, yet the advancements have predominantly focused on English. This study aims to bridge this gap by enhancing temporal reasoning capabilities in low-resource languages. To this end, we introduce a temporal reasoning dataset, $\texttt{mTEMPREASON}$ aimed at the varied degrees of low-resource languages and propose a novel method, Cross-Lingual Time-Sensitive Semantic Alignment ($\texttt{CLiTSSA}$), to elevate temporal reasoning in these contexts. To facilitate this, we construct an extension of $\texttt{mTEMPREASON}$ comprising pairs of parallel cross-language temporal queries along with their anticipated in-language semantic similarity scores. Our empirical evidence underscores the superior performance of $\texttt{CLiTSSA}$ compared to established baselines across three languages- Romanian, German, and French, encompassing three temporal tasks and including a diverse set of four contemporaneous LLMs. This marks a significant step forward in addressing the resource disparity in the temporal reasoning context across languages.

Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languages

Prompt learning has emerged as a promising method for adapting pre-trained visual-language models (VLMs) to a wide range of downstream tasks. Advanced
prompt learning methods usually initialize and optimize the context, e.g., ``a photo of a []'', for downstream task adaptation. While optimizing the context can be effective for improving performance on specific tasks, it can often lead to poor generalization performance on unseen classes or datasets sampled from different distributions. It may be attributed to that textual prompts tend to overfit downstream data distributions, leading to the forgetting of hand-crafted general textual knowledge having a strong generalization ability.
In this paper, we propose a  novel method called Similarity Paradigm with Textual Regularization  (SPTR) for prompt learning without forgetting. SPTR is a two-pronged design, which is an inseparable framework.  1) To   avoid forgetting general textual knowledge, we introduce an adaptive  plan  as a textual regularization to ensure approximation with  hand-crafted features and tuning textual features. 2) To improve model robustness for generalization, we propose a  similarity paradigm for natural alignment score and adversarial alignment score. Both modules share a common objective in addressing generalization issues, aiming to maximize the generalization capability derived from hand-crafted prompts. 4 representative tasks
(i.e.,  non-generalization classification, base-to-novel generalization, cross-dataset generalization,  domain shifts generalization) across  15 datasets demonstrate that SPTR outperforms existing prompt learning methods.

A Similarity Paradigm through Textual Regularization without Forgetting

Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. 
Most model editing methods are solely designed for single-time use and result in a significant forgetting effect in lifelong editing scenarios, where sequential edits are conducted over time.
Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.
However, these methods lack robustness to minor input variations due to the discrete mapping between data and parameters. 
To overcome this challenge, we propose ELDER, a novel approach to create a continuous association between data and adapters. 
ELDER integrates multiple LoRAs through a router network and is trained to establish a smooth data-adapter association, thereby enhancing the edit robustness and generalization of semantically equivalent inputs.
To ensure inputs containing the same knowledge will be processed by the same LoRAs, we design a novel loss to guide the model link LoRA allocations with edit knowledge.
Furthermore, we propose a deferral mechanism to retain the original LLM capabilities post-edit. 
Extensive experiments on GPT-2 XL and LLaMA2-7B demonstrate that ELDER effectively edits models in the lifelong setting, outperforming eight baselines while exhibiting strong scalability and preserving LLMs' general abilities on downstream tasks.

ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA

Currently, 3D rendering for large-scale free camera trajectories, namely, arbitrary input camera trajectories, poses significant challenges: 1) The distribution and observation angles of the cameras are irregular, and various types of scenes are included in the free trajectories; 2) Processing the entire point cloud and all images at once for large-scale scenes requires a substantial amount of GPU memory. This paper presents a Toy-GS method for accurately rendering large-scale free camera trajectories. Specifically, we propose an adaptive spatial division approach for free trajectories to divide cameras and the sparse point cloud of the entire scene into various regions according to camera poses. Training each local Gaussian in parallel for each area enables us to concentrate on texture details and minimize GPU memory usage. Next, we use the multi-view constraint and position-aware point adaptive control (PPAC) to improve the rendering quality of texture details. In addition, our regional fusion approach combines local and global Gaussians to enhance rendering quality with an increasing number of divided areas. Extensive experiments have been carried out to confirm the effectiveness and efficiency of Toy-GS, leading to state-of-the-art results on two public large-scale datasets as well as our SCUTic dataset. Our proposal demonstrates an enhancement of 1.19 dB in PSNR and conserves 7 G of GPU memory when compared to various benchmarks.

Toy-GS: Assembling Local Gaussians for Precisely Rendering Large-Scale Free Camera Trajectories

Multi-task learning (MTL) is widely utilized across a variety of real-world applications, including recommendation systems. For instance, in the field of e-commerce, MTL is commonly employed to simultaneously model click, conversion, and user dwelling time. Among a various of MTL models, the Multi-gate Mixture-of-Experts (MMoE) has gained significant popularity. However, MMoE suffers from the polarization issue during training, where the weights of certain experts tend to converge towards 0. To address this issue, we propose a novel method called $\textbf{B}$agging-$\textbf{E}$xpert $\textbf{net}$work (BEnet) for multi-task learning. BEnet effectively mitigates the problem of polarization and achieves excellent performance in multi-task learning. It incorporates a bagging layer and an attention mechanism to encourage experts focusing on diverse knowledge domains. Simultaneously, polarization is avoided as different experts execute respective duties and specialize in distinct domains. Experimental results on real-world datasets demonstrate that BEnet has strong robustness and outperforms other state-of-the-art (SOTA) MTL methods.

Downloads

Next from AAAI 2025

SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos

Similar lecture

GRPose: Learning Graph Relations for Human Image Generation with Pose Priors

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2025

SpatioTemporal Learning for Human Pose Estimation in Sparsely-Labeled Videos

Similar lecture

GRPose: Learning Graph Relations for Human Image Generation with Pose Priors

Downloads