United States

Since the recent studies  (KDD&#39;20) done by Krichene and Rendle on the sampling based top-k evaluation metric for recommendation,  there have been a lot of debate on the validity of using sampling for evaluating recommendation algorithms. Though their work and the recent work done by Li et. al. (KDD&#39;20) have proposed some basic approach for mapping the sampling based metrics to their counter-part in the global evaluation which uses the entire dataset, there is still lack of understanding how sampling should be used for recommendation evaluation, and the proposed approaches either are rather ad-hoc or can only work on simple metrics, like Recall/Hit-Ratio. In this paper, we introduce some principled approach to derive the estimators of top-k metric based on sampling. Our approaches utilize the weighted MLE and maximal entropy approach to recover the global rank distribution and then utilize that for estimation. The experimental results shows significant advantages of using our approaches for evaluating recommendation algorithms based on top-k metrics.

AAAI 2021

On Estimating Recommendation Evaluation Metrics under Sampling

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-23 is the Thirty-Seventh AAAI Conference on Artificial Intelligence. The theme of this conference is to create collaborative bridges within and beyond AI. Like previous AAAI conferences, AAAI-23 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and two new activities: a Bridge Program and a Lab Program. Many of these activities are tailored to the theme of bridges and all are selected according to the highest standards, with additional programs for students and young researchers. 
AAAI is providing you with a conference planner, which you can use to help organize your itinerary of activities. This includes talks to attend in person, talks to attend remotely, breaks with colleagues and your site seeing activities. To access this conference planner, please go to [https://aaai-2023.takemobi.io](https://aaai-2023.takemobi.io).

In order to access this site, you need to register. If you haven't already, please register [here](https://aaai.org/Conferences/AAAI-23/registration/).


AAAI 2023

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines.

Since the recent studies  (KDD'20) done by Krichene and Rendle on the sampling based top-k evaluation metric for recommendation,  there have been a lot of debate on the validity of using sampling for evaluating recommendation algorithms. Though their work and the recent work done by Li et. al. (KDD'20) have proposed some basic approach for mapping the sampling based metrics to their counter-part in the global evaluation which uses the entire dataset, there is still lack of understanding how sampling should be used for recommendation evaluation, and the proposed approaches either are rather ad-hoc or can only work on simple metrics, like Recall/Hit-Ratio. In this paper, we introduce some principled approach to derive the estimators of top-k metric based on sampling. Our approaches utilize the weighted MLE and maximal entropy approach to recover the global rank distribution and then utilize that for estimation. The experimental results shows significant advantages of using our approaches for evaluating recommendation algorithms based on top-k metrics.

technical paper

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

Making predictions that are fair with regard to protected attributes (race, gender, age, etc.) has become an important requirement for classification algorithms. Existing techniques derive a fair model from sampled labeled data relying on the assumption that training and testing data are identically and independently drawn (iid) from the same distribution. In practice, distribution shift can and does occur between training and testing datasets as the characteristics of individuals interacting with the machine learning system change. We investigate fairness under covariate shift, a relaxation of the iid assumption in which the inputs or covariates change while the conditional label distribution remains the same. We seek fair decisions under these assumptions on target data with unknown labels. We propose an approach that obtains the predictor that is robust to the worst-case testing performance while satisfying target fairness requirements and matching statistical properties of the source data. We demonstrate the benefits of our approach on benchmark prediction tasks.

Robust Fairness under Covariate Shift

Multi-task learning has increased in importance due to its superior performance by learning multiple different tasks simultaneously and its ability to perform several different tasks using a single model. In medical phenotyping, task labels are costly to acquire and might contain a certain degree of label noise. This decreases the efficiency of using additional human labels as auxiliary tasks when applying multi-task learning to medical phenotyping. In this work, we proposed an effective multi-task learning framework, CO-TASK, to boost multi-task learning performance by generating auxiliary tasks through COmbination of TASK Labels. The proposed CO-TASK framework generates auxiliary tasks without additional labeling effort, is robust to a certain degree of label noise, and can be applied in parallel with various multi-task learning techniques.

We evaluated our performance using the CIFAR-MTL dataset and demonstrated its effectiveness in medical phenotyping using two large-scale ECG phenotyping datasets, an 18 diseases multi-label ECG-P18 dataset and an echocardiogram diagnostic from electrocardiogram dataset ECG-EchoLVH. On the CIFAR-MTL dataset, we doubled the average per-task performance gain of the multi-task learning model from 4.38% to 9.78%. With the proposed task-aware imbalance data sampler, the CO-TASK framework can effectively deal with the different imbalance ratios for the different tasks in electrocardiogram phenotyping datasets. The proposed framework combined with noisy annotations as minor tasks increased the sensitivity by 7.1% compared to the single-task model while maintaining the same specificity as the doctor annotations on the ECG-EchoLVH dataset.

Boosting Multi-task Learning through Combination of Task Labels - with Applications in ECG Phenotyping

Identifying causal effects from observational data is a pervasive challenge found throughout the empirical sciences. Very general methods have been developed to decide the identifiability of a causal quantity from a combination of observational data and causal knowledge about the underlying system. In practice, however, there are still challenges to estimating identifiable causal functionals from finite samples. Recently, a method known as double/debiased machine learning (DML) (Chernozhukov et al. 2018) has been proposed to learn parameters leveraging modern machine learning techniques, which is both robust to model misspecification and bias-reducing. Still, DML has only been used for causal estimation in settings when the back-door condition (also known as conditional ignorability) holds. In this paper, we develop a new, general class of estimators for any identifiable causal functionals that exhibit DML properties, which we name DML-ID. In particular, we introduce a complete identification algorithm that returns an influence function (IF) for any identifiable causal functional. We then construct the DML estimator based on the derived IF. We show that DML-ID estimators hold the key properties of debiasedness and doubly robustness. Simulation results corroborate with the theory.

Estimating Identifiable Causal Effects through Double Machine Learning

Data heterogeneity has been identified as one of the key features in federated learning but often overlooked in the lens of robustness to adversarial attacks. This paper focuses on characterizing and understanding its impact on backdooring attacks in federated learning through comprehensive experiments using synthetic and the LEAF benchmarks. The initial impression driven by our experimental results suggests that data heterogeneity is the dominant factor in the effectiveness of attacks and it may be a redemption for defending against backdooring as it makes the attack less efficient, more challenging to design effective attack strategies, and the attack result also becomes less predictable. However, with further investigations, we found data heterogeneity is more of a curse than a redemption as the attack effectiveness can be significantly boosted by simply adjusting the client-side backdooring timing. More importantly, data heterogeneity may result in overfitting at the local training of benign clients, which can be utilized by attackers to disguise themselves and fool skewed-feature based defenses. In addition, effective attack strategies can be made by adjusting attack data distribution. Finally, we discuss the potential directions of defending the curses brought by data heterogeneity. The results and lessons learned from our extensive experiments and analysis offer new insights for designing robust federated learning methods and systems.

Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning

Link prediction (LP) algorithms propose to each node a ranked list of nodes that are currently non-neighbors, as the most likely candidates for future linkage. Owing to increasing concerns about privacy, users (nodes) may prefer to keep some of their connections protected or private. Motivated by this observation, our goal is to design a differentially private LP  algorithm, which trades off between privacy of the protected node-pairs and the link prediction accuracy. More specifically, we first propose a form of differential privacy on graphs, which models the privacy loss only of those node-pairs which are marked as protected. Next, we develop DPLP, a learning to rank algorithm, which applies a monotone transform to base scores from a non-private LP system, and then adds noise.  DPLP is trained with a privacy induced ranking loss, which optimizes the ranking utility for a given maximum allowed level of privacy leakage of the protected node-pairs. Under a recently introduced latent node embedding model, we present a formal trade-off between privacy and LP utility. Extensive experiments with several real-life graphs and several LP heuristics show that DPLP can trade off between privacy and predictive performance more effectively than several alternatives.

Differentially Private Link Prediction with Protected Connections

Learning causal models with latent variables from observational and experimental data is an important problem. In this paper we present a polynomial-time algorithm that PAC learns the structure and parameters of a rooted  tree-structured causal network of bounded degree where the internal nodes of the tree cannot be observed or manipulated. Our algorithm is the first of its kind to provably learn the structure and parameters of tree-structured causal models with latent internal variables from random examples and active experiments.

PAC Learning of Causal Trees with Latent Variables

Competing Risk Analysis (CRA) aims at the correct estimation of the marginal probability of occurrence of an event in the presence of competing events. Many of the statistical approaches developed for CRA are limited by strong assumptions about the underlying stochastic processes. To overcome these issues and to handle censoring, machine learning approaches for CRA have designed specialized cost functions. However, these approaches are not generalizable, and are computationally expensive. This paper formulates CRA as a cause-specific regression problem and proposes DeepPseudo models, which use simple and effective feed-forward deep neural networks, to predict the cumulative incidence function (CIF) using Aalen-Johansen estimator-based pseudo values. DeepPseudo models capture the time-varying covariate effect on CIF while handling the censored observations. We show how DeepPseudo models can address co-variate dependent censoring by using modified pseudo values. Experiments on real and synthetic datasets demonstrate that our proposed models obtain promising and statistically significant results compared to the state-of-the-art CRA approaches. Furthermore, we show that explainable methods such as Layer-wise Relevance Propagation can be used to interpret the predictions of our DeepPseudo models.

DeepPseudo: Pseudo Value Based Deep Learning Models for Competing Risk Analysis

After observing a snapshot of a social network, a link prediction (LP) algorithm identifies node pairs between which new edges will likely materialize in future. Most LP algorithms estimate a score for currently non-neighboring node pairs, and rank them by this score.   Recent LP systems compute this score by comparing dense, low dimensional vector representations of nodes.  Graph neural networks (GNNs), in particular graph convolutional networks (GCNs), are popular examples.  For two nodes to be meaningfully compared, their embeddings should be indifferent to reordering of their neighbors.  GNNs typically use simple, symmetric set aggregators to ensure this property, but this design decision has been shown to produce representations with limited expressive power.  Sequence encoders are more expressive, but are permutation sensitive by design.  Recent efforts to overcome this dilemma turn out to be unsatisfactory for LP tasks. In response, we propose PermGNN, which aggregates neighbor features using a recurrent, order-sensitive aggregator and directly minimizes an LP loss while it is `attacked' by adversarial generator of neighbor permutations.  PermGNN has superior expressive power compared to earlier GNNs.  Next, we devise an optimization framework to map PermGNN's node embeddings to a suitable locality-sensitive hash, which speeds up reporting the top-K most likely edges for the LP task. Our experiments on diverse datasets show that PermGNN outperforms several state-of-the-art link predictors by a significant margin, and can predict the most likely edges fast.

Adversarial Permutation Guided Node Representations for Link Prediction

As 3D point clouds become the representation of choice for multiple vision and graphics applications, such as autonomous driving, robotics, etc., the generation of them by deep neural networks has attracted increasing attention in the research community. Despite the recent success of deep learning models in classification and segmentation, synthesizing point clouds remains challenging, especially from a single image. State-of-the-art (SOTA) approaches can generate a point cloud from a hidden vector, however, they treat 2D and 3D features equally and disregard the rich shape information within the 3D data. In this paper, we address this problem by integrating image features with 3D prototype features. Specifically, we propose to learn a set of 3D prototype features from a real point cloud dataset and dynamically adjust them through the training. These prototypes are then integrated with incoming image features to guide the point cloud generation process. Experimental results show that our proposed method outperforms SOTA methods on single image based 3D reconstruction tasks.

Single View Point Cloud Generation via Unified 3D Prototype

Social recommendation task aims to predict users' preferences over items with the incorporation of social connections among users, so as to alleviate the sparse issue of collaborative filtering. While many recent efforts show the effectiveness of neural network-based social recommender systems, several important challenges have not been well addressed yet: (i) The majority of models only consider usersâ€™ social connections, while ignoring the inter-dependent knowledge across items; (ii) Most of existing solutions are designed for singular type of user-item interactions, making them infeasible to capture the interaction heterogeneity; (iii) The dynamic nature of user-item interactions has been less explored in many social-aware recommendation techniques. To tackle the above challenges, this work proposes a Knowledge-aware Coupled Graph Neural Network (KCGN) that jointly injects the inter-dependent knowledge across items and users into the recommendation framework. KCGN enables the high-order user- and item-wise relation encoding by exploiting the mutual information for global graph structure awareness. Additionally, we further augment KCGN with the capability of capturing dynamic multi-typed user-item interactive patterns. Experimental studies on real-world datasets show the effectiveness of our method against many strong baselines in a variety of settings. Source codes are available at: https://github.com/xhcdream/KCGN.

Downloads

Next from AAAI 2021

Robust Fairness under Covariate Shift

Similar lecture

Towards Reliable Item Sampling for Recommendation Evaluation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2021

Robust Fairness under Covariate Shift

Similar lecture

Towards Reliable Item Sampling for Recommendation Evaluation

Downloads