Search Results for author: George Karypis

Found 79 papers, 37 papers with code

Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics

1 code implementation EMNLP 2021 Saurav Manchanda, George Karypis

Quantitatively measuring the impact-related aspects of scientific, engineering, and technological (SET) innovations is a fundamental problem with broad applications.

Pre-training Differentially Private Models with Limited Public Data

no code implementations28 Feb 2024 Zhiqi Bu, Xinwei Zhang, Mingyi Hong, Sheng Zha, George Karypis

The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection.

Extreme Miscalibration and the Illusion of Adversarial Robustness

no code implementations27 Feb 2024 Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, George Karypis

Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify.

Adversarial Attack Adversarial Robustness

OpenTab: Advancing Large Language Models as Open-domain Table Reasoners

no code implementations22 Feb 2024 Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, George Karypis

Large Language Models (LLMs) trained on large volumes of data excel at various natural language tasks, but they cannot handle tasks requiring knowledge that has not been trained on previously.

Retrieval

SemPool: Simple, robust, and interpretable KG pooling for enhancing language models

no code implementations3 Feb 2024 Costas Mavromatis, Petros Karypis, George Karypis

Our method, termed SemPool, represents KG facts with pre-trained LMs, learns to aggregate their semantic information, and fuses it at different layers of the LM.

Question Answering

Zero redundancy distributed learning with differential privacy

no code implementations20 Nov 2023 Zhiqi Bu, Justin Chiu, Ruixuan Liu, Sheng Zha, George Karypis

Deep learning using large models have achieved great success in a wide range of domains.

Privacy Preserving

On the accuracy and efficiency of group-wise clipping in differentially private optimization

no code implementations30 Oct 2023 Zhiqi Bu, Ruixuan Liu, Yu-Xiang Wang, Sheng Zha, George Karypis

Recent advances have substantially improved the accuracy, memory cost, and training speed of differentially private (DP) deep learning, especially on large vision and language models with millions to billions of parameters.

Extending Input Contexts of Language Models through Training on Segmented Sequences

no code implementations23 Oct 2023 Petros Karypis, Julian McAuley, George Karypis

Our method benefits both models trained with absolute positional embeddings, by extending their input contexts, as well as popular relative positional embedding methods showing a reduced perplexity on sequences longer than they were trained on.

NameGuess: Column Name Expansion for Tabular Data

1 code implementation19 Oct 2023 Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Shen Wang, Huzefa Rangwala, George Karypis

Recent advances in large language models have revolutionized many sectors, including the database industry.

Text Generation

Coupling public and private gradient provably helps optimization

no code implementations2 Oct 2023 Ruixuan Liu, Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, George Karypis

The success of large neural networks is crucially determined by the availability of data.

DistTGL: Distributed Memory-Based Temporal Graph Neural Network Training

no code implementations14 Jul 2023 Hongkuan Zhou, Da Zheng, Xiang Song, George Karypis, Viktor Prasanna

Evenworse, the tremendous overhead to synchronize the node memory make it impractical to be deployed to distributed GPU clusters.

Graph Representation Learning

Large Language Models of Code Fail at Completing Code with Potential Bugs

1 code implementation NeurIPS 2023 Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, George Karypis

We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs.

Code Completion

Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion

no code implementations1 Jun 2023 Hengzhi Pei, Jinman Zhao, Leonard Lausen, Sheng Zha, George Karypis

To better solve this task, we query a program analyzer for information relevant to a given function call, and consider ways to provide the analyzer results to different code completion models during inference and training.

Code Completion Program Synthesis

XTab: Cross-table Pretraining for Tabular Transformers

1 code implementation10 May 2023 Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, Mahsa Shoaran

The success of self-supervised learning in computer vision and natural language processing has motivated pretraining methods on tabular data.

AutoML Federated Learning +1

Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs

1 code implementation20 Apr 2023 Costas Mavromatis, Vassilis N. Ioannidis, Shen Wang, Da Zheng, Soji Adeshina, Jun Ma, Han Zhao, Christos Faloutsos, George Karypis

Different from conventional knowledge distillation, GRAD jointly optimizes a GNN teacher and a graph-free student over the graph's nodes via a shared LM.

Knowledge Distillation Node Classification

Multimodal Chain-of-Thought Reasoning in Language Models

3 code implementations2 Feb 2023 Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, Alex Smola

Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer.

Language Modelling Science Question Answering

OrthoReg: Improving Graph-regularized MLPs via Orthogonality Regularization

no code implementations31 Jan 2023 Hengrui Zhang, Shen Wang, Vassilis N. Ioannidis, Soji Adeshina, Jiani Zhang, Xiao Qin, Christos Faloutsos, Da Zheng, George Karypis, Philip S. Yu

Graph Neural Networks (GNNs) are currently dominating in modeling graph-structure data, while their high reliance on graph structure for inference significantly impedes them from widespread applications.

Node Classification

SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing

no code implementations10 Dec 2022 Chaoyang He, Shuai Zheng, Aston Zhang, George Karypis, Trishul Chilimbi, Mahdi Soltanolkotabi, Salman Avestimehr

The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost.

ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs

1 code implementation24 Oct 2022 Costas Mavromatis, George Karypis

Our method, termed ReaRev, introduces a new way to KGQA reasoning with respect to both instruction decoding and execution.

Graph Question Answering Knowledge Graphs +3

Injecting Domain Knowledge from Empirical Interatomic Potentials to Neural Networks for Predicting Material Properties

1 code implementation14 Oct 2022 Zeren Shui, Daniel S. Karls, Mingjian Wen, Ilia A. Nikiforov, Ellad B. Tadmor, George Karypis

In recent years, neural network (NN)-based potentials trained on quantum mechanical (DFT-labeled) data have emerged as a more accurate alternative to conventional EIPs.

Drug Discovery Transfer Learning +1

Differentially Private Optimization on Large Model at Small Cost

2 code implementations30 Sep 2022 Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, George Karypis

Our implementation achieves state-of-the-art (SOTA) accuracy with very small extra cost: on GPT2 and at almost the same memory cost (<1% overhead), BK has 1. 03X the time complexity of the standard training (0. 83X training speed in practice), and 0. 61X the time complexity of the most efficient DP implementation (1. 36X training speed in practice).

Privacy Preserving

Differentially Private Bias-Term only Fine-tuning of Foundation Models

1 code implementation30 Sep 2022 Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, George Karypis

We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data.

Privacy Preserving

Variational Causal Inference

2 code implementations13 Sep 2022 Yulun Wu, Layne C. Price, Zichen Wang, Vassilis N. Ioannidis, Robert A. Barton, George Karypis

Estimating an individual's potential outcomes under counterfactual treatments is a challenging task for traditional causal inference and supervised learning approaches when the outcome is high-dimensional (e. g. gene expressions, impulse responses, human faces) and covariates are relatively limited.

Causal Inference counterfactual

Efficient and effective training of language and graph neural network models

no code implementations22 Jun 2022 Vassilis N. Ioannidis, Xiang Song, Da Zheng, Houyu Zhang, Jun Ma, Yi Xu, Belinda Zeng, Trishul Chilimbi, George Karypis

The effectiveness in our framework is achieved by applying stage-wise fine-tuning of the BERT model first with heterogenous graph information and then with a GNN model.

Edge Classification Language Modelling +1

Nimble GNN Embedding with Tensor-Train Decomposition

no code implementations21 Jun 2022 Chunxing Yin, Da Zheng, Israt Nisa, Christos Faloutos, George Karypis, Richard Vuduc

This paper describes a new method for representing embedding tables of graph neural networks (GNNs) more compactly via tensor-train (TT) decomposition.

graph partitioning

ScatterSample: Diversified Label Sampling for Data Efficient Graph Neural Network Learning

no code implementations9 Jun 2022 Zhenwei Dai, Vasileios Ioannidis, Soji Adeshina, Zak Jost, Christos Faloutsos, George Karypis

ScatterSample employs a sampling module termed DiverseUncertainty to collect instances with large uncertainty from different regions of the sample space for labeling.

Active Learning Fraud Detection

Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

no code implementations NAACL 2022 Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, George Karypis

Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks.

Multi-Task Learning Representation Learning

Learning Personalized Item-to-Item Recommendation Metric via Implicit Feedback

no code implementations18 Mar 2022 Trong Nghia Hoang, Anoop Deoras, Tong Zhao, Jin Li, George Karypis

We develop and investigate a personalizable deep metric model that captures both the internal contents of items and how they were interacted with by users.

Metric Learning Recommendation Systems

Joint Learning of Hierarchical Community Structure and Node Representations: An Unsupervised Approach

no code implementations22 Jan 2022 Ancy Sarah Tom, Nesreen K. Ahmed, George Karypis

To account for the structure in the node representations, Mazi generates node representations at each level of the hierarchy, and utilizes them to influence the node representations of the original graph.

Graph Representation Learning Link Prediction +1

TempoQR: Temporal Question Reasoning over Knowledge Graphs

1 code implementation10 Dec 2021 Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N. Ioannidis, Soji Adeshina, Phillip R. Howard, Tetiana Grinberg, Nagib Hakim, George Karypis

The first computes a textual representation of a given question, the second combines it with the entity embeddings for entities involved in the question, and the third generates question-specific time embeddings.

Entity Embeddings Graph Question Answering +4

Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces

1 code implementation27 Nov 2021 Fabio Broccatelli, Richard Trager, Michael Reutlinger, George Karypis, Mufei Li

In this work, we benchmark a variety of single- and multi-task graph neural network (GNN) models against lower-bar and higher-bar traditional machine learning approaches employing human engineered molecular features.

Benchmarking Graph Attention

Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation

no code implementations12 Oct 2021 Cole Hawkins, Vassilis N. Ioannidis, Soji Adeshina, George Karypis

Consistency training is a popular method to improve deep learning models in computer vision and natural language processing.

Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing

no code implementations EMNLP (sustainlp) 2021 Haoyu He, Xingjian Shi, Jonas Mueller, Zha Sheng, Mu Li, George Karypis

We aim to identify how different components in the KD pipeline affect the resulting performance and how much the optimal KD pipeline varies across different datasets/tasks, such as the data augmentation policy, the loss function, and the intermediate representation for transferring the knowledge between teacher and student.

Data Augmentation Hyperparameter Optimization

HeMI: Multi-view Embedding in Heterogeneous Graphs

1 code implementation14 Sep 2021 Costas Mavromatis, George Karypis

Many real-world graphs involve different types of nodes and relations between nodes, being heterogeneous by nature.

Clustering Link Prediction +3

Trust your neighbors: A comprehensive survey of neighborhood-based methods for recommender systems

no code implementations9 Sep 2021 Athanasios N. Nikolakopoulos, Xia Ning, Christian Desrosiers, George Karypis

Collaborative recommendation approaches based on nearest-neighbors are still highly popular today due to their simplicity, their efficiency, and their ability to produce accurate and personalized recommendations.

Recommendation Systems

Position-based Hash Embeddings For Scaling Graph Neural Networks

no code implementations31 Aug 2021 Maria Kalantzi, George Karypis

GNNs compute node representations by taking into account the topology of the node's ego-network and the features of the ego-network's nodes.

Position Recommendation Systems +1

TraverseNet: Unifying Space and Time in Message Passing for Traffic Forecasting

1 code implementation25 Aug 2021 Zonghan Wu, Da Zheng, Shirui Pan, Quan Gan, Guodong Long, George Karypis

This paper aims to unify spatial dependency and temporal dependency in a non-Euclidean space while capturing the inner spatial-temporal dependencies for traffic data.

Attribute

Schema-Aware Deep Graph Convolutional Networks for Heterogeneous Graphs

no code implementations3 May 2021 Saurav Manchanda, Da Zheng, George Karypis

To address this question, we propose our GCN framework 'Deep Heterogeneous Graph Convolutional Network (DHGCN)', which takes advantage of the schema of a heterogeneous graph and uses a hierarchical approach to effectively utilize information many hops away.

IACN: Influence-aware and Attention-based Co-evolutionary Network for Recommendation

1 code implementation4 Mar 2021 Shalini Pandey, George Karypis, Jaideep Srivasatava

The interaction modeling layer is responsible for updating the embedding of a user and an item when the user interacts with the item.

Representation Learning

Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning

1 code implementation27 Feb 2021 Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, George Karypis

Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair, which can capture the relationship between each node and its neighboring substructure in an unsupervised way.

Anomaly Detection Contrastive Learning +1

Learning over Families of Sets -- Hypergraph Representation Learning for Higher Order Tasks

no code implementations19 Jan 2021 Balasubramaniam Srinivasan, Da Zheng, George Karypis

In this work, we exploit the incidence structure to develop a hypergraph neural network to learn provably expressive representations of variable sized hyperedges which preserve local-isomorphism in the line graph of the hypergraph, while also being invariant to permutations of its constituent vertices.

Graph Representation Learning hyperedge classification

An Empirical Comparison of Deep Learning Models for Knowledge Tracing on Large-Scale Dataset

no code implementations16 Jan 2021 Shalini Pandey, George Karypis, Jaideep Srivastava

Recent release of large-scale student performance dataset \cite{choi2019ednet} motivates the analysis of performance of deep learning approaches that have been proposed to solve KT.

Knowledge Tracing

Learning Student Interest Trajectory for MOOCThread Recommendation

no code implementations10 Jan 2021 Shalini Pandey, Andrew Lan, George Karypis, Jaideep Srivastava

The projection operation learns to estimate future embedding of students and threads.

Distant-Supervised Slot-Filling for E-Commerce Queries

no code implementations15 Dec 2020 Saurav Manchanda, Mohit Sharma, George Karypis

Slot-filling refers to the task of annotating individual terms in a query with the corresponding intended product characteristics (product type, brand, gender, size, color, etc.).

Retrieval slot-filling +1

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

1 code implementation11 Oct 2020 Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, George Karypis

To minimize the overheads associated with distributed computations, DistDGL uses a high-quality and light-weight min-cut graph partitioning algorithm along with multiple balancing constraints.

Fraud Detection graph partitioning

PanRep: Universal node embeddings for heterogeneous graphs

no code implementations28 Sep 2020 Vassilis N. Ioannidis, Da Zheng, George Karypis

Learning unsupervised node embeddings facilitates several downstream tasks such as node classification and link prediction.

Link Prediction Node Classification

Heterogeneous Molecular Graph Neural Networks for Predicting Molecule Properties

1 code implementation26 Sep 2020 Zeren Shui, George Karypis

As they carry great potential for modeling complex interactions, graph neural network (GNN)-based methods have been widely used to predict quantum mechanical properties of molecules.

Formation Energy

Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning

2 code implementations15 Sep 2020 Costas Mavromatis, George Karypis

Motivated by this observation, we propose a graph representation learning method called Graph InfoClust (GIC), that seeks to additionally capture cluster-level information content.

Clustering Graph Representation Learning +3

Few-shot link prediction via graph neural networks for Covid-19 drug-repurposing

1 code implementation20 Jul 2020 Vassilis N. Ioannidis, Da Zheng, George Karypis

This paper proposes an inductive RGCN for learning informative relation embeddings even in the few-shot learning regime.

Drug Discovery Few-Shot Learning +5

PanRep: Graph neural networks for extracting universal node embeddings in heterogeneous graphs

1 code implementation20 Jul 2020 Vassilis N. Ioannidis, Da Zheng, George Karypis

Learning unsupervised node embeddings facilitates several downstream tasks such as node classification and link prediction.

Link Prediction Node Classification

Repurpose Open Data to Discover Therapeutics for COVID-19 using Deep Learning

no code implementations21 May 2020 Xiangxiang Zeng, Xiang Song, Tengfei Ma, Xiaoqin Pan, Yadi Zhou, Yuan Hou, Zheng Zhang, George Karypis, Feixiong Cheng

While this study, by no means recommends specific drugs, it demonstrates a powerful deep learning methodology to prioritize existing drugs for further investigation, which holds the potential of accelerating therapeutic development for COVID-19.

DGL-KE: Training Knowledge Graph Embeddings at Scale

1 code implementation18 Apr 2020 Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, George Karypis

Experiments on knowledge graphs consisting of over 86M nodes and 338M edges show that DGL-KE can compute embeddings in 100 minutes on an EC2 instance with 8 GPUs and 30 minutes on an EC2 cluster with 4 machines with 48 cores/machine.

Distributed, Parallel, and Cluster Computing

Context-aware Non-linear and Neural Attentive Knowledge-based Models for Grade Prediction

no code implementations9 Mar 2020 Sara Morsy, George Karypis

Grade prediction for future courses not yet taken by students is important as it can help them and their advisers during the process of course selection as well as for designing personalized degree plans and modifying them based on their performance.

CAWA: An Attention-Network for Credit Attribution

1 code implementation26 Nov 2019 Saurav Manchanda, George Karypis

Experiments on the credit attribution task on a variety of datasets show that the sentence class labels generated by CAWA outperform the competing approaches.

Information Retrieval Multilabel Text Classification +5

Intent term selection and refinement in e-commerce queries

1 code implementation22 Aug 2019 Saurav Manchanda, Mohit Sharma, George Karypis

Moreover, for the tasks of identifying the important terms in a query and for predicting the additional terms that represent product intent, experiments illustrate that our approaches outperform the non-contextual baselines.

A Self-Attentive model for Knowledge Tracing

5 code implementations16 Jul 2019 Shalini Pandey, George Karypis

Knowledge tracing is the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities.

Ad-Hoc Information Retrieval Knowledge Tracing

Feature-based factorized Bilinear Similarity Model for Cold-Start Top-n Item Recommendation

no code implementations22 Apr 2019 Mohit Sharma, Jiayu Zhou, Junling Hu, George Karypis

The user personalized non-collaborative methods based on item features can be used to address this item cold-start problem.

Will this Course Increase or Decrease Your GPA? Towards Grade-aware Course Recommendation

no code implementations22 Apr 2019 Sara Morsy, George Karypis

To this end, we propose two different grade-aware course recommendation approaches to recommend to each student his/her optimal set of courses.

Representation Learning

Adaptive Matrix Completion for the Users and the Items in Tail

1 code implementation22 Apr 2019 Mohit Sharma, George Karypis

In this work, we show that the skewed distribution of ratings in the user-item rating matrix of real-world datasets affects the accuracy of matrix-completion-based approaches.

Collaborative Filtering Low-Rank Matrix Completion +1

Learning from Sets of Items in Recommender Systems

no code implementations22 Apr 2019 Mohit Sharma, F. Maxwell Harper, George Karypis

Our analysis of these ratings shows that though the majority of the users provide the average of the ratings on a set's constituent items as the rating on the set, there exists a significant number of users that tend to consistently either under- or over-rate the sets.

Collaborative Filtering Recommendation Systems

Sparse Neural Attentive Knowledge-based Models for Grade Prediction

no code implementations22 Apr 2019 Sara Morsy, George Karypis

Grade prediction for future courses not yet taken by students is important as it can help them and their advisers during the process of course selection as well as for designing personalized degree plans and modifying them based on their performance.

Distributed representation of multi-sense words: A loss-driven approach

no code implementations14 Apr 2019 Saurav Manchanda, George Karypis

Word2Vec's Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words.

Word Similarity

Text segmentation on multilabel documents: A distant-supervised approach

no code implementations14 Apr 2019 Saurav Manchanda, George Karypis

Segmenting text into semantically coherent segments is an important task with applications in information retrieval and text summarization.

Information Retrieval Multilabel Text Classification +6

Scalable Label Propagation for Multi-relational Learning on the Tensor Product of Graphs

1 code implementation20 Feb 2018 Zhuliu Li, Raphael Petegrosso, Shaden Smith, David Sterling, George Karypis, Rui Kuang

In this paper, we generalize a widely used label propagation model to the normalized tensor product graph, and propose an optimization formulation and a scalable Low-rank Tensor-based Label Propagation algorithm (LowrankTLP) to infer multi-relations for two learning tasks, hyperlink prediction and multiple graph alignment.

Knowledge Graphs Relational Reasoning

Cannot find the paper you are looking for? You can Submit a new open access paper.