Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition
AI-generated Key Points
- Open-set action recognition (OSAR) involves rejecting unknown human action cases that fall outside the distribution of the training set.
- Existing methods for OSAR focus on learning better uncertainty scores, but they often overlook the importance of feature representations.
- The authors propose a novel Prototypical Similarity Learning (PSL) framework to enlarge instance-specific (IS) and class-specific (CS) information in feature representations for better OSAR performance.
- CS information is used for inter-class recognition, while IS information is unique to each sample within a class. Both types of information are crucial for OSAR performance.
- To enlarge IS information, PSL encourages instances to have less than 1 similarity with their corresponding prototypes, retaining more IS information in learned feature representations.
- Video shuffling is introduced into PSL to alleviate misclassification issues caused by OoD videos that share similar appearances with InD videos.
- Shuffled videos are encouraged to have less than 1 similarity with original samples, allowing networks to extract distinct temporal information among them and enlarging CS information.
- Experiments demonstrate that PSL significantly boosts both open-set and closed-set performance on multiple benchmarks, achieving state-of-the-art results.
- The proposed framework provides a novel perspective on analyzing OSAR tasks based on the information bottleneck theory and highlights the importance of retaining both IS and CS information for optimal performance.
Authors: Jun Cen, Shiwei Zhang, Xiang Wang, Yixuan Pei, Zhiwu Qing, Yingya Zhang, Qifeng Chen
Abstract: Open-set action recognition is to reject unknown human action cases which are out of the distribution of the training set. Existing methods mainly focus on learning better uncertainty scores but dismiss the importance of feature representations. We find that features with richer semantic diversity can significantly improve the open-set performance under the same uncertainty scores. In this paper, we begin with analyzing the feature representation behavior in the open-set action recognition (OSAR) problem based on the information bottleneck (IB) theory, and propose to enlarge the instance-specific (IS) and class-specific (CS) information contained in the feature for better performance. To this end, a novel Prototypical Similarity Learning (PSL) framework is proposed to keep the instance variance within the same class to retain more IS information. Besides, we notice that unknown samples sharing similar appearances to known samples are easily misclassified as known classes. To alleviate this issue, video shuffling is further introduced in our PSL to learn distinct temporal information between original and shuffled samples, which we find enlarges the CS information. Extensive experiments demonstrate that the proposed PSL can significantly boost both the open-set and closed-set performance and achieves state-of-the-art results on multiple benchmarks. Code is available at https://github.com/Jun-CEN/PSL.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.