Referring Video Object Segmentation

Overview

Awesome-Referring-Video-Object-Segmentation Awesome

Welcome to starts โญ & comments ๐Ÿ’น & sharing ๐Ÿ˜€ !!

- 2021.12.12: Recent papers (from 2021) 
- welcome to add if any information misses. ๐Ÿ˜Ž

Introduction

image

Referring video object segmentation aims at segmenting an object in video with language expressions.

Unlike the previous video object segmentation, the task exploits a different type of supervision, language expressions, to identify and segment an object referred by the given language expressions in a video. A detailed explanation of the new task can be found in the following paper.

Seonguk Seo, Joon-Young Lee, Bohyung Han, โ€œURVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmarkโ€, European Conference on Computer Vision (ECCV), 2020:https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123600205.pdf

Impressive Works Related to Referring Video Object Segmentation (RVOS)

Cross-modal progressive comprehension for referring segmentation:https://arxiv.org/abs/2105.07175 image

Benchmark

The 3rd Large-scale Video Object Segmentation - Track 3: Referring Video Object Segmentation

Datasets

image

Refer-YouTube-VOS-datasets

  • YouTube-VOS:
wget https://github.com/JerryX1110/awesome-rvos/blob/main/down_YTVOS_w_refer.py
python down_YTVOS_w_refer.py

Folder structure:

${current_path}/
โ””โ”€โ”€ refer_youtube_vos/ 
    โ”œโ”€โ”€ train/
    โ”‚   โ”œโ”€โ”€ JPEGImages/
    โ”‚   โ”‚   โ””โ”€โ”€ */ (video folders)
    โ”‚   โ”‚       โ””โ”€โ”€ *.jpg (frame image files) 
    โ”‚   โ””โ”€โ”€ Annotations/
    โ”‚       โ””โ”€โ”€ */ (video folders)
    โ”‚           โ””โ”€โ”€ *.png (mask annotation files) 
    โ”œโ”€โ”€ valid/
    โ”‚   โ””โ”€โ”€ JPEGImages/
    โ”‚       โ””โ”€โ”€ */ (video folders)
    โ”‚           โ””โ”€โ”€ *.jpg (frame image files) 
    โ””โ”€โ”€ meta_expressions/
        โ”œโ”€โ”€ train/
        โ”‚   โ””โ”€โ”€ meta_expressions.json  (text annotations)
        โ””โ”€โ”€ valid/
            โ””โ”€โ”€ meta_expressions.json  (text annotations)
  • A2D-Sentences:

REPO:https://web.eecs.umich.edu/~jjcorso/r/a2d/

paper:https://arxiv.org/abs/1803.07485

image

Citation:

@misc{gavrilyuk2018actor,
      title={Actor and Action Video Segmentation from a Sentence}, 
      author={Kirill Gavrilyuk and Amir Ghodrati and Zhenyang Li and Cees G. M. Snoek},
      year={2018},
      eprint={1803.07485},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License: The dataset may not be republished in any form without the written consent of the authors.

README Dataset and Annotation (version 1.0, 1.9GB, tar.bz) Evaluation Toolkit (version 1.0, tar.bz)

mkdir a2d_sentences
cd a2d_sentences
wget https://web.eecs.umich.edu/~jjcorso/bigshare/A2D_main_1_0.tar.bz
tar jxvf A2D_main_1_0.tar.bz
mkdir text_annotations

cd text_annotations
wget https://kgavrilyuk.github.io/actor_action/a2d_annotation.txt
wget https://kgavrilyuk.github.io/actor_action/a2d_missed_videos.txt
wget https://github.com/JerryX1110/awesome-rvos/blob/main/down_a2d_annotation_with_instances.py
python down_a2d_annotation_with_instances.py
unzip a2d_annotation_with_instances.zip
#rm a2d_annotation_with_instances.zip
cd ..

cd ..

Folder structure:

${current_path}/
โ””โ”€โ”€ a2d_sentences/ 
    โ”œโ”€โ”€ Release/
    โ”‚   โ”œโ”€โ”€ videoset.csv  (videos metadata file)
    โ”‚   โ””โ”€โ”€ CLIPS320/
    โ”‚       โ””โ”€โ”€ *.mp4     (video files)
    โ””โ”€โ”€ text_annotations/
        โ”œโ”€โ”€ a2d_annotation.txt  (actual text annotations)
        โ”œโ”€โ”€ a2d_missed_videos.txt
        โ””โ”€โ”€ a2d_annotation_with_instances/ 
            โ””โ”€โ”€ */ (video folders)
                โ””โ”€โ”€ *.h5 (annotations files) 

Citation:

@inproceedings{YaXuCaCVPR2017,
  author = {Yan, Y. and Xu, C. and Cai, D. and {\bf Corso}, {\bf J. J.}},
  booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition}},
  tags = {computer vision, activity recognition, video understanding, semantic segmentation},
  title = {Weakly Supervised Actor-Action Segmentation via Robust Multi-Task Ranking},
  year = {2017}
}
@inproceedings{XuCoCVPR2016,
  author = {Xu, C. and {\bf Corso}, {\bf J. J.}},
  booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition}},
  datadownload = {http://web.eecs.umich.edu/~jjcorso/r/a2d},
  tags = {computer vision, activity recognition, video understanding, semantic segmentation},
  title = {Actor-Action Semantic Segmentation with Grouping-Process Models},
  year = {2016}
}
@inproceedings{XuHsXiCVPR2015,
  author = {Xu, C. and Hsieh, S.-H. and Xiong, C. and {\bf Corso}, {\bf J. J.}},
  booktitle = {{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition}},
  datadownload = {http://web.eecs.umich.edu/~jjcorso/r/a2d},
  poster = {http://web.eecs.umich.edu/~jjcorso/pubs/xu_corso_CVPR2015_A2D_poster.pdf},
  tags = {computer vision, activity recognition, video understanding, semantic segmentation},
  title = {Can Humans Fly? {Action} Understanding with Multiple Classes of Actors},
  url = {http://web.eecs.umich.edu/~jjcorso/pubs/xu_corso_CVPR2015_A2D.pdf},
  year = {2015}
}

image

downloading_script

mkdir jhmdb_sentences
cd jhmdb_sentences
wget http://files.is.tue.mpg.de/jhmdb/Rename_Images.tar.gz
wget https://kgavrilyuk.github.io/actor_action/jhmdb_annotation.txt
wget http://files.is.tue.mpg.de/jhmdb/puppet_mask.zip
tar -xzvf  Rename_Images.tar.gz
unzip puppet_mask.zip
cd ..

Folder structure:

${current_path}/
โ””โ”€โ”€ jhmdb_sentences/ 
    โ”œโ”€โ”€ Rename_Images/  (frame images)
    โ”‚   โ””โ”€โ”€ */ (action dirs)
    โ”œโ”€โ”€ puppet_mask/  (mask annotations)
    โ”‚   โ””โ”€โ”€ */ (action dirs)
    โ””โ”€โ”€ jhmdb_annotation.txt  (text annotations)

Citation:

@inproceedings{Jhuang:ICCV:2013,
title = {Towards understanding action recognition},
author = {H. Jhuang and J. Gall and S. Zuffi and C. Schmid and M. J. Black},
booktitle = {International Conf. on Computer Vision (ICCV)},
month = Dec,
pages = {3192-3199},
year = {2013}
}

image image image

Owner
Explorer
Explorer
Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

NuPIC Numenta Platform for Intelligent Computing The Numenta Platform for Intelligent Computing (NuPIC) is a machine intelligence platform that implem

Numenta 6.3k Dec 30, 2022
Hi Guys, here I am providing examples, which will help you in Lerarning Python

LearningPython Hi guys, here I am trying to include as many practice examples of Python Language, as i Myself learn, and hope these will help you in t

4 Feb 03, 2022
Nb workflows - A workflow platform which allows you to run parameterized notebooks programmatically

NB Workflows Description If SQL is a lingua franca for querying data, Jupyter sh

Xavier Petit 6 Aug 18, 2022
Referring Video Object Segmentation

Awesome-Referring-Video-Object-Segmentation Welcome to starts โญ & comments ๐Ÿ’น & sharing ๐Ÿ˜€ !! - 2021.12.12: Recent papers (from 2021) - welcome to ad

Explorer 57 Dec 11, 2022
Effective Use of Transformer Networks for Entity Tracking

Effective Use of Transformer Networks for Entity Tracking (EMNLP19) This is a PyTorch implementation of our EMNLP paper on the effectiveness of pre-tr

5 Nov 06, 2021
Deep Face Recognition in PyTorch

Face Recognition in PyTorch By Alexey Gruzdev and Vladislav Sovrasov Introduction A repository for different experimental Face Recognition models such

Alexey Gruzdev 141 Sep 11, 2022
Implementation detail for paper "Multi-level colonoscopy malignant tissue detection with adversarial CAC-UNet"

Multi-level-colonoscopy-malignant-tissue-detection-with-adversarial-CAC-UNet Implementation detail for our paper "Multi-level colonoscopy malignant ti

CVSM Group - email: <a href=[email protected]"> 84 Nov 22, 2022
ObsPy: A Python Toolbox for seismology/seismological observatories.

ObsPy is an open-source project dedicated to provide a Python framework for processing seismological data. It provides parsers for common file formats

ObsPy 979 Jan 07, 2023
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Pretrained Language Model This repository provides the latest pretrained language models and its related optimization techniques developed by Huawei N

HUAWEI Noah's Ark Lab 2.6k Jan 01, 2023
Dataset and Code for ICCV 2021 paper "Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme"

Dataset and Code for RealVSR Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme Xi Yang, Wangmeng Xiang,

Xi Yang 92 Jan 04, 2023
SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning

SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning This repository is the official implementation of "SHRIMP: Sparser Random Featur

Bobby Shi 0 Dec 16, 2021
Style-based Neural Drum Synthesis with GAN inversion

Style-based Drum Synthesis with GAN Inversion Demo TensorFlow implementation of a style-based version of the adversarial drum synth (ADS) from the pap

Sound and Music Analysis (SoMA) Group 29 Nov 19, 2022
A deep learning model for style-specific music generation.

DeepJ: A model for style-specific music generation https://arxiv.org/abs/1801.00887 Abstract Recent advances in deep neural networks have enabled algo

Henry Mao 704 Nov 23, 2022
gitใ€ŠUSD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentationใ€‹(2020) GitHub: [fig2]

USD-Seg This project is an implement of paper USD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentation, based on FCOS detector f

Ruolin Ye 80 Nov 28, 2022
Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space"

MotionCLIP Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space". Please visit our webpage for mor

Guy Tevet 173 Dec 26, 2022
ๆŸๅญฆๆ ก้€‰่ฏพ็ณป็ปŸGIF้ชŒ่ฏ็ ๆ•ฐๆฎ้›† + Baselineๆจกๅž‹ + ไธŠไธ‹ๆธธ็›ธๅ…ณๅทฅๅ…ท

elective-dataset-2021spring ๆŸๅญฆๆ ก2021ๆ˜ฅๅญฃ้€‰่ฏพ็ณป็ปŸGIF้ชŒ่ฏ็ ๆ•ฐๆฎ้›†๏ผˆ29338ๅผ ๏ผ‰ + ๅ‡†็กฎ็އ98.4%็š„Baselineๆจกๅž‹ + ไธŠไธ‹ๆธธ็›ธๅ…ณๅทฅๅ…ทใ€‚ ๆ•ฐๆฎ้›†้‡‡็”จ ็Ÿฅ่ฏ†ๅ…ฑไบซ็ฝฒๅ-้žๅ•†ไธšๆ€งไฝฟ็”จ 4.0 ๅ›ฝ้™…่ฎธๅฏๅ่ฎฎ ่ฟ›่กŒ่ฎธๅฏใ€‚ Baselineๆจกๅž‹ๅ’ŒไธŠไธ‹ๆธธ็›ธๅ…ณๅทฅๅ…ท้‡‡็”จ

xmcp 27 Sep 17, 2021
using yolox+deepsort for object-tracker

YOLOX_deepsort_tracker yolox+deepsortๅฎž็Žฐ็›ฎๆ ‡่ทŸ่ธช ๆœ€ๆ–ฐ็š„yoloxๅฐๅฐ้ฒœ~~๏ผˆyoloxๆญฃๅค„ๅœจ้ข‘็นๆ›ดๆ–ฐ้˜ถๆฎต๏ผŒๅ› ๆญค็›ดๆŽฅ้“พๆŽฅyoloxไป“ๅบ“ไฝœไธบๅญๆจกๅ—๏ผ‰ Install Clone the repository recursively: git clone --rec

245 Dec 26, 2022
CKD - Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding

Collaborative Knowledge Distillation for Heterogeneous Information Network Embed

zhousheng 9 Dec 05, 2022
Final project for machine learning (CSC 590). Detection of hepatitis C and progression through blood samples.

Hepatitis C Blood Based Detection Final project for machine learning (CSC 590). Dataset from Kaggle. Using data from previous hepatitis C blood panels

Jennefer Maldonado 1 Dec 28, 2021
Visualizing Yolov5's layers using GradCam

YOLO-V5 GRADCAM I constantly desired to know to which part of an object the object-detection models pay more attention. So I searched for it, but I di

Pooya Mohammadi Kazaj 200 Jan 01, 2023