Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Last update: Dec 06, 2022

Overview

TransGanFormer (wip)

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GansFormer and TransGan paper. It will also contain a bunch of tricks I have picked up building transformers and GANs for the last year or so, including efficient linear attention and pixel level attention.

Install

$ pip install transganformer

Usage

$ transganformer --data ./path/to/data

Citations

@misc{jiang2021transgan,
    title   = {TransGAN: Two Transformers Can Make One Strong GAN}, 
    author  = {Yifan Jiang and Shiyu Chang and Zhangyang Wang},
    year    = {2021},
    eprint  = {2102.07074},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

@misc{hudson2021generative,
    title   = {Generative Adversarial Transformers}, 
    author  = {Drew A. Hudson and C. Lawrence Zitnick},
    year    = {2021},
    eprint  = {2103.01209},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

You might also like...

Finding all things on-prem Microsoft for password spraying and enumeration.

msprobe About Installing Usage Examples Coming Soon Acknowledgements About Finding all things on-prem Microsoft for password spraying and enumeration.

205 Jan 9, 2023

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re

189 Nov 22, 2022

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Memory Efficient Attention Pytorch Implementation of a memory efficient multi-head attention as proposed in the paper, Self-attention Does Not Need O(

180 Jan 5, 2023

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

Deformable Attention Implementation of Deformable Attention from this paper in Pytorch, which appears to be an improvement to what was proposed in DET

128 Dec 24, 2022

A curated (most recent) list of resources for Learning with Noisy Labels

321 Jan 9, 2023

The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

11 Nov 27, 2022

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

1k Dec 31, 2022

Unofficial PyTorch implementation of Fastformer based on paper "Fastformer: Additive Attention Can Be All You Need"."

Fastformer-PyTorch Unofficial PyTorch implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Usage : import t

126 Dec 6, 2022

Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Fastformer-Keras Unofficial Tensorflow-Keras implementation of Fastformer based on paper Fastformer: Additive Attention Can Be All You Need. Tensorflo

10 Jan 30, 2022

Comments

kmeans_iters in attention

Hello first at all thank you for taking the time and port all of theses models to pytorch, you are really helping out people like me who have tensorflow as a second language :D

in line 689 of https://github.com/dorarad/gansformer/blob/main/training/network.py there is a loop over kmeans-iter in the transformerlayer, but i can t find this loop in your implementation of Attention, is it implicitly implemented with these einops - i am sadly not fluent with einops or tensorflow :/

regards martin

opened by martinpflaum 0
dual_contrast_loss argument - error while training

When trying to train a model, the dual_contrast_loss argument spits up an error on lightweight-gan.py. My guess is that it's not implemented yet.

At first I thought it was a mismatch (should it be dual_contrastive_loss btw like in lightweight-gan?) but apparently needed to be commented out for training to start successfully.

Very exciting by the way! Training seems to be super fast, about 28h for 150,000 iterations for a 256p model on my 2080 (although it's a very small dataset). I'll share some closer-to-final results once I have them.

Best, Theodore.

opened by TheodoreGalanos 2

Releases(0.0.17)

0.0.17(Apr 27, 2021)

Source code(tar.gz)
Source code(zip)
0.0.16(Apr 15, 2021)

Source code(tar.gz)
Source code(zip)
0.0.15(Apr 13, 2021)

Source code(tar.gz)
Source code(zip)
0.0.14(Apr 13, 2021)

Source code(tar.gz)
Source code(zip)
0.0.12(Apr 12, 2021)

Source code(tar.gz)
Source code(zip)
0.0.11(Apr 12, 2021)

Source code(tar.gz)
Source code(zip)
0.0.10(Apr 11, 2021)

Source code(tar.gz)
Source code(zip)
0.0.9(Apr 11, 2021)

Source code(tar.gz)
Source code(zip)
0.0.8(Apr 11, 2021)

Source code(tar.gz)
Source code(zip)
0.0.7(Apr 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.6(Apr 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Apr 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Apr 7, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Apr 7, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Apr 5, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Apr 5, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need.

GitHub Repository

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Related tags

Overview

TransGanFormer (wip)

Install

Usage

Citations

You might also like...

Finding all things on-prem Microsoft for password spraying and enumeration.

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

A curated (most recent) list of resources for Learning with Noisy Labels

The official implementation of NeurIPS 2021 paper: Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

Unofficial PyTorch implementation of Fastformer based on paper "Fastformer: Additive Attention Can Be All You Need"."

Unofficial Tensorflow-Keras implementation of Fastformer based on paper [Fastformer: Additive Attention Can Be All You Need](https://arxiv.org/abs/2108.09084).

Comments

kmeans_iters in attention

dual_contrast_loss argument - error while training

Releases(0.0.17)

0.0.17(Apr 27, 2021)

0.0.16(Apr 15, 2021)

0.0.15(Apr 13, 2021)

0.0.14(Apr 13, 2021)

0.0.12(Apr 12, 2021)

0.0.11(Apr 12, 2021)

0.0.10(Apr 11, 2021)

0.0.9(Apr 11, 2021)

0.0.8(Apr 11, 2021)

0.0.7(Apr 9, 2021)

0.0.6(Apr 9, 2021)

0.0.5(Apr 9, 2021)

0.0.4(Apr 7, 2021)

0.0.3(Apr 7, 2021)

0.0.2(Apr 5, 2021)

0.0.1(Apr 5, 2021)

Owner

Phil Wang

Training data extraction on GPT-2

Run object detection model on the Raspberry Pi

Density-aware Single Image De-raining using a Multi-stream Dense Network (CVPR 2018)

The pytorch implementation of the paper "text-guided neural image inpainting" at MM'2020

Implementation of Research Paper "Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation"

Video Corpus Moment Retrieval with Contrastive Learning (SIGIR 2021)

a project for 3D multi-object tracking

Official implementation for "Symbolic Learning to Optimize: Towards Interpretability and Scalability"

A 3D Dense mapping backend library of SLAM based on taichi-Lang designed for the aerial swarm.

Changing the Mind of Transformers for Topically-Controllable Language Generation

LSUN Dataset Documentation and Demo Code

Equivariant Imaging: Learning Beyond the Range Space

PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Official repository for Fourier model that can generate periodic signals

code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

Recurrent Variational Autoencoder that generates sequential data implemented with pytorch

Official source code to CVPR'20 paper, "When2com: Multi-Agent Perception via Communication Graph Grouping"

AI Face Mesh: This is a simple face mesh detection program based on Artificial intelligence.

Random Erasing Data Augmentation. Experiments on CIFAR10, CIFAR100 and Fashion-MNIST

Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (CVPR 2022, Oral)