Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Related tags

Third-party APIs Wrapperspyspark_sam
Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
GitHub Repository
AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.

Photo by Denny Müller on Unsplash AWS Automated Inventory ( aws-auto-inventory ) Automates creation of detailed inventories from AWS resources. Table

AWS Samples 123 Dec 26, 2022
Access Undenied parses AWS AccessDenied CloudTrail events, explains the reasons for them, and offers actionable remediation steps. Open-sourced by Ermetic.

Access Undenied on AWS Access Undenied parses AWS AccessDenied CloudTrail events, explains the reasons for them, and offers actionable fixes. Access U

Ermetic 204 Jan 02, 2023
A full-fledged discord bot with moderation and a lot more.

HOT-BOT-POL-POT ⭐ Star me on GitHub m'lady.... hot-bot-pol-pot is a moderation discord bot written using enhanced-dpy library with many functionalitie

Pure Cheekbones 4 Oct 08, 2022
Trading strategy for the Freqtrade crypto bot

NostalgiaForInfinity Trading strategy for the Freqtrade crypto bot Change strategy Add strategies to the user_data/strategies folder and also in the d

iterativ 1.5k Jan 01, 2023
SelfBot, a lots of features: Mass DM , Nuke, Raid... and more!

SelfBot, a lots of features: Mass DM , Nuke, Raid... and more!

eryycr 2 Nov 10, 2021
Freqtrade is a free and open source crypto trading bot written in Python.

Freqtrade is a free and open source crypto trading bot written in Python. It is designed to support all major exchanges and be controlled via Telegram. It contains backtesting, plotting and money man

Kazune Takeda 5 Dec 30, 2021
A pyrogram simple bot for Educational purpose.

A pyrogram simple bot for Educational purpose. To Learn More check at @PyrogramBot or on Documentation Mandatory variables API_ID - Get It From my.tel

SpamShield 10 Dec 06, 2022
Windows版本微信客户端（非网页版）自动化，可实现简单的发送、接收微信消息

wxauto Windows版本微信客户端自动化，可实现简单的发送、接收微信消息

357 Dec 29, 2022
Instagram Account Creator 🥰

Instagram Account Creatorr Instagram account creator that uses requests instead of selenium! Report Bug · Request Feature Features Request based Conta

dropout 9 Oct 27, 2021
Cities bot - A simple example of using aiogram and the wikipedia package

Cities game A simple example of using aiogram and the wikipedia package. The bot

Artem Meller 2 Jan 29, 2022
Automatically pick a winner who Retweeted, Commented, and Followed your Twitter account!

AutomaticTwitterGiveaways automates selecting winners for "Retweet, Comment, Follow" type Twitter giveaways.

1 Jan 13, 2022
Uma API pública contendo informações sobre o unvierso de Roberto Gomez Bolaños.

Chespirito API Objetivo Esta API tem como objetivo ser um ponto de referência para a procura sobre todo o universo do grande Roberto Gomez Bolaños, ta

Pery Lemke 6 Feb 02, 2022
A Python wrapper for the Yelp API v2

python-yelp-v2 A Python wrapper for the Yelp API v2. The structure for this was inspired by the python-twitter library, and some internal methods are

Matthew Conlen 12 Oct 24, 2017
Wordle-bot: A Discord bot to track you and your friends' Wordle scores.

wordle-bot A Discord bot to track you and your friends' Wordle scores, so you can see who's the best! To submit a score to wordle-bot, just paste the

Spencer Murray 8 Feb 16, 2022
Python tool to Check running WebClient services on multiple targets based on @leechristensen

WebClient Service Scanner Python tool to Check running WebClient services on multiple targets based on @tifkin_ idea. This tool uses impacket project.

Pixis 153 Dec 28, 2022
A Collection Manager for the objkt.com Minting Factory

Objkt Collection Manager A Collection Manager for the objkt.com Minting Factory. This contract can create a collection on objkt.com and mint into it.

Asbjorn Enge 5 Nov 22, 2022
A discord token grabber made in Python 3

Discord Token Grabber A Discord token grabber written in Python 3. This version of the grabber only supports Windows. Features Transfers via Discord w

Mega145 4 Aug 04, 2022
The accompanying code for the paper "GMAT: Global Memory Augmentation for Transformers" (Ankit Gupta and Jonathan Berant).

GMAT: Global Memory Augmentation for Transformers This repository contains the accompanying code for the paper: "GMAT: Global Memory Augmentation for

Ankit Gupta 7 Oct 21, 2021
A Bot Telegram Anti Users Channel to automatic ban users who using channel to send message in group.

Tg_Anti_UsersChannel A Bot Telegram Anti Users Channel to automatic ban users who using channel to send message in group. Features: Automatic ban Whit

idzeroid 6 Dec 26, 2021
Python API client library for phpIPAM installations

phpypam: Python API client library for phpIPAM installation As we started to develop phpipam-ansible-modules we used an existing python library for ph

codeaffen 3 Aug 03, 2022