πŸ‘οΈ Tool for Data Extraction and Web Requests.

Related tags

Web Crawlingpythonurlcrawlerexploittoolweb-crawler
Overview

httpmapper πŸ‘οΈ

Languages Repository size License

Project   β€’    Technologies   β€’    Installation    β€’    How it works   β€’    License   

Project 🚧

For educational purposes.

This is a project that I developed, which is basically a Web crawler that navigate in the web, extracting source codes, links, cookies and more. I also did to learn more about requests and data-extraction.

Technologies πŸ› οΈ

This project was developed with the following technologies:

  • Python

Installation πŸš€ 

# cloning repository
git clone https://github.com/vLeeH/httpmapper

# enter on folder 
cd httpmapper 

# update 
sudo apt update 

# running 
python install.py

How it works πŸ”§ 

# using
python install.py 

examples 

# website for this example: https://github.com 

   python install.py 

   Choice: 5
   Website: https://github.com 

   [+] Cookie Name = _octo - Cookie Value = GH1.1.413278149.1633841686
   [+] Cookie Name = logged_in - Cookie Value = no
   [+] Cookie Name = _gh_sess - Cookie Value = ejqBvu%2BSIjM68y7f8niePF8U%2FyrwbGVoKc8iW6FWLil8%2BtsOtGcYSaxw52b%2FhCg%2F275eqHG18jSe4wZ7TFzvlD5Xx6tqvddoSy%2BEdOUlooL7gEpchhK1W8i0Y%2Fg1ARBhrK3saX43%2FjlBEMJX45km%2BPHf39gxk1fO8fc6ytX%2Fp7uX2F1z3hMIep76ooxirYuFzSwBefa3EZU5fZq2OQoV2is8xjiInY72lDSxErMjPKKS6%2B1cjUp9NW7bS5G63%2B9AJCPMwjpdg15qa8aulJ%2FLZg%3D%3D--qTvdBCfTpQiV75Hr--liTEY8bhr%2B0QHWrLVyJZ8w%3D%3D
   [-] Cookie extracter finished!

Note: You need to identify which browser is being used for the header variable.

Contributing πŸ”¨

How can I contribute to the project?

1. Create a fork from httpmapper repository.
2. git clone https://github.com/vLeeH/httpmapper.git
3. cd httpmapper/
4. Make your changes.
5. Commit and make a git push.
6. Open a pull request.

License πŸ“

This project is under the MIT License.

Owner
I'm a young student of Python and Programming.
GitHub Repository
This scrapper scrapes the mail ids of faculty members from a given linl/page and stores it in a csv file

This scrapper scrapes the mail ids of faculty members from a given linl/page and stores it in a csv file

Devansh Singh 1 Feb 10, 2022
Newsscraper - A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

NewsScraper A simple Python 3 module to get crypto or news articles and their content from various RSS feeds. πŸ”§ Installation Clone the repo locally.

Rokas 3 Jan 02, 2022
Comment Webpage Screenshot is a GitHub Action that captures screenshots of web pages and HTML files located in the repository

Comment Webpage Screenshot is a GitHub Action that helps maintainers visually review HTML file changes introduced on a Pull Request by adding comments with the screenshots of the latest HTML file cha

Maksudul Haque 21 Sep 29, 2022
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Parsel Parsel is a BSD-licensed Python library to extract and remove data from HTML and XML using XPath and CSS selectors, optionally combined with re

Scrapy project 859 Dec 29, 2022
This is a webscraper for a specific website

This is a webscraper for a specific website. It is tuned to extract the headlines of that website. With some little adjustments the webscraper is able to extract any part of the website.

Rahul Siyanwal 1 Dec 13, 2021
A tool can scrape product in aliexpress: Title, Price, and URL Product.

Scrape-Product-Aliexpress A tool can scrape product in aliexpress: Title, Price, and URL Product. Usage: 1. Install Python 3.8 3.9 padahal halaman ins

Rahul Joshua Damanik 1 Dec 30, 2021
Scrape and display grades onto the console

WebScrapeGrades About The Project This Project is a personal project where I learned how to webscrape using python requests. Being able to get request

Cyrus Baybay 1 Oct 23, 2021
An helper library to scrape data from Instagram effortlessly, using the Influencer Hunters APIs.

Instagram Scraper An utility library to scrape data from Instagram hassle-free Go to the website Β» View Demo Β· Report Bug Β· Request Feature About The

2 Jul 06, 2022
EBay-email-tracker - Scapes an entire search page of a particular item on eBay and sends regular updates to an email address

Introduction This is a project I built with the sole intent to learn more about

Luis M. Capdevielle 1 Jan 14, 2022
Grab the changelog from releases on Github

release-notes-scraper This simple script can be used to grab the release notes for projects from github that do not keep a CHANGELOG, but publish thei

Dan ČermÑk 4 Apr 01, 2022
Crawler job that scrapes comments from social media posts and saves them in a S3 bucket.

Toxicity comments crawler Crawler job that scrapes comments from social media posts and saves them in a S3 bucket. Twitter Tweets and replies are scra

Douglas Trajano 2 Jan 24, 2022
Crawl the information of a given keyword on Google search engine

Crawl the information of a given keyword on Google search engine

4 Nov 09, 2022
A distributed crawler for weibo, building with celery and requests.

A distributed crawler for weibo, building with celery and requests.

SpiderClub 4.8k Jan 03, 2023
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Gerapy Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Documentation Documentation

Gerapy 2.9k Jan 03, 2023
A simple proxy scraper that utilizes the requests module in python.

Proxy Scraper A simple proxy scraper that utilizes the requests module in python. Usage Depending on your python installation your commands may vary.

3 Sep 08, 2021
A Telegram crawler to search groups and channels automatically and collect any type of data from them.

Introduction This is a crawler I wrote in Python using the APIs of Telethon months ago. This tool was not intended to be publicly available for a numb

39 Dec 28, 2022
Dude is a very simple framework for writing web scrapers using Python decorators

Dude is a very simple framework for writing web scrapers using Python decorators. The design, inspired by Flask, was to easily build a web scraper in just a few lines of code. Dude has an easy-to-lea

Ronie Martinez 326 Dec 15, 2022
Html Content / Article Extractor, web scrapping lib in Python

Python-Goose - Article Extractor Intro Goose was originally an article extractor written in Java that has most recently (Aug2011) been converted to a

Xavier Grangier 3.8k Jan 02, 2023
A low-code tool that generates python crawler code based on curl or url

KKBA Intruoduction A low-code tool that generates python crawler code based on curl or url Requirement Python = 3.6 Install pip install kkba Usage Co

8 Sep 20, 2021
A simple code to fetch comments below an Instagram post and save them to a csv file

fetch_comments A simple code to fetch comments below an Instagram post and save them to a csv file usage First you have to enter your username and pas

2 Jul 14, 2022