BREP : Binary Search in plaintext and gzip files

Last update: Dec 24, 2021

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Search large files in O(log n) time using binary search.
We support plaintext and Gzipped files.

Benchmark : 8x faster than `grep` on a 2GB dataset !

brep is usually faster than grep for >1GB datasets.

Check tests/benchmark.py to reproduce the results.

grep ^777 test.txt : 1.594 s (15 runs)
brep 777 test.txt : 206.8 ms (15 runs)

Installation

pip install brep or pip install . from this repo

Index your file

In order to conduct binary search, your file needs to be sorted.
We recommend GNU sort, as it's multithreaded and supports large files.
LC_ALL=C sort -u -o output_file input_file

BREP supports compressed file in the GZIP format.
We recommend pigz for quick multicore compression : pigz file

Usage

Provide 1 prefix search term and 1 filepath
brep 77777 test/large.gz

You can also search from our Python class

from brep import Search

for result in Search("77777", "test/large.gz"):
    print(result)

Contribute

PRs are welcome!

Install dev dependencies: pip install -e .[dev]
Test and lint before submitting: pytest && flake8

Todo

Reimplement in Rust
Faster gz size estimation
Search multiple strings at once

dotsend is a web application which helps you to upload your large files and share file via link

0 Dec 3, 2022

A wrapper for DVD file structure and ISO files.

vs-parsedvd DVDs were an error. A wrapper for DVD file structure and ISO files. You can find me in the IEW Discord server

7 Nov 17, 2022

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

PyFstab Generator PyFstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems. NOTE : Th

2 Nov 9, 2021

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Fasta Uploader A tool for batch processing large fasta files and accompanying metadata table to repositories via API The python fasta_uploader.py scri

Centre for Infectious Disease and One Health

1 Dec 9, 2021

Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

1 Nov 12, 2021

A simple file module for creating, editing and saving files.

1 Nov 25, 2021

This program can help you to move and rename many files at once

This program can help you to rename and save many files in a folder in seconds, but don't give the same name to files, it can delete both files.

1 Oct 10, 2022

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

asammdf is a fast parser and editor for ASAM (Association for Standardization of Automation and Measuring Systems) MDF (Measurement Data Format) files

440 Dec 31, 2022

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a two dimensional vector for Desmos and there is a method to output the coefficients as LaTeX code.

12 Jan 1, 2023

BREP : Binary Search in plaintext and gzip files

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Benchmark : 8x faster than grep on a 2GB dataset !

Installation

Index your file

Usage

Contribute

Todo

You might also like...

dotsend is a web application which helps you to upload your large files and share file via link

A wrapper for DVD file structure and ISO files.

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Python interface for reading and appending tar files

A simple file module for creating, editing and saving files.

This program can help you to move and rename many files at once

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Releases(v1.0.1)

v1.0.1(Nov 7, 2021)

Owner

Arnaud de Saint Meloir

File-manager - A basic file manager, written in Python

Python module that parse power builder file (PBD) and analyze code

File storage with API access. Used as a part of the Swipio project

Pti-file-format - Reverse engineering the Polyend Tracker instrument file format

Dragon Age: Origins toolset to extract/build .erf files, patch language-specific .dlg files, and view the contents of files in the ERF or GFF format

Automatically generates a TypeQL script for doing entity and relationship insertions from a .csv file, so you don't have to mess with writing TypeQL.

This is a junk file creator tool which creates junk files in Internal Storage

Python package to read and display segregated file names present in a directory based on type of the file

Various converters to convert value sets from CSV to JSON, etc.

A simple bulk file renamer, written in python.

This is a file deletion program that asks you for an extension of a file (.mp3, .pdf, .docx, etc.) to delete all of the files in a dir that have that extension.

Listreqs is a simple requirements.txt generator. It's an alternative to pipreqs

Simple, convenient and cross-platform file date changing library. 📝📅

CleverCSV is a Python package for handling messy CSV files.

A python wrapper for libmagic

Kartothek - a Python library to manage large amounts of tabular data in a blob store

A simple library for temporary storage of small files

Lumar - Smart File Creator

Object-oriented file system path manipulation

Python's Filesystem abstraction layer

Benchmark : 8x faster than `grep` on a 2GB dataset !