Monitor and log Network and Disks statistics in MegaBytes per second.

Related tags

Loggingdiskmetricsnetworkiopytorch-lightning
Overview

iometrics

Python

Monitor and log Network and Disks statistics in MegaBytes per second.

Install

pip install iometrics

Usage

Pytorch-lightning integration

from pytorch_lightning import Trainer
from iometrics.pytorch_lightning.callbacks import NetworkAndDiskStatsMonitor

net_disk_stats = NetworkAndDiskStatsMonitor()

trainer = Trainer(callbacks=[net_disk_stats])

Metrics generated

  • network/recv_MB_per_sec – Received MB/s on all relevant network interfaces as a SUM.
  • network/sent_MB_per_sec – Sent MB/s on all relevant network interfaces as a SUM.
  • disk/util% – Disk utilization percentage as the average of all disk devices.
  • disk/read_MB_per_sec – Disks read MB/s as the sum of all disk devices.
  • disk/writ_MB_per_sec – Disks written MB/s as the sum of all disk devices.
  • disk/io_read_count_per_sec – Disks read I/O operations per second as the sum of all disk devices.
  • disk/io_writ_count_per_sec – Disks written I/O operations per second as the sum of all disk devices.

Screen shot

Pure-Python implementation (zero dependencies)

Quick check

python -c 'from iometrics.example import usage; usage()'

Example output

|        Network (MBytes/s)       | Disk Util |            Disk MBytes          |           Disk I/O          |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    | I/O Write |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  | val | avg |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| ---:| ---:|
|    4.6 |    3.5 |   0.1 |   0.1 |  49 |   2 |   52.8 |    1.1 |   0.0 |   0.9 |    211 |      4 |   5 |  18 |
|    4.1 |    3.5 |   0.1 |   0.1 |  61 |   3 |   60.4 |    2.4 |  40.3 |   1.7 |    255 |     10 | 149 |  21 |

Full code

import time
from iometrics import NetworkMetrics, DiskMetrics
from iometrics.example import DUAL_METRICS_HEADER
net  = NetworkMetrics()
disk = DiskMetrics()
for i in range(100):
    time.sleep(1)
    net.update_stats()
    disk.update_stats()
    if i % 15 == 0:
        print(DUAL_METRICS_HEADER)
    row = (
        f"| {net.mb_recv_ps.val:6.1f} | {net.mb_recv_ps.avg:6.1f} "
        f"| {net.mb_sent_ps.val:5.1f} | {net.mb_sent_ps.avg:5.1f} "
        f"| {int(disk.io_util.val):3d} | {int(disk.io_util.avg):3d} "
        f"| {disk.mb_read.val:6.1f} | {disk.mb_read.avg:6.1f} "
        f"| {disk.mb_writ.val:5.1f} | {disk.mb_writ.avg:5.1f} "
        f"| {int(disk.io_read.val):4d} | {int(disk.io_read.avg):4d} "
        f"| {int(disk.io_writ.val):3d} | {int(disk.io_writ.avg):3d} "
        f"|"
    )
    print(row)

Contributing

See CONTRIBUTING.md

You might also like...
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 4, 2021
A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

Alpwuf 1 Nov 16, 2021
This is a wonderful simple python tool used to store the keyboard log.

Keylogger This is a wonderful simple python tool used to store the keyboard log. Record your keys. It will capture passwords and credentials in a comp

Rithin Lehan 2 Nov 25, 2021
Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

James Kelly 1 Feb 25, 2022
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

null 23 Oct 4, 2021
Robust and effective logging for Python 2 and 3.
Robust and effective logging for Python 2 and 3.

Robust and effective logging for Python 2 and 3.

Chris Hager 1k Jan 4, 2023
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Soda Data Monitoring 51 Jan 1, 2023
A Fast, Extensible Progress Bar for Python and CLI
A Fast, Extensible Progress Bar for Python and CLI

tqdm tqdm derives from the Arabic word taqaddum (تقدّم) which can mean "progress," and is an abbreviation for "I love you so much" in Spanish (te quie

tqdm developers 23.7k Jan 1, 2023
Rich is a Python library for rich text and beautiful formatting in the terminal.
Rich is a Python library for rich text and beautiful formatting in the terminal.

Rich 中文 readme • lengua española readme • Läs på svenska Rich is a Python library for rich text and beautiful formatting in the terminal. The Rich API

Will McGugan 41.5k Jan 7, 2023
Comments
  • Add Docker Support

    Add Docker Support

    Change Summary

    Add Docker Support

    Related issue number

    https://github.com/elgalu/iometrics/issues/3

    Checklist

    • [ ] Unit tests for the changes exist
    • [ ] Tests pass on CI and coverage is maintained
    • [x] Documentation reflects the changes where applicable
    opened by m0rt3z4gh 0
  • Not working properly in containers (Docker)

    Not working properly in containers (Docker)

    Network metrics properly reported in the Host:

    |        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
|  417.2 |  417.2 |   3.0 |   3.0 |   8 |   8 |    0.9 |    0.9 | 370.5 | 370.5 |    220 |    220 | 11897 | 11897 |
|  335.1 |  376.1 |   2.4 |   2.7 |  10 |   9 |    1.4 |    1.1 | 275.5 | 323.0 |    335 |    278 |  9168 | 10532 |
|  434.2 |  395.5 |   3.2 |   2.9 |   9 |   9 |    3.9 |    2.1 | 169.9 | 272.0 |    653 |    403 | 11450 | 10838 |

    Network metrics are not visible in the container Disclaimer: the timeline extracts are not perfectly synchronized

    |        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
|    0.0 |    0.0 |   0.0 |   0.0 |   7 |   9 |    0.9 |    1.3 |  82.4 | 556.4 |    216 |    310 | 20093 |  9335 |
|    0.0 |    0.0 |   0.0 |   0.0 |   9 |   9 |   24.9 |    2.0 | 375.4 | 550.8 |    783 |    325 |  3604 |  9156 |
|    0.0 |    0.0 |   0.0 |   0.0 |  10 |   9 |   60.5 |    3.8 | 103.9 | 537.2 |   1315 |    355 | 12689 |  9263 |

    To reproduce

    1. generate network I/O outside the container (in the host)
    2. start a container without any special flags, without any admin or additional capabilities

    TODO

    Find a fix with the least possible privileges

    bug 
    opened by elgalu 0
  • Features/azdo pipeline

    Features/azdo pipeline

    Change Summary

    Related issue number

    Checklist

    • [ ] Unit tests for the changes exist
    • [ ] Tests pass on CI and coverage is maintained
    • [ ] Documentation reflects the changes where applicable
    opened by ugurgural 0
Releases(0.0.8)
Owner
Leo Gallucci
Leo is the creator of docker-selenium which aims to get Selenium running as simple and fast as possible
Leo Gallucci
GitHub Repository https://pypi.org/project/iometrics/
This open-source python3 script is a builder to the very popular token logger that is on my github that many people use.

Discord-Logger-Builder This open-source python3 script is a builder to the very popular token logger that is on my github that many people use. This i

Local 4 Nov 17, 2021
Integrates a UPS monitored by NUT into OctoPrint

OctoPrint UPS This OctoPrint plugin interfaces with a UPS monitored by NUT (Network UPS Tools). Requirements NUT must be configured by the user. This

Shawn Bruce 11 Jul 05, 2022
A python logging library

logi v1.3.4 instolation the lib works on python 3x versions pip install logi examples import import logi log = logger(path='C:/file path', timestamp=T

2 Jul 06, 2022
giving — the reactive logger

giving is a simple, magical library that lets you log or "give" arbitrary data throughout a program and then process it as an event stream.

Olivier Breuleux 0 May 24, 2022
Logging system for the TPC software.

tpc_logger Logging system for the TPC software. The TPC Logger class provides a singleton for logging information within C++ code or in the python API

UC Davis Machine Learning 1 Jan 10, 2022
Summarize LSF job properties by parsing log files.

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Kim 4 Jan 09, 2022
Espion is a mini-keylogger tool that keeps track of all keys a user presses on his/her keyboard

Espion is a mini-keylogger tool that keeps track of all keys a user presses on his/her keyboard. The details get displayed on the terminal window and also stored in a log file.

Anurag.R.Simha 1 Apr 24, 2022
pyEventLogger - a simple Python Library for making customized Logs of certain events that occur in a program

pyEventLogger is a simple Python Library for making customized Logs of certain events that occur in a program. The logs can be fully customized and can be printed in colored format or can be stored i

Siddhesh Chavan 2 Nov 03, 2022
🐑 Syslog Simulator hazır veya kullanıcıların eklediği logları belirtilen adreslere ve port'a seçilen döngüde syslog ile gönderilmesini sağlayan araçtır. | 🇹🇷

syslogsimulator hazır ürün loglarını SIEM veya log toplayıcısına istediğiniz portta belirli sürelerde göndermeyi sağlayan küçük bir araçtır.

Enes Aydın 3 Sep 28, 2021
Small toolkit for python multiprocessing logging to file

Small Toolkit for Python Multiprocessing Logging This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation) In

Qishuai 1 Nov 10, 2021
Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

James Kelly 1 Feb 25, 2022
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 04, 2021
Stand-alone parser for User Access Logging from Server 2012 and newer systems

KStrike Stand-alone parser for User Access Logging from Server 2012 and newer systems BriMor Labs KStrike This script will parse data from the User Ac

BriMor Labs 69 Nov 01, 2022
Beautifully colored, quick and simple Python logging

Python Quick Logging | QLogging Beautifully colored, quick and simple Python logging. This logger is based on Python logging package Screenshots: Term

45 Sep 25, 2022
A small utility to pretty-print Python tracebacks. ⛺

TBVaccine TBVaccine is a utility that pretty-prints Python tracebacks. It automatically highlights lines you care about and deemphasizes lines you don

Stavros Korokithakis 365 Nov 11, 2022
dash-manufacture-spc-dashboard is a dashboard for monitoring read-time process quality along manufacture production line

In our solution based on plotly, dash and influxdb, the user will firstly generate the specifications for different robots, and then a wide range of interactive visualizations for different machines

Dequn Teng 1 Feb 13, 2022
Fancy console logger and wise assistant within your python projects

Fancy console logger and wise assistant within your python projects. Made to save tons of hours for common routines.

BoB 5 Apr 01, 2022
Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Luca 2 Mar 27, 2022
Lazy Profiler is a simple utility to collect CPU, GPU, RAM and GPU Memory stats while the program is running.

lazyprofiler Lazy Profiler is a simple utility to collect CPU, GPU, RAM and GPU Memory stats while the program is running. Installation Use the packag

Shankar Rao Pandala 28 Dec 09, 2022
Vibrating-perimeter - Simple helper mod that logs how fast you are mining together with a simple buttplug.io script to control a vibrator

Vibrating Perimeter This project consists of a small minecraft helper mod that writes too a log file and a script that reads said log. Currently it on

Heart[BOT] 0 Nov 20, 2022