Log Generator 3000 (LG3K)

🚀 The ultimate modular log generation tool, designed for modern systems! 🌍

---

Documentation Guides

Log Generation Guide

Comprehensive guide for generating logs with LG3K:

• Log format specifications
• Module-specific examples
• Error handling patterns
• Progress tracking
• Best practices for log generation

Llama Training How-To

Step-by-step guide for training llama3.2:3b-instruct-fp16 with LG3K logs:

• Hardware and software requirements
• Environment setup for Windows, macOS, and Ubuntu
• Training data generation
• Model configuration
• Training script implementation
• Monitoring and troubleshooting
• Memory optimization for 8-12GB VRAM GPUs

Developer Guide

Detailed guide for developers integrating LG3K:

• Configuration file management
• Programmatic usage examples
• Error handling strategies
• Progress tracking implementation
• File cleanup patterns
• Module-specific integrations
• JSON output handling
• LLM format generation

---

Installation

From PyPI (Recommended)

The easiest way to install LG3K is from PyPI:

pip install lg3k

This will install the latest stable version with all required dependencies.

From Source (Development)

For the latest development version or contributing:

git clone https://github.com/mikl0s/LG3K.git
cd LG3K
python -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows
pip install -r requirements-dev.txt
pip install -e .
pre-commit install

Requirements

• Python (latest stable version)
• Dependencies are automatically installed with pip
• Optional: rich package for enhanced display
• Optional: torch and psutil for GPU-optimized LLM training

---

About

Welcome to Log Generator 3000—a fully modular log generation tool designed to simplify testing and monitoring across diverse systems. It supports web servers, APIs, databases, firewalls, and more, with special support for LLM training data generation.

This project was conceptualized, developed, and published entirely on an iPad during a Saturday evening of football—and yes, the team we were rooting for won! 🎉

Curious about the full story? Read more here.

---

Contributing

We believe in the power of community! LG3K becomes more valuable with each new contribution, whether it's adding new log types, improving existing ones, or enhancing the core functionality.

Ways to Contribute

1. Add New Log Types 📝

   • Create new modules for different systems
   • Enhance existing log formats
   • Add more realistic log patterns

2. Improve Core Features 🛠️

   • Enhance performance
   • Add new configuration options
   • Improve error handling

3. Documentation 📚

   • Improve documentation
   • Add examples
   • Write tutorials

4. Testing 🧪

   • Add unit tests
   • Report bugs
   • Suggest improvements

Getting Started with Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature-name)
3. Set up development environment:

   python -m venv venv
   source venv/bin/activate  # or `venv\Scripts\activate` on Windows
   pip install -r requirements-dev.txt  # Installs all development dependencies
   pip install -e .  # Install package in development mode
   pre-commit install

4. Run tests:

   pytest  # Runs tests with coverage report

5. Make your changes (the pre-commit hooks will ensure code quality)
6. Write tests if applicable
7. Update documentation if needed
8. Commit your changes (git commit -m 'Add feature')
9. Push to your branch (git push origin feature-name)
10. Open a Pull Request

Development Guidelines

• Code is formatted with Black (88 characters line length)
• Imports are sorted with isort
• Code quality is checked with Flake8
• All functions and modules have docstrings
• Changes are covered by tests (when applicable)

---

📂 Project Structure

lg3k/
├── __init__.py          # Package initialization
├── modules/             # Folder containing all log generation modules
│   ├── web_server.py    # Module for web server logs
│   ├── database.py      # Module for database logs
│   ├── api.py           # Module for API logs
│   ├── firewall.py      # Module for firewall logs
│   ├── nas.py           # Module for NAS logs
│   ├── os.py            # Module for OS logs
│   ├── network.py       # Module for network logs
│   ├── printer.py       # Module for printer logs
│   ├── smarthome.py     # Module for smart home devices and IoT
├── utils/               # Folder containing utility functions
│   ├── config.py        # Utilities for configuration handling
│   ├── progress.py      # Utilities for progress and threading
│   ├── timestamp.py     # Timestamp generation utilities

---

✨ Features

• Dynamic Module Loading: Easily add new log types by creating a module in the modules/ folder.
• Scalable and Modular: Keep your codebase clean and maintainable by separating log logic into distinct files.
• Docker-Style Progress: Real-time progress tracking with Docker-like display for each module.
• Smart Home Support: Generate logs for IoT devices, ESP32/ESP8266, Zigbee/Z-Wave, and security cameras.
• High Volume: Generate up to 1,000,000 log entries per module.
• Rich UI: Beautiful, real-time progress bar for generating logs (with fallback to simple mode).
• Fully Configurable: Modify the configuration file to control active services, total logs, threading, and more.
• JSON Output Mode: Get structured output in JSON format for easy parsing and automation.
• Configuration Generation: Generate default configuration files with --generate-config.
• Code Quality: Enforced by Black, isort, and Flake8 through pre-commit hooks.
• 94% Test Coverage: Comprehensive test suite ensuring reliability.
• LLM Training Format: Generate logs in a format optimized for training Large Language Models.
• Rich-Free Operation: Can run without the rich package installed using --simple or --llm options.
• Graceful Error Handling: Comprehensive error handling with informative messages.
• Progress Status: Real-time status updates for each module (Running, Complete, Error, Cancelled).
• File Cleanup: Automatic cleanup of generated files with keep-files option.
• Module Analysis: Built-in log analysis capabilities for each module type.

---

Getting Started

Prerequisites

• Python (latest stable version)
• For users:

  pip install -r requirements.txt

• For developers:

  pip install -r requirements-dev.txt
  pip install -e .
  pre-commit install

---

Quick Start

1. Install the package:

   pip install lg3k

2. Generate logs:

   lg3k --count 1000 --threads 4

3. Generate logs without rich UI:

   lg3k --count 1000 --threads 4 --simple

4. Generate logs in LLM format:

   lg3k --count 1000 --threads 4 --llm-format

5. View help:

   lg3k --help

---

Generating Logs for LLM Training

To generate logs in a format suitable for training Large Language Models:

lg3k --llm-format

This will:

• Generate logs in instruction-tuning format
• Include detailed analysis for each log entry
• Structure data for easy model training
• Support both string and JSON log formats
• Handle errors gracefully with informative messages

For more details, see the Llama Training How-To.

---

Developer Guide

Looking to integrate LG3K into your application or AI model? Check out our Developer Guide for:

• 🔧 Programmatic usage examples
• 🤖 AI integration patterns
• 📊 Log format specifications
• ⚡ Performance optimization tips
• 🧪 Integration testing strategies
• 🛠️ Configuration file generation
• 📋 JSON output mode usage
• 🎯 Error handling best practices
• 📈 Progress tracking implementation
• 🧹 File cleanup strategies

---

Available Modules

• Infrastructure

  • web_server - Web server access logs
  • database - Database operations
  • api - API endpoint logs
  • firewall - Security events
  • nas - Storage operations
  • os - System logs
  • network - Network traffic
  • printer - Print jobs

• Smart Home & IoT

  • Smart home devices (thermostats, lights, sensors)
  • ESP32/ESP8266 microcontrollers
  • Zigbee/Z-Wave devices
  • Security cameras and doorbells

---

📊 Sample Output

Docker-Style Progress Display

1ff5d2e3: web_server   [=========>  ] 50.0%
5520ebfb: database     [==========] Complete
e9c8c5d6: api         [>         ] Waiting
7a1b3c4d: smarthome   [=======>   ] 35.0%
...

Rich UI (Default Mode)

[Thread 1] ██████▉ 90% (90/100 logs)
[Thread 2] █████████▌ 100% (100/100 logs) Completed: logs_part2.json
...

Simple Mode

Starting log generation for 1000 logs across 10 files.
Thread 1 completed generating ./logs/logs_part1.json
Thread 2 completed generating ./logs/logs_part2.json
...

JSON Output Mode

Use the --json flag for structured output in a single line (ideal for parsing):

lg3k --count 1000 --threads 4 --json

This outputs a single line of JSON with detailed information (formatted here for readability):

{
    "success": true,
    "logs_generated": 1000,
    "time_taken": 1.23,
    "files": ["logs/part1.json", "logs/part2.json"],
    "stats": {
        "total_files": 2,
        "avg_logs_per_file": 500,
        "total_size_bytes": 12345
    },
    "timing": {
        "start_time": "2024-03-22T12:34:56.789012",
        "duration_seconds": 1.23,
        "logs_per_second": 813.0
    },
    "config": {
        "output_directory": "logs",
        "file_format": ".json"
    }
}

In case of errors (also single-line output):

{
    "success": false,
    "logs_generated": 0,
    "time_taken": 0.0,
    "files": [],
    "error": {
        "message": "Error message here",
        "type": "ErrorType"
    }
}

---

📜 License

This project is licensed under the MIT License. See the LICENSE file for details.

---

💬 Questions?

Feel free to open an issue or contact us at [email protected].

---

🌌 Show Your Support

If you love Log Generator 3000, give us a ⭐ on GitHub! Spread the word and help others test their systems with ease.