Skip to content

Binary File Detection

Jump to bottom
Mikhail Deynekin edited this page Oct 12, 2026 · 3 revisions

Binary File Detection

Historical page (sr 6.x). It documents the Bash implementation, which sr 7 replaced with a Python rewrite and a different command line. Nothing below applies to 7.x; the page is kept for reference. Current documentation: Home, Installation, Quick-Start and sr help (27 topics). The text below is unchanged.

Automatically detect and skip binary files to prevent corruption of output and maintain data integrity.

Table of Contents

Overview

Multi-layer detection prevents accidental modification of binary files like images, archives, executables. sr-search-replace uses a sophisticated multi-layer approach to identify binary content:

  1. File Extension Analysis - Quick check using known binary extensions
  2. Magic Number Detection - Identifies file types by byte signatures
  3. Character Encoding Detection - Analyzes byte patterns for text encoding
  4. Statistical Analysis - Examines null bytes and character distribution

Detection Algorithm Flowchart

The binary detection process follows this comprehensive algorithm:

┌─────────────────────────────────────┐
│  FILE RECEIVED FOR PROCESSING       │
└──────────────┬──────────────────────┘
               │
               ▼
        ┌──────────────────┐
        │  Check if file   │
        │  is readable     │
        │  and accessible  │
        └──────┬───────────┘
               │
        ┌──────▼───────────┐
        │   SUCCESS?       │
        └─┬────────────┬───┘
          │ NO         │ YES
          │            │
          ▼            ▼
      ┌─────────┐   ┌──────────────────────┐
      │  ERROR  │   │ STEP 1: EXTENSION    │
      │  SKIP   │   │ Check file extension │
      └─────────┘   │ against binary list  │
                    └──────┬───────────────┘
                           │
                    ┌──────▼──────────┐
                    │ BINARY EXT?     │
                    └─┬───────────┬───┘
                      │ YES       │ NO
                      │           │
                      ▼           ▼
                  ┌─────────┐   ┌──────────────────────┐
                  │ SKIP    │   │ STEP 2: MAGIC NUMBER │
                  │ BINARY  │   │ Read first 512 bytes │
                  └─────────┘   │ Check magic signatures
                                └──────┬───────────────┘
                                       │
                                ┌──────▼──────────┐
                                │ MAGIC MATCH?    │
                                └─┬───────────┬───┘
                                  │ YES       │ NO
                                  │           │
                                  ▼           ▼
                              ┌─────────┐   ┌──────────────────────┐
                              │ SKIP    │   │ STEP 3: ENCODING     │
                              │ BINARY  │   │ Check byte patterns  │
                              └─────────┘   │ for UTF-8/ASCII      │
                                            └──────┬───────────────┘
                                                   │
                                            ┌──────▼──────────┐
                                            │ NULL BYTES?     │
                                            └─┬───────────┬───┘
                                              │ YES       │ NO
                                              │           │
                                              ▼           ▼
                                          ┌─────────┐   ┌──────────────────────┐
                                          │ SKIP    │   │ STEP 4: STATISTICS   │
                                          │ BINARY  │   │ Analyze byte distrib.│
                                          └─────────┘   │ Count control chars  │
                                                        └──────┬───────────────┘
                                                               │
                                                        ┌──────▼──────────┐
                                                        │ HIGH CONTROL    │
                                                        │ CHAR COUNT?     │
                                                        └─┬───────────┬───┘
                                                          │ YES       │ NO
                                                          │           │
                                                          ▼           ▼
                                                      ┌─────────┐   ┌──────────────┐
                                                      │ SKIP    │   │ TREAT AS     │
                                                      │ BINARY  │   │ TEXT / PROCESS
                                                      └─────────┘   └──────────────┘

Enable Binary Detection

Use the --skip-binary flag to enable multi-layer binary file detection:

sr --find "pattern" --replace "text" --skip-binary --recursive .

Or with file pattern:

sr --find "pattern" --replace "text" --skip-binary *.txt

Detection Methods

sr-search-replace employs four complementary detection methods:

1. Null Byte Scanning

The most reliable and fastest method - scans for null bytes (0x00) which indicate binary content:

# Any null byte indicates binary file
if grep -q $'\x00' file.bin; then echo "BINARY"; fi

Effectiveness: 99%+
Speed: Instant (first null byte found)
False Positives: Extremely rare

2. Magic Number Analysis

Identifies file types by reading file signatures (magic bytes) from file headers:

# Examples of magic numbers:
# PNG:  89 50 4E 47 (hex) = \x89PNG
# JPEG: FF D8 FF (hex)
# ZIP:  50 4B 03 04 (hex) = PK
# ELF:  7F 45 4C 46 (hex) = .ELF

Common Binary Signatures:

  • Images: PNG, JPG, GIF, BMP
  • Archives: ZIP, RAR, TAR, GZIP
  • Executables: ELF, Mach-O, PE (Windows)
  • Documents: PDF, Office formats

3. Character Encoding Detection

Analyzes byte sequences to detect if file is valid UTF-8 or ASCII:

# Check for valid UTF-8 encoding
if file -i file.txt | grep -q "charset=iso-8859-1\|charset=unknown"; then
    echo "POTENTIALLY BINARY"
fi

Indicators of Binary:

  • Invalid UTF-8 sequences
  • High proportion of control characters (0x00-0x1F)
  • Invalid character sequences
  • Mixed or unknown encoding

4. Statistical Analysis

Examines overall byte distribution and control character frequency:

# Count control characters (0x00-0x1F)
binary_count=$(od -An -tx1 file.txt | grep -o "[0-1][0-9a-f]" | wc -l)
if [ $binary_count -gt $threshold ]; then
    echo "BINARY"
fi

Thresholds:

  • < 1% control chars: Likely text
  • 1-5% control chars: Borderline (use other methods)

  • 5% control chars: Likely binary

Configuration

Environment Variables

# Enable or disable binary detection
export SR_SKIP_BINARY="true"

# Custom binary file extensions (colon-separated)
export SR_BINARY_EXTENSIONS=".bin:.exe:.dll:.so:.dylib"

# Binary detection threshold (percentage of control chars)
export SR_BINARY_THRESHOLD="5"

# Timeout for binary detection (milliseconds)
export SR_BINARY_TIMEOUT="1000"

# Enable detailed binary detection logging
export SR_DEBUG_BINARY="true"

Configuration File

{
  "skipBinary": true,
  "binaryExtensions": [
    ".bin", ".exe", ".dll", ".so", ".dylib",
    ".jpg", ".jpeg", ".png", ".gif", ".bmp",
    ".zip", ".rar", ".tar", ".gz", ".7z",
    ".pdf", ".doc", ".docx", ".xls", ".xlsx"
  ],
  "binaryThreshold": 5,
  "binaryTimeout": 1000,
  "debugBinary": false
}

Direct Code Configuration

In Python implementation, modify these constants at the top of sr.py:

# Binary detection configuration
BINARY_THRESHOLD = 5  # Percentage of control chars (0x00-0x1F)
BINARY_TIMEOUT = 1000  # Milliseconds
SKIP_BINARY = True  # Enable by default

# Known binary extensions
BINARY_EXTENSIONS = {
    # Images
    '.jpg', '.jpeg', '.png', '.gif', '.bmp', '.ico', '.tiff', '.webp',
    # Archives
    '.zip', '.rar', '.7z', '.tar', '.gz', '.bz2', '.xz',
    # Executables
    '.exe', '.dll', '.so', '.dylib', '.o', '.a', '.lib',
    # Documents
    '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx',
    # Media
    '.mp3', '.mp4', '.avi', '.mov', '.flv', '.mkv', '.wav'
}

# Magic number signatures for file identification
MAGIC_NUMBERS = {
    b'\x89PNG': 'png',
    b'\xff\xd8\xff': 'jpeg',
    b'GIF87a': 'gif',
    b'GIF89a': 'gif',
    b'BM': 'bmp',
    b'%PDF': 'pdf',
    b'PK\x03\x04': 'zip',
    b'Rar!': 'rar',
    b'\x7fELF': 'elf',
    b'MZ': 'pe',
}

In Bash implementation, modify sr.sh:

# Binary detection configuration
readonly BINARY_THRESHOLD=5
readonly BINARY_TIMEOUT=1000
readonly SKIP_BINARY=true

# Known binary extensions
readonly BINARY_EXTENSIONS="bin exe dll so dylib jpg jpeg png gif bmp zip rar tar gz pdf"

# Function to check if file is binary
is_binary() {
    local file="$1"
    local ext="${file##*.}"
    
    # Check extension
    if [[ "$BINARY_EXTENSIONS" =~ $ext ]]; then
        return 0  # Binary
    fi
    
    # Check for null bytes
    if grep -q $'\x00' "$file" 2>/dev/null; then
        return 0  # Binary
    fi
    
    return 1  # Text
}

Supported Formats

Automatically Detected:

Images: PNG, JPG, GIF, BMP, ICO, TIFF, WebP, SVG (SVG treated as text)

Archives: ZIP, TAR, GZ, BZ2, 7Z, RAR, XZ, Gzip, Bzip2

Executables: ELF, PE (Windows), Mach-O (macOS), Cygwin executables

Documents: PDF, Office (DOC, DOCX, XLS, XLSX, PPT, PPTX), OpenOffice

Media: MP3, MP4, AVI, MOV, FLV, MKV, WAV, FLAC

Databases: SQLite, MySQL dump binary, PostgreSQL binary

Not Detected (Treated as Text):

  • Text files with binary-like extensions (may require --force-text)
  • XML, JSON, YAML, TOML files
  • Source code files (.c, .cpp, .js, .py, .rb, .go, .rs, etc.)
  • Configuration files (.conf, .ini, .cfg)
  • Markup files (.md, .rst, .asciidoc, .html, .xml)
  • Log files (.log, .txt)

Implementation Details

Detection Priority

The algorithm checks methods in this order:

  1. File Extension (Fastest) - O(1) operation

    • Pre-computed set of known binary extensions
    • Useful for common formats

  2. Magic Number (Fast) - O(1) operation on first 512 bytes

    • Identifies file type regardless of extension
    • Reliable for standard formats

  3. Null Byte Scanning (Very Fast) - O(n) but stops at first match

    • Most reliable single indicator
    • Finds binary status almost instantly

  4. Encoding Detection (Medium) - O(n) on full file read

    • Secondary confirmation
    • Analyzes character sequences

  5. Statistical Analysis (Slowest) - O(n) on full file

    • Last resort check
    • Uses byte distribution patterns

Performance Optimization

Two-Stage Detection:

Stage 1: Quick checks (extension, magic, null bytes) - ~1-10ms Stage 2: Deep analysis (encoding, statistics) - Only if Stage 1 inconclusive

Caching:

  • Binary detection results cached per file
  • Cache invalidated on file modification
  • Significant speedup for repeated operations

Early Exit:

  • Process exits immediately after detecting binary
  • No need to read entire file
  • Typical file analyzed in < 1ms

Override Detection

Force processing of files detected as binary:

# Force treat as text (skip binary detection)
sr --find "pattern" --replace "text" --force-text file.bin

# Process with confirmation
sr --find "pattern" --replace "text" --skip-binary --confirm *.bin

Use Cases for Override:

  1. Text-like Binary Formats: Some binary formats contain mostly text (e.g., SVG, XML in archives)

  2. Custom Text Encodings: Custom character sets not detected by standard methods

  3. Intentional Binary Modification: Rare case where binary editing is desired

  4. Testing: Developers testing sr-search-replace functionality

Performance Considerations

Detection Speed

Method Speed Files/sec Best For
Extension Instant 1000+ Quick filtering
Magic Number < 1ms 100+ Standard formats
Null Byte Scan 1-5ms 50+ Most cases
Encoding Check 5-20ms 10-20 Uncertain files
Full Statistics 20-100ms 1-10 Edge cases

Memory Usage

  • Binary detection: ~1-5 MB per file (buffer size dependent)
  • Cache storage: ~1 KB per detected file
  • No significant memory overhead

Recommendations

  • Use --skip-binary by default for safety
  • Combine with --exclude for faster processing: sr --skip-binary --exclude "*.bin" --exclude "*.exe" ...
  • Enable caching for large directory trees
  • Use --dry-run before processing binary-heavy directories

See Also

sr 7.0.1

Home

Начало Установка Быстрый старт Примеры

Справка Совпадение и regex Выбор файлов Сессии и бэкапы Откат JSON и автоматизация Конфигурация Автообновление Устранение неполадок

Ещё Справочник опций Коды возврата man sr Миграция с 6.x README

Clone this wiki locally