diff options
Diffstat (limited to 'docs/FEATURE_DOCUMENTATION.md')
| -rw-r--r-- | docs/FEATURE_DOCUMENTATION.md | 911 |
1 files changed, 0 insertions, 911 deletions
diff --git a/docs/FEATURE_DOCUMENTATION.md b/docs/FEATURE_DOCUMENTATION.md deleted file mode 100644 index 30d08d4..0000000 --- a/docs/FEATURE_DOCUMENTATION.md +++ /dev/null @@ -1,911 +0,0 @@ -# Download History Tracking & Duplicate Prevention Feature - -## Overview - -This feature adds comprehensive download history tracking with duplicate prevention capabilities to tidal-dl-ng. It provides users with visual feedback about previously downloaded tracks and the ability to prevent redundant downloads through a persistent JSON-based history system. - -## Table of Contents - -1. [Features](#features) -2. [Architecture](#architecture) -3. [Implementation Details](#implementation-details) -4. [User Interface](#user-interface) -5. [File Structure](#file-structure) -6. [API Reference](#api-reference) -7. [Testing](#testing) -8. [Usage Examples](#usage-examples) - ---- - -## Features - -### Core Functionality - -- **Persistent Download History**: Tracks all downloaded tracks in a JSON file that persists across application restarts -- **Duplicate Prevention**: Automatically skips tracks that have already been downloaded (when enabled) -- **Toggle Control**: Users can enable/disable duplicate prevention via GUI menu -- **Source Tracking**: Records the source of each download (playlist, album, mix, manual) -- **Thread-Safe Operations**: All history operations are thread-safe for concurrent downloads -- **Atomic File Operations**: Prevents corruption through atomic write operations -- **Import/Export**: Allows users to backup and restore their download history -- **Statistics**: Provides insights into download history (total tracks, by source type, dates) -- **Visual Feedback**: Console messages are color-coded (green) for better visibility - -### User Benefits - -- ✅ Avoid wasting bandwidth on duplicate downloads -- ✅ Keep track of what has been downloaded -- ✅ Quickly identify already-downloaded content -- ✅ Portable history through import/export -- ✅ Source-based organization of download history - ---- - -## Architecture - -### Design Pattern: Singleton - -The `HistoryService` uses the Singleton pattern to ensure only one instance manages the download history throughout the application lifecycle. - -```python -class HistoryService(metaclass=SingletonMeta): - """Single instance managing all download history operations""" -``` - -### Data Structure: Track-Centric - -The history uses a track-centric approach with track IDs as keys for O(1) lookup performance: - -```json -{ - "settings": { - "preventDuplicates": true - }, - "tracks": { - "track_id": { - "sourceType": "playlist", - "sourceId": "pl-uuid", - "sourceName": "Playlist Name", - "downloadDate": "2025-11-29T12:34:56.789Z" - } - } -} -``` - -### Thread Safety - -All critical operations are protected by a `threading.Lock`: - -```python -with self._lock: - # Critical section - atomic operations - self.history_data[track_id] = entry - self._save_history_internal() -``` - ---- - -## Implementation Details - -### 1. History Service (`tidal_dl_ng/history.py`) - -**File**: `tidal_dl_ng/history.py` (new file, ~550 lines) - -**Purpose**: Central service for managing download history with JSON persistence. - -#### Key Methods - -##### `add_track_to_history()` - -```python -def add_track_to_history( - self, - track_id: str, - source_type: str = "manual", - source_id: str | None = None, - source_name: str | None = None -) -> None: - """Add a track to download history with source metadata.""" -``` - -##### `should_skip_download()` - -```python -def should_skip_download(self, track_id: str) -> bool: - """ - Determine if a track should be skipped based on: - 1. Track exists in history - 2. Duplicate prevention is enabled - - Returns True if both conditions are met. - """ -``` - -##### `update_settings()` - -```python -def update_settings(self, **kwargs: Any) -> None: - """Update settings and persist immediately to JSON.""" -``` - -#### Atomic Write Implementation - -```python -def _save_history_internal(self) -> None: - """ - Atomic write pattern: - 1. Write to temporary file - 2. Atomic rename (os.replace on Windows) - 3. Cleanup on error - """ - with tempfile.NamedTemporaryFile(...) as tmp_file: - json.dump(data, tmp_file) - tmp_path = tmp_file.name - - os.replace(tmp_path, self.file_path) # Atomic operation -``` - -#### Corruption Recovery - -```python -try: - data = json.load(f) -except (json.JSONDecodeError, ValueError): - # Create backup of corrupted file - backup_path = self.file_path.with_suffix(".json.bak") - shutil.copy2(self.file_path, backup_path) - # Start fresh - self.history_data = {} -``` - ---- - -### 2. Download Integration (`tidal_dl_ng/download.py`) - -**Modifications**: Integration points in existing download flow - -#### Pre-Download Check - -```python -def item(self, ...) -> tuple[bool, pathlib.Path | str]: - # Step 2b: Duplicate prevention - if isinstance(media, Track): - track_id = str(media.id) - if self.history_service.should_skip_download(track_id): - self.fn_logger.info( - f"Skipped item '{name_builder_item(media)}' (already in history)." - ) - return False, path_media_dst -``` - -#### Post-Download Recording - -```python -# Step 6: Add to history after successful download -if download_success and isinstance(media, Track): - try: - self.history_service.add_track_to_history( - track_id=str(media.id), - source_type=source_type, - source_id=source_id, - source_name=source_name - ) - except Exception as e: - self.fn_logger.warning(f"Failed to add track to history: {e}") -``` - -**Note**: Videos are intentionally excluded from history tracking as they use a different workflow. - ---- - -### 3. GUI Integration (`tidal_dl_ng/gui.py`) - -**Modifications**: Tools menu integration - -#### Menu Action Creation - -```python -def _init_menu_actions(self) -> None: - """Initialize custom menu actions.""" - # Create or find Tools menu - tools_menu = self._get_or_create_tools_menu() - - # Add View History action - self.a_view_history = QtGui.QAction("View Download History...", self) - self.a_view_history.triggered.connect(self.on_view_history) - tools_menu.addAction(self.a_view_history) - - # Add separator - tools_menu.addSeparator() - - # Add duplicate prevention toggle - self.a_toggle_duplicate_prevention = QtGui.QAction( - "Prevent Duplicate Downloads", self - ) - self.a_toggle_duplicate_prevention.setCheckable(True) - is_preventing = self.history_service.get_settings().get("preventDuplicates", True) - self.a_toggle_duplicate_prevention.setChecked(is_preventing) - self.a_toggle_duplicate_prevention.triggered.connect( - self.on_toggle_duplicate_prevention - ) - tools_menu.addAction(self.a_toggle_duplicate_prevention) -``` - -#### Toggle Handler - -```python -def on_toggle_duplicate_prevention(self, enabled: bool) -> None: - """Toggle duplicate download prevention on or off. - - Args: - enabled: Whether duplicate prevention is enabled. - """ - self.history_service.update_settings(preventDuplicates=enabled) - status_msg = "enabled" if enabled else "disabled" - logger_gui.info(f"Duplicate download prevention {status_msg}") - self.s_statusbar_message.emit( - StatusbarMessage( - message=f"Duplicate prevention {status_msg}.", - timeout=2500 - ) - ) -``` - ---- - -### 4. Logger Configuration (`tidal_dl_ng/logger.py`) - -**Modifications**: Color configuration for INFO messages - -#### Problem - -By default, `coloredlogs` doesn't apply any color to INFO level messages (`'info': {}`), making them appear gray and less visible. - -#### Solution - -```python -# Configure custom level styles to make INFO messages green -level_styles = coloredlogs.DEFAULT_LEVEL_STYLES.copy() -level_styles['info'] = {'color': 'green'} - -formatter = coloredlogs.ColoredFormatter( - fmt=log_fmt, - level_styles=level_styles -) -``` - -#### Result - -All INFO messages (including skip messages) now display in **green** in the application console: - -``` -> Downloaded item 'Track Name'. # Green -> Skipped item 'Track Name' (already in history). # Green -> Finished list 'Playlist Name'. # Green -``` - ---- - -## User Interface - -### Tools Menu - -**Location**: Menu Bar → Tools - -**Structure**: - -``` -Tools -├── View Download History... -├── ───────────────────── -└── ☑ Prevent Duplicate Downloads -``` - -### Menu Actions - -#### 1. View Download History... - -Opens the Download History dialog showing: - -- All downloaded tracks -- Grouped by source (playlists, albums, mixes) -- Download dates and metadata -- Import/Export buttons -- Clear history option - -#### 2. Prevent Duplicate Downloads (Checkable) - -- **Checked (Default)**: Tracks in history will be automatically skipped -- **Unchecked**: Allows re-downloading tracks even if in history -- State is persisted to JSON file -- Immediate effect on download behavior -- Status message displayed in status bar - -### Console Messages - -#### Downloaded Item (Success) - -``` -> Downloaded item 'Artist - Track Title'. -``` - -**Color**: Green - -#### Skipped Item (Duplicate) - -``` -> Skipped item 'Artist - Track Title' (already in history). -``` - -**Color**: Green - -#### List Finished - -``` -> Finished list 'Playlist Name'. -``` - -**Color**: Green - ---- - -## File Structure - -### JSON File Location - -**Path**: `{config_base}/downloaded_history.json` - -**Config Base Locations**: - -- **Windows**: `%APPDATA%\tidal-dl-ng\` -- **macOS**: `~/Library/Application Support/tidal-dl-ng/` -- **Linux**: `~/.config/tidal-dl-ng/` - -### JSON Schema - -#### Version 1 (Current) - -```json -{ - "_schema_version": 1, - "_last_updated": "2025-11-29T12:34:56.789Z", - "settings": { - "preventDuplicates": true - }, - "tracks": { - "123456": { - "sourceType": "playlist", - "sourceId": "uuid-123-456", - "sourceName": "My Awesome Playlist", - "downloadDate": "2025-11-29T12:30:00.000Z" - }, - "789012": { - "sourceType": "album", - "sourceId": "uuid-789-012", - "sourceName": "Amazing Album", - "downloadDate": "2025-11-29T12:35:00.000Z" - } - } -} -``` - -#### Field Descriptions - -| Field | Type | Description | -| ---------------------------- | ------------ | ---------------------------------------------------------- | -| `_schema_version` | integer | Schema version for future migrations | -| `_last_updated` | ISO 8601 | Last modification timestamp | -| `settings.preventDuplicates` | boolean | Enable/disable duplicate prevention | -| `tracks` | object | Map of track_id → track metadata | -| `tracks[id].sourceType` | string | Source type: "playlist", "album", "mix", "manual", "track" | -| `tracks[id].sourceId` | string\|null | UUID of source, null for manual | -| `tracks[id].sourceName` | string\|null | Display name of source | -| `tracks[id].downloadDate` | ISO 8601 | When the track was downloaded | - -### Legacy Format Migration - -The service automatically migrates from the legacy format (tracks at root level): - -```json -{ - "_schema_version": 1, - "123456": { - "sourceType": "playlist", - ... - } -} -``` - -to the new format with separate `settings` and `tracks` sections. - ---- - -## API Reference - -### HistoryService - -#### Initialization - -```python -from tidal_dl_ng.history import HistoryService - -history = HistoryService() # Singleton - always returns same instance -``` - -#### Core Methods - -##### Track Operations - -```python -# Add track to history -history.add_track_to_history( - track_id="123456", - source_type="playlist", - source_id="pl-uuid", - source_name="My Playlist" -) - -# Check if track is downloaded -is_downloaded = history.is_downloaded("123456") # Returns bool - -# Get track information -info = history.get_track_info("123456") # Returns dict or None - -# Remove track from history -removed = history.remove_track_from_history("123456") # Returns bool -``` - -##### Duplicate Prevention - -```python -# Check if download should be skipped -should_skip = history.should_skip_download("123456") -# Returns True if: -# 1. Track exists in history AND -# 2. preventDuplicates setting is enabled - -# Update duplicate prevention setting -history.update_settings(preventDuplicates=True) # or False -``` - -##### Settings Management - -```python -# Get current settings -settings = history.get_settings() -# Returns: {"preventDuplicates": bool} - -# Update settings -history.update_settings(preventDuplicates=False) -# Immediately persists to JSON -``` - -##### Data Views - -```python -# Get history grouped by source -by_source = history.get_history_by_source() -# Returns: { -# "playlist_uuid-123": [track_info, ...], -# "album_uuid-456": [track_info, ...], -# "manual_manual": [track_info, ...] -# } - -# Get statistics -stats = history.get_statistics() -# Returns: { -# "total_tracks": int, -# "by_source_type": {"playlist": 10, "album": 5, ...}, -# "oldest_download": "ISO 8601", -# "newest_download": "ISO 8601" -# } -``` - -##### Import/Export - -```python -# Export history -success, message = history.export_history("/path/to/export.json") -# Returns: (True, "Successfully exported N tracks") or (False, "Error message") - -# Import history (merge mode) -success, message = history.import_history("/path/to/import.json", merge=True) -# merge=True: Adds to existing history -# merge=False: Replaces existing history - -# Clear all history -history.clear_history() # Destructive - removes all tracks -``` - -##### Utilities - -```python -# Get history file path -path = history.get_history_file_path() -# Returns: "/path/to/downloaded_history.json" - -# Save history manually (usually not needed) -history.save_history() # Most methods auto-save -``` - ---- - -## Testing - -### Test Coverage - -**Total**: 91 tests across 4 test files - -#### Test Files - -1. **test_history_service.py** (38 tests) - - - Service initialization - - CRUD operations on tracks - - Duplicate prevention logic - - Settings management - - JSON persistence and corruption recovery - - Import/Export functionality - - Statistics calculation - - Thread safety - -2. **test_download_duplicate_prevention.py** (10 tests) - - - Download skip logic - - History integration - - Log message formatting - - Post-download history updates - - Settings toggle effects - -3. **test_gui_duplicate_prevention.py** (22 tests) - - - Menu creation and structure - - Action state management - - Handler behavior - - Settings persistence - - Status messages - -4. **test_logger_configuration.py** (20 tests) - - Color configuration - - Formatter setup - - Message formatting - - coloredlogs integration - -### Running Tests - -```bash -# All tests -pytest tests/ -v - -# Specific test file -pytest tests/test_history_service.py -v - -# With coverage -pytest tests/ --cov=tidal_dl_ng --cov-report=html - -# Specific test class -pytest tests/test_history_service.py::TestDuplicatePrevention -v -``` - -### Test Results - -``` -======================== 91 passed in 0.91s ======================== -✅ test_history_service.py: 38 passed -✅ test_download_duplicate_prevention.py: 10 passed -✅ test_gui_duplicate_prevention.py: 22 passed -✅ test_logger_configuration.py: 20 passed -``` - ---- - -## Usage Examples - -### Example 1: Basic Download with History - -```python -from tidal_dl_ng.download import Download -from tidal_dl_ng.history import HistoryService - -# Initialize download service -download = Download(...) - -# Download a track (automatic history tracking) -success, path = download.item( - file_template="{artist_name} - {track_title}", - media=track, - source_type="playlist", - source_id="pl-uuid-123", - source_name="My Playlist" -) - -# Track is automatically added to history if successful - -# Check if track was downloaded -history = HistoryService() -if history.is_downloaded(str(track.id)): - print("Track is in history!") -``` - -### Example 2: Preventing Duplicates - -```python -from tidal_dl_ng.history import HistoryService - -history = HistoryService() - -# Enable duplicate prevention (default) -history.update_settings(preventDuplicates=True) - -# Check before downloading -track_id = "123456" -if history.should_skip_download(track_id): - print(f"Track {track_id} already downloaded - skipping") -else: - # Proceed with download - download.item(...) -``` - -### Example 3: Exporting History for Backup - -```python -from tidal_dl_ng.history import HistoryService - -history = HistoryService() - -# Export current history -success, message = history.export_history("/backup/my_history.json") -if success: - print(f"✅ {message}") -else: - print(f"❌ {message}") - -# Later, import it back -success, message = history.import_history( - "/backup/my_history.json", - merge=True # Merge with existing -) -``` - -### Example 4: Getting Download Statistics - -```python -from tidal_dl_ng.history import HistoryService - -history = HistoryService() - -# Get statistics -stats = history.get_statistics() - -print(f"Total tracks downloaded: {stats['total_tracks']}") -print(f"By source type:") -for source_type, count in stats['by_source_type'].items(): - print(f" {source_type}: {count}") -print(f"Oldest download: {stats['oldest_download']}") -print(f"Newest download: {stats['newest_download']}") -``` - -### Example 5: Viewing History by Source - -```python -from tidal_dl_ng.history import HistoryService - -history = HistoryService() - -# Get history grouped by source -by_source = history.get_history_by_source() - -for source_key, tracks in by_source.items(): - print(f"\n{source_key} ({len(tracks)} tracks):") - for track in tracks: - print(f" - Track {track['track_id']} - {track['source_name']}") - print(f" Downloaded: {track['download_date']}") -``` - -### Example 6: GUI Toggle Integration - -```python -# In GUI code -def on_toggle_duplicate_prevention(self, enabled: bool) -> None: - """User toggled the menu option.""" - # Update setting - self.history_service.update_settings(preventDuplicates=enabled) - - # Show feedback - status = "enabled" if enabled else "disabled" - self.show_status(f"Duplicate prevention {status}") - - # Setting is immediately persisted to JSON -``` - ---- - -## Migration Guide - -### From No History System - -1. **No action needed**: The system automatically creates an empty history on first run -2. Existing downloads won't be in history initially -3. New downloads will be tracked going forward - -### From Legacy Format - -The system automatically migrates old format files: - -**Old format** (tracks at root): - -```json -{ - "_schema_version": 1, - "123456": { "sourceType": "...", ... } -} -``` - -**New format** (tracks in section): - -```json -{ - "_schema_version": 1, - "settings": { "preventDuplicates": true }, - "tracks": { - "123456": { "sourceType": "...", ... } - } -} -``` - -Migration happens automatically on load - no user intervention required. - ---- - -## Performance Considerations - -### Lookup Performance - -- **Track existence check**: O(1) - uses Python dict -- **Should skip check**: O(1) - dict lookup + boolean check -- **Memory usage**: ~100 bytes per track entry -- **File size**: ~150-200 bytes per track in JSON - -### Benchmarks - -| Operation | Time (avg) | Notes | -| -------------------------- | ---------- | ------------------- | -| Add track | <1ms | Includes JSON write | -| Check if downloaded | <0.1ms | Dict lookup only | -| Should skip check | <0.1ms | Dict lookup + bool | -| Load history (1000 tracks) | ~50ms | On startup only | -| Save history (1000 tracks) | ~100ms | Atomic write | - -### Concurrency - -- **Thread-safe**: All operations use `threading.Lock` -- **No deadlocks**: Lock held for minimal time -- **No race conditions**: Atomic file writes prevent corruption - ---- - -## Troubleshooting - -### Issue: History file corrupted - -**Solution**: The system automatically: - -1. Creates a backup `.json.bak` file -2. Starts with empty history -3. Logs the corruption for investigation - -### Issue: Duplicate prevention not working - -**Check**: - -1. Is the setting enabled? `history.get_settings()["preventDuplicates"]` -2. Is the track actually in history? `history.is_downloaded(track_id)` -3. Check logs for skip messages - -### Issue: History lost after crash - -**Note**: The system uses atomic writes, so: - -- Either the old file is intact, OR -- The new file is complete -- Never partially written - -### Issue: Import fails - -**Common causes**: - -- Invalid JSON syntax → Fix JSON file -- Missing required fields → Add `sourceType`, `downloadDate` -- Wrong file format → Use export from same version - ---- - -## Future Enhancements - -### Planned Features - -- [ ] Track file paths in history (verify file still exists) -- [ ] Smart re-download detection (file deleted but in history) -- [ ] Quality-based history (allow re-download if better quality available) -- [ ] Batch operations (mark multiple as downloaded/not downloaded) -- [ ] Search and filter in history dialog -- [ ] Export to CSV/Excel for analysis -- [ ] Sync history across devices - -### API Stability - -The current API is considered **stable** and follows semantic versioning: - -- Patch versions (1.0.x): Bug fixes only -- Minor versions (1.x.0): New features, backward compatible -- Major versions (x.0.0): Breaking changes to API or file format - ---- - -## Contributing - -### Code Standards - -All code follows the project's AGENTS.md guidelines: - -- ✅ Type hints for all functions -- ✅ Docstrings (Google style) -- ✅ PEP 8 compliance (via Black, Ruff) -- ✅ Thread safety where needed -- ✅ Comprehensive tests -- ✅ Error handling - -### Adding New Features - -1. Update `HistoryService` class -2. Add tests to `test_history_service.py` -3. Update JSON schema if needed -4. Update this documentation -5. Run all tests: `pytest tests/` -6. Run quality checks: `make check` - ---- - -## License - -This feature is part of tidal-dl-ng and follows the same license as the main project. - ---- - -## Changelog - -### Version 1.0.0 (2025-11-29) - -**Initial Release** - -- ✅ Persistent JSON-based download history -- ✅ Duplicate prevention with toggle -- ✅ Thread-safe operations -- ✅ Atomic file writes -- ✅ Corruption recovery -- ✅ Import/Export functionality -- ✅ Statistics and source views -- ✅ GUI integration (Tools menu) -- ✅ Green console messages for INFO level -- ✅ Comprehensive test suite (91 tests) -- ✅ Complete documentation - ---- - -## Support - -For issues, questions, or feature requests: - -1. Check this documentation -2. Review test files for usage examples -3. Open an issue on GitHub with: - - Description of the problem - - Steps to reproduce - - Expected vs actual behavior - - Log output (if applicable) - ---- - -**Last Updated**: 2025-11-29 -**Version**: 1.0.0 -**Status**: Production Ready ✅ |