Skip to content

feat: Add comprehensive error handling module with custom exceptions#106

Open
xingzihai wants to merge 3 commits intomvanhorn:mainfrom
xingzihai:feature/improved-error-handling
Open

feat: Add comprehensive error handling module with custom exceptions#106
xingzihai wants to merge 3 commits intomvanhorn:mainfrom
xingzihai:feature/improved-error-handling

Conversation

@xingzihai
Copy link
Copy Markdown

Summary

This PR adds a unified error handling module (scripts/lib/errors.py) that significantly improves error handling and logging capabilities across the last30days-skill project.

Changes

1. Custom Exception Hierarchy

A structured exception hierarchy with the following types:

  • Last30DaysError: Base exception for all project errors
  • NetworkError: Network-level errors (DNS, connection refused)
  • APIError: External API errors with status code and response body
  • RateLimitError: Rate limiting errors (HTTP 429) with retry_after support
  • TimeoutError: Operation timeout errors
  • AuthError: Authentication/authorization errors
  • ConfigError: Configuration errors
  • ValidationError: Input validation errors
  • ParsingError: Data parsing errors
  • DependencyError: Missing dependency errors with install instructions
  • SourceUnavailableError: Data source unavailability errors

2. Rich Error Context

  • ErrorContext dataclass for structured debugging information
  • ErrorSeverity enum (LOW, MEDIUM, HIGH, FATAL)
  • ErrorCategory enum for error classification
  • Automatic traceback capture in debug mode
  • Serialization support via to_dict()

3. Retry Mechanism

  • @with_retry decorator with exponential backoff
  • Support for RateLimitError.retry_after to respect API rate limits
  • Configurable retry callbacks
  • Jitter to avoid thundering herd problem
  • Async version available

4. Utility Functions

  • ErrorAggregator for collecting multiple errors in parallel operations
  • wrap_exception() to convert generic exceptions to Last30DaysError
  • safe_call() for safe function execution with fallback
  • http_error_to_last30days() to bridge existing http.HTTPError

5. Comprehensive Test Suite

  • 43 unit tests covering all exception types
  • Tests for retry decorator behavior
  • Tests for error aggregation and utilities

Testing

All 43 tests pass:

python3 -m unittest tests.test_errors -v

Benefits

  1. Better Debugging: Rich error context with source, operation, and extra data
  2. Structured Error Handling: Categorized errors with severity levels
  3. Automatic Recovery: Built-in retry mechanism for transient failures
  4. API Integration: RateLimitError.retry_after is automatically used by retry decorator
  5. Backward Compatible: Works alongside existing http.HTTPError and RedditRateLimitError

- Add centralized path configuration in env.py with XDG-compliant defaults
- Support environment variable overrides for all paths:
  - LAST30DAYS_CONFIG_DIR for config directory
  - LAST30DAYS_DATA_DIR for data directory
  - LAST30DAYS_DATABASE_PATH for database file
  - LAST30DAYS_CACHE_DIR for cache directory
  - LAST30DAYS_OUTPUT_DIR for output directory
  - LAST30DAYS_BRIEFS_DIR for briefs directory
- Update all modules to use centralized path functions
- Update UI to dynamically show actual config paths
- Add fallback to temp directories when permission denied
- Update README.md with path customization documentation

This allows users to customize storage locations for different
deployment environments (containers, shared systems, etc.)
This PR adds extensive test coverage for:

**Score Module (test_score_comprehensive.py):**
- All engagement calculation functions (Reddit, X, YouTube, TikTok, Instagram, HN, Bluesky, TruthSocial, Polymarket)
- Edge cases for log1p_safe utility function
- Detailed tests for normalize_to_100 with edge cases
- Scoring functions for all item types including YouTube, TikTok, Instagram, Polymarket
- WebSearch scoring with query type awareness
- Sort order with query-type-aware tiebreakers
- Date confidence penalties
- Unknown engagement penalties

**Normalize Module (test_normalize_comprehensive.py):**
- Comprehensive tests for all item types (Reddit, X, YouTube, TikTok, Instagram, HN, Bluesky, TruthSocial, Polymarket)
- Date filtering edge cases including boundary dates and None dates
- Items_to_dicts conversion for all item types
- Engagement parsing for different platforms
- Cross-type normalization consistency tests

Total: 92 new tests added
Adds unified error handling with:
- Custom exception hierarchy (APIError, RateLimitError, TimeoutError, etc.)
- ErrorContext for rich debugging information
- @with_retry decorator with exponential backoff
- ErrorAggregator for parallel operations
- Comprehensive test suite (43 tests)
xingzihai added a commit to xingzihai/last30days-skill that referenced this pull request Mar 26, 2026
The query 'nano banana pro prompting' was tested in both test_product_queries
and test_howto_queries with conflicting expectations. Since 'prompting' matches
_HOWTO_PATTERNS before _PRODUCT_PATTERNS due to priority order, the correct
result is 'how_to'. Removed the incorrect assertion from test_product_queries.

This is a pre-existing test bug (from PR mvanhorn#65), not introduced by PR mvanhorn#106.
@mvanhorn
Copy link
Copy Markdown
Owner

See my reply on #109 covering all your PRs. Can't commit to anything right now with the v3.0 refactor underway, but I promise I'll consider each one once it lands. Appreciate the work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants