v2.4.1

v2.4.1

This release resolves the Searx port inconsistency issue #215 and improves the stability and maintainability of the Data Leak module. All relevant configuration files (Watcher/settings.py, .env, docker-compose.yml, and Searx/searx/settings.yml) now consistently use port 8080 for Searx.

This ensures that Watcher can correctly query Searx without encountering connection errors.

Update Procedure

Please follow this process :

1. Pull the latest Docker image from the repository.
2. Stop running containers:

   docker compose down

3. Update your .env file:
   • Check DATA_LEAK_SEARX_URL
4. Rebuild and restart containers with :

   docker compose down
   docker compose up

What’s Changed

Searx Port Standardization:

• Fixed inconsistent default ports (8080 vs 8888). Watcher now consistently uses http://searx:8080/.
• Watcher respects the DATA_LEAK_SEARX_URL environment variable if set, allowing custom Searx hosts/ports.

Data Leak Module

• Minor bug fixes and improvements in notification handling.

Full Changelog: v2.4...v2.4.1

v2.4

v2.4

This release brings major improvements to the Threat Watcher module, including a new word reliability scoring, state-of-the-art NER detection, reduced false positives, a smarter trending algorithm, and several bug fixes and optimizations.

Update Procedure for Docker

Please follow this process:

[WARNING] Manual Deletion Step:

This operation will permanently delete all existing data in the Source, BannedWord, and TrendyWord tables.
If you have custom sources, banned words, or other critical data, make sure to back them up or export them before proceeding.

Before anything else, clean existing data to avoid conflicts. Run the following commands in the Django shell in this order:

python manage.py shell -c "from threats_watcher.models import Source, BannedWord, TrendyWord; Source.objects.all().delete(); BannedWord.objects.all().delete(); TrendyWord.objects.all().delete()"

Then continue with the update procedure:

1. Pull the latest Docker image from the repository.

2. Stop all containers:

   docker compose down

3. Apply database migrations and Repopulate the database with the new blocklist and sources (new fields added):

   docker compose run watcher bash
   python manage.py migrate
   python manage.py populate_db

4. Restart the containers:

   docker compose up

If you run Watcher without Docker

1. Install all system dependencies

 sudo apt update && sudo apt install -y \
     build-essential \
     libsasl2-dev \
     libldap2-dev \
     libssl-dev \
     curl \
     git

2. Install Rust (required for tokenizers/transformers)

curl https://sh.rustup.rs -sSf | sh -s -- -y
source $HOME/.cargo/env

3. (Re)install Python dependencies

pip install --upgrade pip
pip install --no-cache-dir -r requirements.txt

4. Install torch, torchvision, torchaudio with CPU support

pip install --extra-index-url https://download.pytorch.org/whl/cpu torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0

5. Install NLTK dependencies

python ./nltk_dependencies.py

What’s Changed

ThreatWatcher – Major Improvements

• Reliability scoring for each trending word:

  • Each source in sources.csv now features a confident score (1 = 100%, 2 = 50%, 3 = 20%).
  • The reliability for each word is the average confidence of the sources where it appeared.
  • New field shown in UI (“Reliability %” column).

• Entity extraction now uses BERT-base-NER:

  • Improved word/entity detection in news titles.
  • 10× smaller blocklist needed; blocklist file reduced.
  • Vastly fewer false positives.
  • For more information on BERT-base-NER : https://huggingface.co/dslim/bert-base-NER

• Trending algorithm refactor:

  • Now only the last 30 days of news headlines are used for trending word calculation.
  • Old: Words could “dominate” from historic surges (e.g. 200 hits a year ago + 1 this month = trending).
  • New: Words must truly be trending this month to rank.
  • Minimum occurrences for trend detection reduced from 7 → 5.

• Improved testing coverage:

  • Three new unit tests added in the backend to validate recent changes.
  • Existing frontend tests adjusted to reflect UI updates (e.g. Reliability column).

• Improved Entity Detection, Reliability Scoring, and Trending Algorithm by @ygalnezri and @LeonNadot in #224

• v2.4 by @ygalnezri and @LeonNadot in #225

Breaking changes & warnings

• If you use custom code for word parsing/blocklist:
  • Review your blocklist (now much smaller).
  • Word detection logic has changed (BERT, NER).
• sources.csv structure:
  • Now requires a confident column.
  • Ensure your source feeds are updated to match the new format.
• Database migration required (new fields).
• Minimum word occurrence is now 5 (was 7), can be changed in settings.py.

Full Changelog: v2.3...v2.4

v2.3

v2.3

This release introduces a significant upgrade in automated testing coverage, improving both back-end and front-end reliability. The goal is to ensure that every new feature or modification in Watcher is backed by robust, automated unit tests. This update directly strengthens the stability and maintainability of the project across all modules.

Update Procedure

No breaking change or configuration is required for this release. However, contributors must ensure they follow the updated test commands and conventions detailed in the updated documentation.

What’s Changed

Test Coverage

Unit Tests (Back-End)

• 99 Django unit tests across the main back-end modules:

  • common/tests.py
  • watcher/tests.py
  • Individual tests.py files per module

End-to-End Tests (Front-End with Cypress)

• 4 full-featured Cypress suites covering the entire front-end application:
  • DataLeak.cy.js: 31 tests
  • DnsFinder.cy.js: 32 tests
  • SiteMonitoring.cy.js: 26 tests
  • ThreatsWatcher.cy.js: 31 tests
• Total: 120 Cypress tests successfully passed with no failures or pending cases.

CI/CD Integration

• All tests are automatically executed in our CI/CD pipeline using GitHub Actions:

  • Triggered on: Push, Pull Requests, and manual workflow dispatch
  • Execution: Both back-end and front-end tests run automatically
  • Coverage: Full test suite validation before code integration

• The CI/CD workflow ensures that:

  • No broken code reaches the main branch
  • All new features are properly tested

Developer Notes

All test commands must now be run from the Watcher/Watcher directory:

cd Watcher/Watcher

Back-End Tests

To run all Django unit tests:

python manage.py test

Front-End Tests

Before running front-end tests, you need to create a test superuser:

python manage.py shell -c "
from django.contrib.auth.models import User
User.objects.create_superuser('Watcher', '[email protected]', 'Watcher', first_name='Unit-Test Cypress', last_name='Watcher')"

To run all Cypress Test Runner:

npm run test:e2e

If you want more information about the commands and other instructions, please refer to the documentation.

Why it matters

This update lays the foundation for a more robust and scalable Watcher ecosystem. By enforcing test-driven development and automated validation, we ensure faster releases, fewer regressions, and a cleaner development experience for contributors.

From now on: All Pull Requests must include tests for new functionality. PRs without adequate test coverage may be rejected

Full Changelog: v2.2.0...v2.3

v2.2.0

v2.2.0

This release focuses on a complete overhaul of the MISP integration, improved code modularity, and the resolution of several front-end and dependency issues. It also addresses important issues such as #2 (creating MISP objects instead of attributes) and #206 (HTTP 500 errors when exporting IOC), improving overall export reliability and functionality. These updates aim to increase stability, maintainability, and flexibility when using Watcher with MISP.

Update Procedure

Please follow this process :

1. Pull the latest Docker image from the repository.
2. Apply any migrations:

   docker compose down
   docker compose run watcher bash
   python manage.py migrate

3. Update your .env file:
   • Check MISP_URL, MISP_KEY, and MISP_VERIFY_SSL.
4. Rebuild and restart containers with :

   docker compose down
   docker compose up

[WARNING] If you were using custom scripts for MISP export, you will need to adapt them to the new common.misp module.

What’s Changed

MISP Integration Redesign

• New centralized MISP logic: The common module now handles all MISP-related interactions by @ygalnezri in #207
• Shared MISP module across all features (e.g. dns_finder, website_monitoring), avoiding code duplication.
• Possibility to specify a MISP Event UUID when exporting IOCs:

  • If the UUID exists, Watcher will detect and update the corresponding event.

  • Full support for both automatic and manual updates.

  • Enhanced export logic for MISP objects with support for object creation (e.g., domain, ip-port).

    This redesign addresses key issues including #2 (creating MISP objects instead of attributes) and #206 (HTTP 500 error when exporting IOC in Twisted DNS finder).

For more details on the new MISP integration, the MISP Export section of the documentation has been fully updated: MISP Export

• Change your variable name based on the new .env format: the setting has been renamed from ALLOWED_HOST to ALLOWED_HOSTS. You must now explicitly define it in .env (e.g., ALLOWED_HOSTS=localhost,127.0.0.1) to match your environment's domain list. This update ensures proper host validation in settings.py and prevents runtime errors during container startup by @ygalnezri in #185

Dependency and Security Updates

• Fixed minor display issues detected during development
• Bump @babel/runtime from 7.26.0 to 7.27.0 in /Watcher by @dependabot #189
• Bump axios from 1.7.9 to 1.8.2 in /Watcher by @dependabot #190
• Bump react-router and react-router-dom in /Watcher by @dependabot #191

Full Changelog: v2.1.4...v2.2.0

v2.1.4

v2.1.4

This update focuses on improving stability in the site monitoring process and fixing environment configuration issues related to host validation.

Update Procedure

Please follow this process.

What’s Changed

• Change your variable name based on the new .env format: the setting has been renamed from ALLOWED_HOST to ALLOWED_HOSTS. You must now explicitly define it in .env (e.g., ALLOWED_HOSTS=localhost,127.0.0.1) to match your environment's domain list. This update ensures proper host validation in settings.py and prevents runtime errors during container startup by @ygalnezri in #185
• Resolved an issue in SiteSerializer and monitoring_init function, allowing for smooth execution of the site monitoring process without blocking subsequent domain additions by @ygalnezri in #185

Full Changelog: v2.1.3...v2.1.4

v2.1.3

v2.1.3

This update improves domain name validation and fixes issues related to regex validation for Domain Name & Ticket ID. Additionally, a bug in the search functionality for the Alert model in the DNS Finder and Data Leak modules has been resolved.

Update Procedure

Please follow this process.

What’s Changed

• Fixed regex validation for Domain Name & Ticket ID on the front-end, ensuring accurate input validation by @ygalnezri
• Improved domain name validation with tldextract, enhancing reliability in domain handling by @ygalnezri
• Fixed a bug in search functionality for the Alert model, resolving issues in the DNS Finder and Data Leak modules on the admin interface by @ygalnezri
• Refactored the site creation method in Site Monitoring module by @radomir-mijovic

New Contributors

• @radomir-mijovic made their first contribution in #179

Full Changelog: v2.1.2...v2.1.3

v2.1.2

v2.1.2

This update fixes the handling of parent domain actions in DNS Finder, improving tagging accuracy in TheHive. It also enhances TheHive integration by resolving an issue that prevented the creation of grouped alerts for DNS Finder. Additionally, RSS sources have been updated, with obsolete sources removed and new, relevant cybersecurity sources added. Lastly, a time-based verification mechanism has been introduced in Website Monitoring to prevent duplicate alerts from being generated unnecessarily.

Update Procedure

[WARNING] RSS Sources Update:

We have removed obsolete RSS sources and replaced them with new sources related to cybersecurity. To populate the new RSS sources, run the following command:

python manage.py populate_db

Refer to the updated documentation for details: Update Watcher.

What’s Changed

• Fixed incorrect action handling for parent domains in DNS Finder, improving tagging accuracy in TheHive by @ygalnezri
• Resolved the issue preventing group alerts from being created in TheHive for DNS Finder by @ygalnezri
• Updated RSS sources, adding relevant ones and removing obsolete ones by @ygalnezri
• Fixed duplicate alerts in Website Monitoring, adding a time interval check to prevent unnecessary duplicates by @ygalnezri
• Fixed domain identification logic in DNS Finder, ensuring proper handling of TLDs with multiple segments (e.g., second-level TLDs) by @ygalnezri
• v2.1.2 by @ygalnezri in #171

Full Changelog: v2.1.1...v2.1.2

v2.1.1

v2.1.1

This release improves TheHive integration, refining TLP, PAP, Severity, and observable tags, while enhancing alert and case management. The system now verifies existing entries before updating them with new observables, preventing duplicates. Additionally, Dockerfile optimizations enhance container performance and security.

Watcher now has the ability to automatically feed cases and alerts by adding DNS Finder alerts linked to a monitored domain in Website Monitoring. Watcher will add subdomains to the case or alert of the parent domain and automatically update it.

---

Update Procedure

Please follow this process.

• If you want, you can update the .env configuration to include credentials and endpoints for TheHive and other notification channels. Refer to the updated documentation for details: Update Watcher.

---

What’s Changed

• This release enhances TheHive integration, ensuring alerts and cases are automatically updated with new observables for better incident tracking by @ygalnezri.
  • Watcher now automatically feeds cases and alerts by adding DNS Finder alerts linked to a monitored domain in Website Monitoring. Subdomains are added to the case or alert of the parent domain, ensuring they are automatically updated.
• Refined TLP, PAP, Severity, and observable tags, improving classification and response accuracy by @ygalnezri.
• Optimized the Dockerfile following best practices, improving performance and security by @0xlildoudou in #163
• Bump django from 5.0.10 to 5.0.11 in /Watcher by @dependabot in #168

---

New Contributors

• @0xlildoudou made their first contribution in #163

---

Full Changelog: v2.1...v2.1.1

v2.1

v2.1

This release focuses on a major transformation of the notification system, introducing new integrations with TheHive, Citadel, and Slack, alongside an enhancement to email notifications. Users can now automate alert creation in TheHive, send notifications via Citadel's APIs, and streamline team communication through Slack. Additionally, email notifications have been upgraded to SMTPS for improved security. These updates aim to enhance communication, security, and collaboration across multiple platforms.

Update Procedure

[MANDATORY] Update Watcher:
This version includes significant changes to the notification system. It is essential to follow these steps:

1. Pull the latest Docker image from the repository.
2. Update your docker-compose.yml file as per the latest version on GitHub.
3. Apply migrations for the newly created module:

   python manage.py migrate

4. Update configurations in the .env file to include credentials and endpoints for the new notification channels. Refer to the updated documentation for details: Update Watcher.

New Features

• Common Module Implementation:
  A new Django app has been introduced to centralize generic functions shared by multiple modules. This update simplifies the codebase and enhances maintainability.

• Notification System Creation:

  • Email Notifications via SMTPS:
    Transitioned from SMTP to SMTPS for enhanced security. More details and information about this: Configure your Email notifications by @ygalnezri.
  • TheHive Integration:
    Automatic alert creation in TheHive via APIs. More details and information about this: Configure your TheHive notifications by @ygalnezri.
  • Citadel Integration:
    Notifications are now supported through the enterprise application Citadel via APIs. More details and information about this: Configure your Citadel notifications by @ygalnezri.
  • Slack Notifications:
    Notifications can now be delivered directly through Slack via APIs for better team collaboration. More details and information about this: Configure your Slack notifications by @ygalnezri.

What’s Changed

• Redesigned email templates for improved clarity and user experience by @ygalnezri.
• Refactored the docker-compose.yml file to use env_file for better readability and reduced redundancy by @ygalnezri.
• Updated the Dockerfile to align with the latest best practices for Django applications by @ygalnezri.
• Revised documentation to provide detailed setup instructions for the new notification system and its integrations by @ygalnezri
• v2.1 by @ygalnezri in #166

Full Changelog: v2.0.2...v2.1

v2.0.2

What's Changed

• Bump path-to-regexp and react-router-dom in /Watcher by @dependabot in #156
• Bump django from 5.0.8 to 5.0.9 in /Watcher by @dependabot in #158

Full Changelog: v2.0.1...v2.0.2