web-watcher

A lightweight bash script to watch APIs and websites for changes — get instant notifications from your terminal.

web-watcher polls any URL at a configurable interval, compares responses, and sends you a desktop notification + terminal alert when something changes. Built for monitoring sneaker drops, stock APIs, price changes, website updates — anything with a URL.

Features

API & Website monitoring — watch JSON APIs or full web pages
Auto-detection — automatically detects JSON vs HTML from Content-Type
Change threshold — set a minimum % of change to trigger alerts (ignore noise)
Desktop notifications — native macOS (osascript) and Linux (notify-send) support
Slack, Discord & Telegram — webhook notifications to messaging platforms
jq filtering — target specific JSON fields (e.g. .products[].price)
HTML selector — grep patterns to monitor specific parts of a page
Custom headers & auth — Bearer tokens, cookies, Basic auth, custom User-Agent
POST support — watch API search endpoints with custom request bodies
Retry logic — configurable retries with backoff on failure
Snapshots — save every response to disk for later analysis
Diff output — see exactly what changed between checks
Logging — timestamped log file of all changes
Cron-friendly — --once mode for single checks with exit codes

Installation

Homebrew

If you use Homebrew, you can install web-watcher via the tap:

brew install maxgfr/tap/web-watcher
web-watcher --help

Manual

git clone https://github.com/maxgfr/web-watcher.git
cd web-watcher
chmod +x script.sh

Optionally, add it to your PATH:

sudo ln -s "$(pwd)/script.sh" /usr/local/bin/web-watcher

Dependencies

Only curl is required. jq is needed if you use the --filter option. In website mode, webindex is recommended for the most accurate HTML-to-text extraction (brew install maxgfr/tap/webindex); the Homebrew formula for web-watcher installs it automatically. perl (present on macOS and nearly every Linux distribution) is recommended as a fallback; without either tool, a sed/awk fallback is used. python3 is only needed to run the test suite.

macOS:

brew install curl jq

Ubuntu/Debian:

sudo apt-get install curl jq

Arch:

sudo pacman -S curl jq

Quick Start

# Watch a JSON API every 30 seconds
./script.sh https://api.example.com/products

# Watch a sneaker stock page, check every 10 seconds
./script.sh -i 10 -m website https://www.nike.com/launches

# Watch a JSON API with auth, filter on price
./script.sh -i 15 \
  -H 'Authorization: Bearer mytoken' \
  -f '.products[].price' \
  https://api.sneakers.com/v1/stock

Agent skill

Install the web-watcher skill with the skills CLI:

npx skills add maxgfr/web-watcher --skill web-watcher

For a global Codex installation:

npx skills add maxgfr/web-watcher --skill web-watcher --agent codex --global

The skill is manual only in Codex (allow_implicit_invocation: false) and Claude Code (disable-model-invocation: true). Invoke it as $web-watcher in Codex or /web-watcher in Claude Code, for example:

$web-watcher Watch https://example.com/ every 60 seconds for 10 checks and show me content changes.

It guides the agent through selecting content, inspecting the baseline, starting the watch, and reporting changes or errors. The CLI must be installed separately, for example with brew install maxgfr/tap/web-watcher. Installing the skill does not start a watch. Other agents may handle invocation policies differently.

Usage

web-watcher [options] <url>

Request Options

Option	Description	Default
`-X, --method <METHOD>`	HTTP method (GET, POST, PUT...)	`GET`
`-H, --header <header>`	Custom header (repeatable)	—
`-d, --data <body>`	Request body for POST/PUT	—
`-C, --cookie <cookie>`	Cookie string or file path	—
`-A, --user-agent <ua>`	Custom User-Agent	`web-watcher/<version>`
`--auth <user:pass>`	Basic auth credentials	—
`--timeout <secs>`	Request timeout	`15`
`--no-follow`	Don't follow redirects	follows
`--insecure`	Allow insecure SSL	disabled

Watch Options

Option	Description	Default
`-i, --interval <secs>`	Seconds between checks	`30`
`-p, --threshold <percent>`	Min change % to trigger alert	`0` (any)
`-n, --max-runs <num>`	Stop after N checks (0 = unlimited)	`0`
`--once`	Run single check then exit	disabled
`--baseline-file <file>`	Persist baseline to disk (for `--once`)	—
`--retries <num>`	Retries on failure	`3`
`--retry-delay <secs>`	Delay between retries	`5`

Content Options

Option	Description	Default
`-m, --mode <mode>`	`api`, `website`, or `auto`	`auto`
`-f, --filter <jq>`	jq filter for JSON (e.g. `.data.price`)	—
`-s, --selector <pattern>`	Grep pattern for HTML content	—
`--strip-html`	Force strip HTML tags	disabled
`--full-page`	Keep navigation, header, footer, aside and cookie-banner text (website mode)	disabled
`--ignore <regex>`	Drop lines matching this pattern before comparing (repeatable)	—

--ignore accepts POSIX extended regular expressions (ERE) and applies in all modes.

Notification Options

Option	Description	Default
`--slack <url>`	Slack incoming webhook URL	—
`--discord <url>`	Discord webhook URL	—
`--telegram-token <token>`	Telegram bot token	—
`--telegram-chat <chat_id>`	Telegram chat ID	—

Webhook calls fail loudly: an HTTP error from Slack, Discord or Telegram is reported with a [WARN] line instead of being ignored. Messages are sent as plain text, so URLs containing _, * or & are delivered unchanged.

Environment Variables

Variable	Description	Default
`WW_TELEGRAM_API`	Base URL of the Telegram Bot API (useful for proxies or tests)	`https://api.telegram.org`
`WW_HTML_STRIPPER`	Force the HTML-to-text implementation: `webindex`, `perl`, or `sed`	`webindex` if installed → `perl` if available → `sed`

webindex is preferred for balanced nested tags, charset detection, and main-content isolation. The perl and sed backends use regex-based stripping and a sed/awk scanner, respectively, without balancing nested tags: nested <nav> blocks are cut at the first </nav>.

Output Options

Option	Description	Default
`-l, --log <file>`	Log changes to file	—
`--snapshot-dir <dir>`	Save response snapshots	—
`--diff`	Show unified diff on change	disabled
`--no-sound`	Disable terminal bell	enabled
`-q, --quiet`	Only show changes	disabled
`-v, --verbose`	Debug output	disabled
`--no-color`	Disable colors	enabled

Modes Explained

Mode	Behavior
`auto`	Looks at `Content-Type` header: JSON → `api`, HTML → `website`
`api`	Compares raw response body (JSON, XML, plain text)
`website`	Strips HTML tags, normalizes whitespace, compares text content

In website mode, the compared text has one line per block (paragraph, list item, table cell, heading…). The change percentage is the share of these lines that differ. Source line breaks become spaces, including inside <pre> blocks; <br> starts a new line. Page chrome (nav, header, footer, aside) and cookie-banner lines are removed by default; use --full-page to keep them.

Examples

Watch a sneaker API for stock changes

./script.sh -i 10 \
  -H 'Authorization: Bearer mytoken' \
  -H 'Accept: application/json' \
  -f '.products[] | {name, price, available}' \
  https://api.sneakers.com/v1/stock

Watch a website for content changes (5% threshold)

./script.sh -m website -p 5 -i 120 \
  https://www.nike.com/launches

Watch Hacker News, ignoring the lines that always move

web-watcher --once -m website --ignore 'ago|points' --baseline-file /tmp/hn.txt https://news.ycombinator.com/

Watch a full page, including navigation and cookie-banner text

web-watcher -m website --full-page --diff https://example.com/

Watch a specific section of a webpage

./script.sh -m website \
  -s 'class="product-price"' \
  -i 60 \
  https://www.shop.com/product/air-jordan-1

POST request to a search API

./script.sh -X POST \
  -H 'Content-Type: application/json' \
  -d '{"query": "jordan 1", "size": "42"}' \
  -f '.results[].price' \
  -i 30 \
  https://api.shop.com/search

Log everything + save snapshots

./script.sh -i 60 \
  -l changes.log \
  --snapshot-dir ./snapshots \
  --diff \
  https://api.example.com/data

Use with cron (check every 5 minutes)

# crontab -e
*/5 * * * * /path/to/script.sh --once --baseline-file /tmp/ww_status.txt -q -l /var/log/web-watcher.log https://api.example.com/status

The --baseline-file flag persists the previous response to disk so --once can compare across cron runs. With a threshold, a minor change below it does not replace the stored baseline, so drift accumulates until it crosses the threshold, exactly as in continuous mode.

Exit codes for --once mode:

0 — No change detected, change below threshold, or first run
1 — Fetch error
2 — Change detected (meeting the threshold, if set)

With --max-runs, the exit code is 1 if no fetch ever succeeded, otherwise 0.

Watch with cookie and Basic auth

./script.sh \
  --auth admin:secret123 \
  -C "session=abc123; token=xyz" \
  -i 45 \
  https://internal.company.com/api/dashboard

Quiet mode for background monitoring

./script.sh -q --no-sound -l watch.log -i 30 https://api.example.com/prices &

How It Works

┌─────────────────┐
│   Fetch URL     │ ← curl with headers, auth, cookies, retries
└────────┬────────┘
         │
┌────────▼────────┐
│  Detect Mode    │ ← auto / api / website (from Content-Type)
└────────┬────────┘
         │
┌────────▼────────┐
│ Process Content │ ← jq filter → grep selector → strip HTML → ignore matching lines
└────────┬────────┘
         │
┌────────▼────────┐
│ Compare with    │ ← diff-based change % calculation
│ Previous        │
└────────┬────────┘
         │
    ┌────▼────┐
    │ Changed?│
    └────┬────┘
     No  │  Yes
     │   │
     │   ├── Check threshold
     │   ├── Send notification (desktop + terminal)
     │   ├── Log to file
     │   ├── Save snapshot
     │   └── Show diff
     │
     ▼
   Sleep interval → Loop

Notifications

Platform	Method
macOS	`osascript` — native Notification Center with sound
Linux	`notify-send` — standard desktop notification
Slack	Incoming webhook — posts to a channel
Discord	Webhook — posts to a channel
Telegram	Bot API — sends a message to a chat
All	Terminal bell (`\a`) + colored terminal output

Slack

Create an Incoming Webhook in your Slack workspace
Pass the webhook URL with --slack:

./script.sh --slack https://hooks.slack.com/services/T.../B.../xxx \
  -i 30 https://api.example.com/data

Discord

In your Discord channel, go to Settings > Integrations > Webhooks and create a webhook
Pass the webhook URL with --discord:

./script.sh --discord https://discord.com/api/webhooks/123/abc \
  -i 30 https://api.example.com/data

Telegram

Pass your Telegram bot token and chat ID with --telegram-token and --telegram-chat:

./script.sh --telegram-token 123456:ABC-DEF --telegram-chat 987654321 \
  -i 30 https://api.example.com/data

Upgrading

After upgrading to this version, an existing --baseline-file captured by an older version in website mode will be reported as changed exactly once because the stored text representation changed from one word per line to one line per block. The change percentage is now per text line. <pre> blocks lose their internal line breaks.

Tips

Start with short intervals for testing (-i 5), then increase for production
Use --verbose to debug header/response issues
Combine --diff with --log to keep full audit trails
Use --threshold to avoid false positives on dynamic sites (ads, timestamps, etc.)
Use --snapshot-dir to build a history of responses you can analyze later
JSON API? Always use -f to target the fields you care about — avoids noise from metadata changes

Tests

The test suite needs python3 (it serves fixtures from a local HTTP server, no network access required) and runs every test under bash from your PATH plus /bin/bash, and under the C locale plus fr_FR.UTF-8 when it is installed (to catch decimal-comma formatting bugs):

bash tests/run.sh              # whole suite
bash tests/run.sh t_help_shows_usage t_once_captures_baseline   # a subset
WW_TEST_SHELLS=bash WW_TEST_LOCALES=C bash tests/run.sh          # single shell / locale

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
skills/web-watcher		skills/web-watcher
tests		tests
.releaserc		.releaserc
.version-hook.sh		.version-hook.sh
LICENSE		LICENSE
README.md		README.md
script.sh		script.sh

Latest commit

History

Folders and files

Repository files navigation