Skip to content

Commit 720ee52

Browse files
aidenapplclaude
aidenappl
and
claude
committed
Skip uncrawlable outlets, sanitise feed URLs, fix parser attribution
Three follow-ups from watching the crawl summaries in production. Blocked outlets. Reuters and Barron's answer every request with 401 no matter what headers are sent, and accounted for 239 of the 241 remaining backlog articles. They are now ingested for their headline and symbols but excluded from the crawl query and the backlog count, so the backlog stays a useful health signal instead of resting permanently on an uncrawlable residue. Configurable via CRAWL_BLOCKED_DOMAINS; "none" disables it. The exclusion is applied in SQL for the same reason the retry exclusion is: the listing is capped and ordered newest-first, so filtering afterwards would return batches made entirely of blocked rows. URL sanitising. The feed intermittently appends stray characters to a URL — a trailing backtick stored as %60 was turning a live Amazon article into a permanent 404. Feed URLs now pass through tools.NormalizeURL before being cached or stored. Parser attribution. Named parsers hardcoded their outlet as the author when their byline selector missed, which suppressed the real byline the generic extractor could find; TechCrunch articles were credited to "TechCrunch" instead of their writer. The placeholders are gone and parseGeneric now runs behind every named parser, filling only fields left empty. Verified against live pages: TechCrunch now resolves the actual byline. Also adds a livecheck-tagged extraction harness for checking parsers against real pages, and logs empty extractions at Warn with the domain so the outlets the extractor cannot handle are visible without enabling debug logging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rfuggi7P8UanYTYuXQkVm3
1 parent f58d78c commit 720ee52

16 files changed

Lines changed: 504 additions & 33 deletions

‎AGENTS.md‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ network or database.
7979
| `CRAWL_MAX_ATTEMPTS` | `5` | Consecutive failures before the retry delay stops doubling |
8080
| `CRAWL_RETRY_BACKOFF` | `15m` | First retry delay after a failure; doubles each time |
8181
| `CRAWL_RETRY_BACKOFF_MAX` | `6h` | Ceiling on that delay |
82+
| `CRAWL_BLOCKED_DOMAINS` | built-in list | Comma-separated outlets to ingest but never crawl; `none` disables blocking |
8283

8384
Bad values fall back to the default rather than stopping the service — a typo
8485
in `LOG_LEVEL` must never prevent a boot.
@@ -115,6 +116,17 @@ known outlets and `parseGeneric` for everything else, so no URL is ever
115116
unhandled. Adding an outlet means adding to `domainParsers` — never adding a
116117
branch that leaves other domains unparsed.
117118

119+
**Named parsers must not substitute placeholder values.** `parseGeneric` runs
120+
behind every named parser and fills only the fields still empty, so a missed
121+
byline is recovered from JSON-LD or meta tags. A hardcoded `"TechCrunch"`
122+
author looks filled, suppresses that recovery, and silently misattributes every
123+
article — which is exactly what happened before this rule existed.
124+
125+
**Blocked outlets are ingested but never crawled.** Some outlets answer every
126+
request with 401 no matter what headers are sent. They are excluded from the
127+
crawl query and from the backlog count, so the backlog stays a meaningful
128+
health signal instead of resting permanently on an uncrawlable residue.
129+
118130
## 6. Domain & architecture
119131

120132
Three goroutines run alongside the HTTP server, all cancelled by the same
@@ -149,11 +161,29 @@ Retry state is in memory, so a restart gives every article a fresh start —
149161
usually what you want, since a deploy is what fixes a bad parser.
150162

151163
**Extraction.** `Scrape` builds a Colly collector rooted at `<html>` and
152-
dispatches to the parser for the host. `parseGeneric` prefers schema.org
164+
dispatches to the parser for the host, then runs `parseGeneric` behind it to
165+
fill any fields the named parser left empty. `parseGeneric` prefers schema.org
153166
JSON-LD (`articleBody`, `headline`, `author`), falling back to picking the
154167
densest block of `<p>` text while excluding nav, header, footer, aside and
155168
figure. A result under `MinBodyLength` is treated as no article at all.
156169

170+
**Ingest hygiene.** Feed URLs are passed through `tools.NormalizeURL` before
171+
they are cached or stored: the feed intermittently appends stray characters
172+
(a trailing backtick arriving as `%60`, for instance), and storing one verbatim
173+
turns a live article into a permanent 404 no parser can rescue.
174+
175+
To check extraction against real pages when adding a parser or chasing a
176+
persistent `items_empty` count, use the live-check tool — it is excluded from
177+
normal runs by a build tag:
178+
179+
```bash
180+
printf '%s\n' "https://example.com/article" > /tmp/urls.txt
181+
URLS_FILE=/tmp/urls.txt go test ./scraper -tags livecheck -run TestLiveExtraction -v
182+
183+
# What does the named parser add over the generic extractor?
184+
COMPARE_URL="https://example.com/article" go test ./scraper -tags livecheck -run TestLiveGenericComparison -v
185+
```
186+
157187
**Auth:** none. The API is read-only and public.
158188

159189
## 7. Ecosystem & related repos
@@ -193,7 +223,10 @@ health signal:
193223
crawler is wedged, not idle. This is the reason a summary is emitted even
194224
when every counter is zero.
195225
- `fail_forbidden` or `fail_http_401` concentrated on one domain means that
196-
outlet is blocking us or is paywalled.
226+
outlet is blocking us or is paywalled. If it is unfixable, add it to
227+
`CRAWL_BLOCKED_DOMAINS` rather than letting it retry forever.
228+
- `no article body extracted` warnings are deduplicated per domain and name the
229+
outlets the extractor cannot handle — the input to the next parser fix.
197230

198231
To investigate a specific article, set `LOG_LEVEL=DEBUG` — every per-item line
199232
carries `news_id`.

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@ a small read-only HTTP API.
1515
Article extraction is domain-aware: known outlets get purpose-built parsers,
1616
and everything else falls through to a generic extractor that reads schema.org
1717
JSON-LD or picks the densest block of paragraph text on the page — so no URL is
18-
ever left unparsed.
18+
ever left unparsed. Outlets behind hard paywalls are ingested for their
19+
headline and symbols but never crawled.
1920

2021
## Role in the Sentiment Scraper ecosystem
2122

@@ -74,6 +75,7 @@ Tests need neither network nor database — the crawler tests run against local
7475
| `CRAWL_MAX_ATTEMPTS` | `5` | Consecutive failures before the retry delay stops doubling |
7576
| `CRAWL_RETRY_BACKOFF` | `15m` | First retry delay after a failure; doubles each time |
7677
| `CRAWL_RETRY_BACKOFF_MAX` | `6h` | Ceiling on that delay |
78+
| `CRAWL_BLOCKED_DOMAINS` | built-in list | Comma-separated outlets to ingest but never crawl; `none` disables blocking |
7779

7880
### Logging
7981

‎background/CheckCrawlers.background.go‎

Lines changed: 29 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package background
33
import (
44
"errors"
55
"log/slog"
6+
"net/url"
67
"time"
78

89
"github.com/aidenappl/SentimentScraperAPI/db"
@@ -34,8 +35,9 @@ func CheckCrawlers() {
3435

3536
now := time.Now()
3637
deferred := retries.Deferred(now)
38+
blocked := scraper.BlockedURLPatterns()
3739

38-
total, err := query.CountNewsNeedingCrawl(db.DB)
40+
total, err := query.CountNewsNeedingCrawl(db.DB, blocked)
3941
if err != nil {
4042
slog.Error("failed to count crawl backlog", "reason", "query", "err", err)
4143
} else {
@@ -44,9 +46,10 @@ func CheckCrawlers() {
4446
logging.Crawl.SetDeferred(len(deferred))
4547

4648
news, err := query.ListNews(db.DB, query.ListNewsRequest{
47-
HasBodyContent: tools.BoolP(false),
48-
Limit: tools.IntP(env.CrawlBatchLimit),
49-
ExcludeIDs: deferred,
49+
HasBodyContent: tools.BoolP(false),
50+
Limit: tools.IntP(env.CrawlBatchLimit),
51+
ExcludeIDs: deferred,
52+
ExcludeURLPatterns: blocked,
5053
})
5154
if err != nil {
5255
slog.Error("failed to list news items for crawling", "reason", "query", "err", err)
@@ -84,7 +87,17 @@ func CheckCrawlers() {
8487
// request error.
8588
if errors.Is(err, scraper.ErrEmptyBody) {
8689
logging.Crawl.IncEmpty()
87-
slog.Debug("no article body extracted", "news_id", id, "url", articleURL)
90+
91+
// Warn, not Debug: this is the one failure mode the fetch path
92+
// cannot report, and it is deduplicated per domain, so it costs
93+
// one line per outlet per window while making it obvious which
94+
// outlets the extractor cannot handle.
95+
slog.Warn("no article body extracted",
96+
"reason", "empty",
97+
"domain", domainOf(articleURL),
98+
"news_id", id,
99+
"url", articleURL,
100+
)
88101
}
89102

90103
continue
@@ -108,3 +121,14 @@ func CheckCrawlers() {
108121
slog.Debug("scraped news item", "news_id", id, "chars", len(article.ArticleBody))
109122
}
110123
}
124+
125+
// domainOf keeps the dedup key low-cardinality: one entry per outlet rather
126+
// than one per article.
127+
func domainOf(raw string) string {
128+
u, err := url.Parse(raw)
129+
if err != nil {
130+
return ""
131+
}
132+
133+
return u.Hostname()
134+
}

‎background/background.go‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import (
77
"github.com/aidenappl/SentimentScraperAPI/query"
88
"github.com/aidenappl/SentimentScraperAPI/scraper"
99
"github.com/aidenappl/SentimentScraperAPI/state"
10+
"github.com/aidenappl/SentimentScraperAPI/tools"
1011
)
1112

1213
func Google() {
@@ -31,6 +32,10 @@ func NewsFilter() {
3132
}
3233

3334
for _, item := range news {
35+
// The feed occasionally appends stray characters to a URL; storing one
36+
// verbatim turns a live article into a permanent 404.
37+
item.Article.URL = tools.NormalizeURL(item.Article.URL)
38+
3439
// Check if the news item already exists.
3540
if _, exists := state.GetFromNewsCache(item.Article.URL); exists {
3641
continue
@@ -43,10 +48,14 @@ func NewsFilter() {
4348
}
4449

4550
// A failed first scrape leaves body and author empty; the item still
46-
// gets inserted so CheckCrawlers can pick it up.
51+
// gets inserted so CheckCrawlers can pick it up. Blocked outlets are
52+
// not attempted at all — they are ingested for their headline and
53+
// symbols, and never crawled.
4754
var body, authors string
48-
if article, err := scraper.Scrape(item.Article.URL); err == nil {
49-
body, authors = article.ArticleBody, article.AuthorName
55+
if !scraper.IsBlocked(item.Article.URL) {
56+
if article, err := scraper.Scrape(item.Article.URL); err == nil {
57+
body, authors = article.ArticleBody, article.AuthorName
58+
}
5059
}
5160

5261
if err := query.InsertNews(db.DB, item, query.InsertNewsRequest{

‎env/env.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ var (
3838

3939
// CrawlRetryBackoffMax caps that delay.
4040
CrawlRetryBackoffMax = getEnvDuration("CRAWL_RETRY_BACKOFF_MAX", 6*time.Hour, time.Second)
41+
42+
// CrawlBlockedDomains is a comma-separated list of outlets that are
43+
// ingested but never crawled. Empty means "use the built-in list"; set it
44+
// to "none" to disable blocking entirely.
45+
CrawlBlockedDomains = getEnv("CRAWL_BLOCKED_DOMAINS", "")
4146
)
4247

4348
func getEnv(key string, fallback string) string {

‎main.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import (
77
"net/http"
88
"os"
99
"os/signal"
10+
"strings"
1011
"sync"
1112
"syscall"
1213
"time"
@@ -17,6 +18,7 @@ import (
1718
"github.com/aidenappl/SentimentScraperAPI/logging"
1819
"github.com/aidenappl/SentimentScraperAPI/middleware"
1920
"github.com/aidenappl/SentimentScraperAPI/routers"
21+
"github.com/aidenappl/SentimentScraperAPI/scraper"
2022
"github.com/aidenappl/SentimentScraperAPI/sentiment"
2123
"github.com/aidenappl/SentimentScraperAPI/state"
2224
"github.com/gorilla/mux"
@@ -26,6 +28,12 @@ import (
2628
func main() {
2729
logging.Init(env.LogLevel, env.LogSummaryInterval)
2830

31+
if env.CrawlBlockedDomains == "none" {
32+
scraper.SetBlockedDomains(nil)
33+
} else if env.CrawlBlockedDomains != "" {
34+
scraper.SetBlockedDomains(strings.Split(env.CrawlBlockedDomains, ","))
35+
}
36+
2937
// Ping DB
3038
if err := db.PingDB(); err != nil {
3139
logging.Fatal("failed to connect to the database", "err", err)

‎query/CountNewsNeedingCrawl.query.go‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,25 @@ import (
77
"github.com/aidenappl/SentimentScraperAPI/db"
88
)
99

10-
// CountNewsNeedingCrawl returns how many articles still have no body content.
10+
// CountNewsNeedingCrawl returns how many articles still have no body content,
11+
// excluding any whose URL matches one of excludeURLPatterns.
1112
//
1213
// This is the true backlog. The crawl batch is capped, so counting the rows a
1314
// batch returned would just report the cap back and hide a growing queue.
14-
func CountNewsNeedingCrawl(dbc db.Queryable) (int, error) {
15+
// Blocked outlets are excluded because they can never be crawled — counting
16+
// them would hold the backlog permanently above zero and destroy its value as
17+
// a health signal.
18+
func CountNewsNeedingCrawl(dbc db.Queryable, excludeURLPatterns []string) (int, error) {
1519
psql := sq.StatementBuilder.PlaceholderFormat(sq.Dollar)
1620

1721
q := psql.Select("COUNT(*)").
1822
From("website.news n").
1923
Where(sq.Or{sq.Eq{"n.body_content": nil}, sq.Eq{"n.body_content": ""}})
2024

25+
for _, pattern := range excludeURLPatterns {
26+
q = q.Where(sq.NotLike{"n.article_url": pattern})
27+
}
28+
2129
query, args, err := q.ToSql()
2230
if err != nil {
2331
return 0, fmt.Errorf("error building SQL query: %w", err)

‎query/ListNews.query.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,10 @@ type ListNewsRequest struct {
2222
// articles it can actually attempt rather than the same failing rows.
2323
ExcludeIDs []int `json:"-"`
2424

25+
// ExcludeURLPatterns drops rows whose article_url matches any SQL LIKE
26+
// pattern. The crawler uses it to keep blocked outlets out of the batch.
27+
ExcludeURLPatterns []string `json:"-"`
28+
2529
// Selectors
2630
ID *int `json:"id"`
2731
}
@@ -122,6 +126,10 @@ func ListNews(dbc db.Queryable, req ListNewsRequest) ([]structs.News, error) {
122126
q = q.Where(sq.NotEq{"n.id": req.ExcludeIDs})
123127
}
124128

129+
for _, pattern := range req.ExcludeURLPatterns {
130+
q = q.Where(sq.NotLike{"n.article_url": pattern})
131+
}
132+
125133
query, args, err := q.ToSql()
126134
if err != nil {
127135
return nil, fmt.Errorf("error building SQL query: %w", err)

‎scraper/Web.scraper.go‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -58,13 +58,13 @@ func Scrape(url string) (*ScrapedArticle, error) {
5858
parse, named := parserFor(e.Request.URL.Hostname())
5959
parse(e, article)
6060

61-
// A named parser that comes back empty means the outlet changed its
62-
// markup; fall back to the generic extractor rather than losing the
63-
// article entirely.
64-
if !named || len(article.ArticleBody) >= MinBodyLength {
65-
return
61+
// Always run the generic extractor behind a named parser. It only
62+
// fills fields left empty, so the named parser keeps what it got right
63+
// while its gaps — a byline selector that has drifted, a body the
64+
// outlet re-templated — are covered rather than lost.
65+
if named {
66+
parseGeneric(e, article)
6667
}
67-
parseGeneric(e, article)
6868
})
6969

7070
c.OnRequest(func(r *colly.Request) {

‎scraper/blocklist.scraper.go‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
package scraper
2+
3+
import (
4+
"net/url"
5+
"strings"
6+
)
7+
8+
// BlockedDomains are outlets that cannot be crawled at all — hard paywalls
9+
// that answer every request with 401 regardless of headers.
10+
//
11+
// Articles from these outlets are still ingested, so their headline, source
12+
// and symbols remain available; they are simply never queued for crawling.
13+
// Retrying them forever would keep the backlog permanently non-zero and make
14+
// it useless as a health signal.
15+
var BlockedDomains = []string{
16+
"reuters.com",
17+
"barrons.com",
18+
"wsj.com",
19+
}
20+
21+
// SetBlockedDomains replaces the blocklist. An empty list clears it.
22+
func SetBlockedDomains(domains []string) {
23+
cleaned := make([]string, 0, len(domains))
24+
for _, d := range domains {
25+
if d = strings.ToLower(strings.TrimSpace(d)); d != "" {
26+
cleaned = append(cleaned, strings.TrimPrefix(d, "www."))
27+
}
28+
}
29+
BlockedDomains = cleaned
30+
}
31+
32+
// IsBlocked reports whether a URL belongs to a blocked outlet.
33+
func IsBlocked(rawURL string) bool {
34+
u, err := url.Parse(strings.TrimSpace(rawURL))
35+
if err != nil {
36+
return false
37+
}
38+
39+
host := strings.ToLower(u.Hostname())
40+
host = strings.TrimPrefix(host, "www.")
41+
42+
for _, domain := range BlockedDomains {
43+
if host == domain || strings.HasSuffix(host, "."+domain) {
44+
return true
45+
}
46+
}
47+
48+
return false
49+
}
50+
51+
// BlockedURLPatterns renders the blocklist as SQL LIKE patterns for excluding
52+
// blocked outlets from a query.
53+
//
54+
// The exclusion has to happen in SQL rather than after the fact: the crawl
55+
// listing is ordered newest-first and capped, so filtering afterwards would
56+
// hand back a batch made entirely of blocked rows and starve everything else.
57+
func BlockedURLPatterns() []string {
58+
patterns := make([]string, 0, len(BlockedDomains)*2)
59+
for _, domain := range BlockedDomains {
60+
// Matches https://domain/... and https://any.sub.domain/...
61+
patterns = append(patterns, "%//"+domain+"/%", "%."+domain+"/%")
62+
}
63+
64+
return patterns
65+
}

0 commit comments

Comments
 (0)