upstream-frappe-crm/crm/domain_enrichment/README.md

# Domain Enrichment

Auto-enrich a CRM Lead, Deal or Organization from its **website** — no paid data
providers. Given a domain, the engine crawls the public site, runs **admin-tunable,
rule-based** extractors (company name, description, logo, industry,
emails, phones, social links), records everything with full provenance, and fills
the mapped CRM fields according to configurable write policies.

The guiding principle: **config is data, the engine is code.** Everything a
sales-ops admin would tune — keywords, social patterns,
field mappings, crawl limits, enabled sources, the SSRF allow/block lists — lives
in **doctypes you edit in the desk**, never in Python. The engine knows no keywords
itself; it loads rules and executes them.

---

## What & why

- **Domain-based.** The only input is a website URL. No API keys, no per-record
  cost, no third-party enrichment vendor.
- **Rule-based / public-data.** Industry and social profiles
  are all matched by **rules an admin owns** (`CRM Enrichment Rule`).
  The mechanics (crawling, HTML parsing, the favicon scorer, JSON-LD parsing,
  readability diagnosis, email/phone regexes) are pure code because they are
  algorithms, not tunable knowledge.
- **Provenance preserved.** Every extracted value records `{value, source, method}`
  — which page it came from and how it was found — so a reviewer can judge how much
  to trust it. The full provenance JSON is stored on each run.
- **Self-contained.** The only outbound traffic is fetching the target site itself —
  no third-party APIs.
- **Framework-native.** Outbound HTTP uses `frappe.utils.get_request_session`;
  background work uses `frappe.enqueue`; progress streams over
  `frappe.publish_realtime`.

---

## Architecture

```
DESK ADMIN (data)                         ENGINE (code, rule-agnostic)        RESULT (data)
 CRM Enrichment Settings (Single) ─┐       crm/domain_enrichment/
 CRM Enrichment Rule (+Pattern) ───┼─►get_config()──► config.py              CRM Enrichment Run
 CRM Enrichment Field Mapping ─────┘      http.py (session+SSRF+cap)           (Dynamic Link to
                                          crawler.py (BFS; caps from cfg)       Lead/Deal/Org)
        │                                 extractors.py (rule executors)              ▲
        ▼                                 pipeline.py → EnrichmentResult              │
   bench migrate syncs                    api.py (@whitelist) / tasks.py        mapper.py writes
   native fields onto                     (enqueue + realtime)                  mapped fields onto
   Lead/Deal/Organization                                                      the record (policy)
```

### Module map (`crm/domain_enrichment/`)

| File | Responsibility |
|---|---|
| `config.py` | `get_config()` → `EnrichmentConfig` (settings + `Rule`/`Mapping` objects), built on demand (not cached). `get_settings()` / `auto_enrich_enabled_for()` are the cheap Settings-only reads for hot paths. |
| `http.py` | `fetch(url, cfg)` on the framework session; the **SSRF guard** (`validate_url`); byte cap; HTML-only filter; never-raise `(status, html, error, final_url)` contract. |
| `crawler.py` | Same-domain, depth-limited BFS. Caps / link-priority order / skip patterns from config. |
| `extractors.py` | **Generic rule executor** (`apply_keyword_rules`) + the pure mechanics. No literal keyword tables. |
| `result.py` | `EnrichmentResult` and its `{value, source, method}` provenance schema. Pure dataclasses, no framework import. |
| `pipeline.py` | `run(website, cfg, progress)` orchestrates crawl → extract → assemble `EnrichmentResult`. Never writes to the DB. |
| `mapper.py` | `apply_to_document(doc, result, cfg)` — the **single** result→CRM-field authority, driven by Field Mapping records + write policies. |
| `tasks.py` | `run_enrichment` (enqueued worker) + `write_run` (the **single** run-history writer). Streams realtime progress; never raises. |
| `api.py` | Whitelisted `enrich` (enqueue) + `retry` (re-run from a Run). Type-annotated; permission- and rate-limited. |
| `cross_record.py` | `copy_enrichment_from_organization` — the one link-time Org→Lead/Deal copy. |
| `install.py` | Idempotent seeder: translates the original constant tables into Rule + Field Mapping records. |

---

## Configuration guide

Three doctypes hold all the tunable knowledge. `get_config()` reads them on demand
(no cache), so an edit takes effect on the very next enrichment without a restart.

### 1. CRM Enrichment Settings (Single)

| Field | Meaning |
|---|---|
| `enabled` | Master on/off for the whole feature, across Lead, Deal and Organization. |
| `auto_enrich` | When `1`, a newly-created CRM **Lead, Deal or Organization** that has a website is enriched automatically in a background job (`after_insert`). A new Deal with a new Organization enriches both (each crawls and writes its own fields). Default `0` — enrichment is otherwise manual via the Enrich button. |
| `max_pages` / `max_depth` | Crawl breadth/depth caps (BFS). |
| `request_timeout` | Per-request timeout (seconds). |
| `max_download_bytes` | Hard cap on bytes read per page (streamed). |
| `retry_count` | Transient-error retries on the session. |
| `user_agent` | Crawler User-Agent header. |
| `allowed_domains` / `blocked_domains` (child) | SSRF allow/block lists (subdomain-aware). A blocked host is always rejected; if an allow list exists, only listed hosts pass. |
| `link_priority_order` (child) | `keyword` + `weight`: a crawl-ordering hint — links whose URL/anchor contain these are fetched first (higher weight = sooner). Defaults are seeded on install. |
| `skip_patterns` (child) | URL substrings/regexes to never crawl. |

If the Single has never been saved, the engine falls back to sane defaults
(`config.DEFAULT_SETTINGS`).

### 2. CRM Enrichment Rule (+ CRM Enrichment Rule Pattern)

The heart of the knowledge base. One record = one rule.

| Field | Meaning |
|---|---|
| `rule_name` | Unique name (the record's id). |
| `rule_type` | `Industry` / `Social`. |
| `enabled` | Disabled rules are skipped by the config builder. |
| `target_value` | The label a match emits (tech name, network). Ignored for `Industry`. |
| `industry` | (Industry rules only) the linked `CRM Industry` a match classifies to. |
| `weight` | Multiplier on the hit count (industry scoring). |
| `match_scope` | `Headline` / `Full Text` / `HTML` / `Headers` / `URL` — which slice of the page the rule matches. |
| `patterns` (child) | One or more `pattern` strings; `is_regex` toggles substring vs. regex. Substrings are matched case-insensitively. |

### 3. CRM Enrichment Field Mapping

The single source of result→field truth. One record = one field written.

| Field | Meaning |
|---|---|
| `enabled` | Disabled mappings are skipped. |
| `source_key` | One of the 13 frozen result keys (below). |
| `target_doctype` | `CRM Lead` / `CRM Deal` / `CRM Organization`. |
| `target_fieldname` | The field on that doctype to fill (validated to exist). |
| `write_policy` | `Fill if empty` / `Always refresh` / `Override defaults` (below). |
| `default_values` | (Override defaults) newline-separated values treated as "unset". |
| `create_missing_link` | (Link fields) auto-create the linked master if missing (e.g. a new `CRM Industry`). |

**The 13 `source_key`s:** `company_name`, `description`, `logo`, `industry`,
`primary_email`, `primary_phone`, `secondary_phone`, `linkedin`,
`twitter`, `github`, `facebook`, `instagram`, `youtube`.

**Write policies:**
- **Fill if empty** — write only when the target field is currently empty.
- **Always refresh** — overwrite (or clear) with the fresh value, even if the field
  had a previous value. Used for fully engine-derived fields that should always
  reflect the latest crawl.
- **Override defaults** — treat the listed `default_values` (plus empty) as "unset"
  and fill over them, but respect any real user value. Useful for a field that ships
  with a meaningless placeholder default that enrichment should replace.

### Worked examples

**Add a new industry.** Create a `CRM Industry` (or let `create_missing_link` do it),
then a `CRM Enrichment Rule`: `rule_type = Industry`, `industry = <your industry>`,
`match_scope = Headline`, and add patterns (e.g. `biotech`, `genomics`). Save — the
classifier picks it up on the next run.

**Add a social network.** New `CRM Enrichment Rule`: `rule_type = Social`,
`target_value = "mastodon"`, `match_scope = HTML`, and a regex pattern
(`is_regex = 1`) such as `@[A-Za-z0-9_]+@mastodon\.social`.

**Remap a result field / change a write policy.** Edit the relevant
`CRM Enrichment Field Mapping` — change `target_fieldname`, or flip `write_policy`
from `Fill if empty` to `Always refresh`. To stop a field being written, untick
`enabled` on its mapping.

**Tune crawl limits / SSRF.** Raise `max_pages`/`max_depth` for deeper crawls; add a
`skip_patterns` row (e.g. `/blog/`) to skip noisy sections; add a `blocked_domains`
row to hard-block a host, or `allowed_domains` rows to restrict crawling to an
allow-list only.

---

## How a run works (end to end)

1. **Trigger** — the SPA `EnrichFromWebsite` button (or the desk form button) calls
   the whitelisted `api.enrich(reference_doctype, reference_name)`.
2. **Validate + enqueue** — `enrich` checks the doctype is enabled, the user has
   `write` permission, and the record has a `website`, then `frappe.enqueue`s
   `tasks.run_enrichment` on the `long` queue with a per-doc `job_id` and
   `deduplicate=True` (a second click while one is in-flight is a no-op).
3. **Crawl** — `pipeline.run` builds the session, runs the BFS crawler (SSRF-guarded
   `http.fetch` per page), and emits progress callbacks.
4. **Rule execution** — the extractors run the config rules: industry classification,
   social profiles, plus the mechanics (company info, emails, phones).
5. **Map** — `mapper.apply_to_document` writes the mapped fields onto the origin doc
   per each mapping's write policy; the doc is saved (normal, permission-respecting).
6. **Run record** — `tasks.write_run` persists exactly one `CRM Enrichment Run` with
   the summary fields + full `raw_json` provenance.
7. **Realtime** — progress and the terminal result stream to the triggering user over
   the `domain_enrichment_progress` event.

Progress event payload (mirrors `crm/api/whatsapp.py`):

```json
{"reference_doctype": "...", "reference_name": "...", "step": 3, "total": 9,
 "message": "...", "status": "running|completed|error", "payload": {}}
```

The terminal `completed` event's `payload` carries `{filled_fields, notes, **result.flat()}`.

### Run lifecycle (state machine)

```
        enqueue (dedupe)      worker start          all steps ok
 [Queued] ───────────► [Running] ─────────────────────────────► [Completed]
                           │
                           │  exception / blocked / unreachable
                           ▼
                       [Failed]   (reason in notes; frappe.log_error)
```

`tasks.run_enrichment` never raises to the worker: on any exception the Run is
written as `Failed` (with the traceback truncated into `notes`), the error is logged
via `frappe.log_error`, and an `error` event is published.

---

## Cross-record behavior

There is exactly **one** cross-record copy:
`cross_record.copy_enrichment_from_organization(target_doc)`. When a **Lead** or
**Deal** links an already-enriched **Organization**, the Organization's stored
enriched fields are copied onto the **empty** fields of the Lead/Deal
(fill-empty semantics, even for mappings normally configured "Always refresh"), so
user-entered data is never touched.

- **Lead seam:** `CRMLead.create_organization` (existing-org branch).
- **Deal seam:** `CRMDeal.validate` when `organization` is set/changed.
- An Organization is "enriched" if it has a **completed** `CRM Enrichment Run`, or
  (fallback) any populated native enriched field.

**No background fan-out.** The copy is synchronous, in-request, Organization →
Lead/Deal only. There is no job, no propagation to other linked records, and no
write back to the Organization. The field-writing reuses `mapper.apply_to_document`
(the single authority) — no second copy implementation exists.

---

## SSRF Guard

- **SSRF guard (`http.validate_url`).** Mandatory and not provided by the framework.
  It rejects non-`http(s)` schemes, resolves the hostname, and rejects any URL that
  resolves to a loopback / private / link-local / reserved / multicast / unspecified
  address — including the cloud-metadata endpoint `169.254.169.254`. It honors the
  `blocked_domains` / `allowed_domains` lists and **re-validates after every
  redirect** (redirects are followed manually). The private-network rejection is
  **unconditional** — there is deliberately no setting to disable it (a crawler that
  fetches user-supplied URLs must never be turnable into an internal-network probe).
- **DNS-rebinding TOCTOU — closed by IP pinning.** The guard validates the resolved
  addresses and the connection is then made to one of those exact addresses
  (`http._pinned_get`): the URL netloc is rewritten to the validated IP while the
  Host header, TLS SNI and certificate verification stay on the original hostname
  (urllib3 `server_hostname`). A host that alternates DNS answers (short TTL) can no
  longer pass validation with a public IP and serve the request from a private one.
  Each redirect hop is re-validated and re-pinned.
- **Permissions.** `api.enrich` enforces the Settings allow-list and
  `doc.check_permission("write")` — a user who cannot edit the record cannot enrich
  it. The worker save is a normal permission-respecting save. `create_missing_link`
  auto-creation is permission-respecting too: `_ensure_link_target` only creates a
  master (`CRM Industry`) when the enriching user has create rights on it, so a
  scraped value can never escalate into an insert the user couldn't do by hand. The
  only `ignore_permissions` write is the engine-owned `CRM Enrichment Run` audit log
  — never a cross-record write to data the user can't edit.
- **Rate limiting.** Every entry point that triggers a fetch — `enrich` and
  `retry` — is decorated with `rate_limit(limit=ENRICH_RATE_LIMIT, seconds=60)`
  (per user, per route). Both only enqueue a per-doc-deduplicated job.

---

## Extending in code

- **Add an extractor mechanic** (e.g. a new heuristic): add a pure function to
  `extractors.py`, call it from `pipeline.run`, and surface its output on
  `EnrichmentResult` (`result.py`). Keep tunable knowledge in rules, not constants.
- **Add a new `rule_type`:** add the option to the `CRM Enrichment Rule` Select, add
  a generic executor (or reuse `apply_keyword_rules`) in
  `extractors.py`, and wire it into `pipeline.run` via `cfg.rules("<Type>")`. Keep the
  rule vocabulary small (substring/regex + scope + weight) — do **not** grow a DSL.
- **Add a new `source_key`:** extend `mapper.get_value_for_source_key`, add the option
  to the Field Mapping Select, and (if it needs new data) populate it on
  `EnrichmentResult`.

---

## Limitations & future

- **JS-rendered sites.** Enrichment reads server-delivered HTML only (`requests` +
  BeautifulSoup), so a fully client-rendered site yields only `<head>` metadata; the
  run records the "main content is JavaScript-rendered" note (`READABILITY_MESSAGES["empty"]`)
  and fields may be blank. A headless-Chromium render fallback was prototyped but
  **intentionally removed** — the per-worker Chromium memory footprint isn't worth it
  for a fallback. (The implementation is preserved locally in the gitignored
  `CHROMIUM_FALLBACK.local.md` if it ever needs to be reintroduced.)
- **Logo vs image.** The `logo` is the company's **link icon** (`extract_logo`): the
  best declared `<link rel=icon>` — scalable SVG > largest raster / apple-touch — with
  `/favicon.ico` as the fallback. It's the crisp, square brand mark suited to an avatar.
  The larger social/brand image (`extract_image`: JSON-LD `Organization.logo` →
  `og:image`/`twitter:image`) is captured separately as `image` in the run JSON — it is
  **not** used as the logo (`og:image` is usually a wide banner) and isn't mapped to a
  record field by default.
- **Scheduled re-enrichment — _future feature, not implemented._** Triggering is
  either manual (Enrich button) or automatic on Lead/Deal/Organization creation (`auto_enrich`);
  there is no *scheduled* re-enrichment of existing records. A future version could enqueue periodic re-enrichment of
  stale records (e.g. via a scheduled job that re-runs `tasks.run_enrichment` for
  records whose last `CRM Enrichment Run` is older than N days).
- **TLS fingerprinting.** The fetcher uses `requests`, so its TLS ClientHello
  fingerprints (JA3/JA4) as a non-browser client. Bot walls that fingerprint TLS
  (Cloudflare, Akamai, DataDome) can still block it even though `http.build_session`
  sends browser-like headers. Real evasion needs a TLS-impersonating client
  (`curl_cffi`) or a headless browser — deliberately out of scope here.

---

## Tests

Framework-native tests live in `crm/domain_enrichment/tests/` and are discoverable by
`bench run-tests --app crm`:

- **Pure unit (`frappe.tests.UnitTestCase`, no DB, offline fixtures):**
  `test_crawler.py`, `test_extractors.py`, `test_pipeline.py`, `test_ssrf.py`. The
  rule executors are exercised with in-memory `Rule` objects (`tests/fixtures.py`);
  the SSRF guard mocks `socket.getaddrinfo` so DNS is deterministic and offline.
- **Integration (`frappe.tests.IntegrationTestCase`, real test DB):**
  `test_integration.py` — seeder idempotency, mapper write-policies, the Run writer,
  API permission/allow-list enforcement, and the link-time Org→Lead/Deal copy. The
  crawl is monkeypatched to a canned `EnrichmentResult`, so the suite is network-free.

```bash
bench --site <site> run-tests --app crm --module crm.domain_enrichment.tests.test_extractors
bench --site <site> run-tests --app crm --module crm.domain_enrichment.tests.test_integration
```