Auto-enrich a CRM Lead, Deal or Organization from its website — no paid data providers. Given a domain, the engine crawls the public site, runs admin-tunable, rule-based extractors (company name, description, logo, industry, emails, phones, social links), records everything with full provenance, and fills the mapped CRM fields according to configurable write policies.

The guiding principle: config is data, the engine is code. Everything a sales-ops admin would tune — keywords, social patterns, field mappings, crawl limits, enabled sources, the SSRF allow/block lists — lives in doctypes you edit in the desk, never in Python. The engine knows no keywords itself; it loads rules and executes them.

What & why

Domain-based. The only input is a website URL. No API keys, no per-record cost, no third-party enrichment vendor.
Rule-based / public-data. Industry and social profiles are all matched by rules an admin owns (CRM Enrichment Rule). The mechanics (crawling, HTML parsing, the favicon scorer, JSON-LD parsing, readability diagnosis, email/phone regexes) are pure code because they are algorithms, not tunable knowledge.
Provenance preserved. Every extracted value records {value, source, method} — which page it came from and how it was found — so a reviewer can judge how much to trust it. The full provenance JSON is stored on each run.
Self-contained. The only outbound traffic is fetching the target site itself — no third-party APIs.
Framework-native. Outbound HTTP uses frappe.utils.get_request_session; background work uses frappe.enqueue; progress streams over frappe.publish_realtime.

Architecture

DESK ADMIN (data)                         ENGINE (code, rule-agnostic)        RESULT (data)
 CRM Enrichment Settings (Single) ─┐       crm/domain_enrichment/
 CRM Enrichment Rule (+Pattern) ───┼─►get_config()──► config.py              CRM Enrichment Run
 CRM Enrichment Field Mapping ─────┘      http.py (session+SSRF+cap)           (Dynamic Link to
                                          crawler.py (BFS; caps from cfg)       Lead/Deal/Org)
        │                                 extractors.py (rule executors)              ▲
        ▼                                 pipeline.py → EnrichmentResult              │
   bench migrate syncs                    api.py (@whitelist) / tasks.py        mapper.py writes
   native fields onto                     (enqueue + realtime)                  mapped fields onto
   Lead/Deal/Organization                                                      the record (policy)

Module map (`crm/domain_enrichment/`)

File	Responsibility
`config.py`	`get_config()` → `EnrichmentConfig` (settings + `Rule`/`Mapping` objects), built on demand (not cached). `get_settings()` / `auto_enrich_enabled_for()` are the cheap Settings-only reads for hot paths.
`http.py`	`fetch(url, cfg)` on the framework session; the SSRF guard (`validate_url`); byte cap; HTML-only filter; never-raise `(status, html, error, final_url)` contract.
`crawler.py`	Same-domain, depth-limited BFS. Caps / link-priority order / skip patterns from config.
`extractors.py`	Generic rule executor (`apply_keyword_rules`) + the pure mechanics. No literal keyword tables.
`result.py`	`EnrichmentResult` and its `{value, source, method}` provenance schema. Pure dataclasses, no framework import.
`pipeline.py`	`run(website, cfg, progress)` orchestrates crawl → extract → assemble `EnrichmentResult`. Never writes to the DB.
`mapper.py`	`apply_to_document(doc, result, cfg)` — the single result→CRM-field authority, driven by Field Mapping records + write policies.
`tasks.py`	`run_enrichment` (enqueued worker) + `write_run` (the single run-history writer). Streams realtime progress; never raises.
`api.py`	Whitelisted `enrich` (enqueue) + `retry` (re-run from a Run). Type-annotated; permission- and rate-limited.
`cross_record.py`	`copy_enrichment_from_organization` — the one link-time Org→Lead/Deal copy.
`install.py`	Idempotent seeder: translates the original constant tables into Rule + Field Mapping records.

Configuration guide

Three doctypes hold all the tunable knowledge. get_config() reads them on demand (no cache), so an edit takes effect on the very next enrichment without a restart.

1. CRM Enrichment Settings (Single)

Field	Meaning
`enabled`	Master on/off for the whole feature, across Lead, Deal and Organization.
`auto_enrich`	When `1`, a newly-created CRM Lead, Deal or Organization that has a website is enriched automatically in a background job (`after_insert`). A new Deal with a new Organization enriches both (each crawls and writes its own fields). Default `0` — enrichment is otherwise manual via the Enrich button.
`max_pages` / `max_depth`	Crawl breadth/depth caps (BFS).
`request_timeout`	Per-request timeout (seconds).
`max_download_bytes`	Hard cap on bytes read per page (streamed).
`retry_count`	Transient-error retries on the session.
`user_agent`	Crawler User-Agent header.
`allowed_domains` / `blocked_domains` (child)	SSRF allow/block lists (subdomain-aware). A blocked host is always rejected; if an allow list exists, only listed hosts pass.
`link_priority_order` (child)	`keyword` + `weight`: a crawl-ordering hint — links whose URL/anchor contain these are fetched first (higher weight = sooner). Defaults are seeded on install.
`skip_patterns` (child)	URL substrings/regexes to never crawl.

If the Single has never been saved, the engine falls back to sane defaults (config.DEFAULT_SETTINGS).

2. CRM Enrichment Rule (+ CRM Enrichment Rule Pattern)

The heart of the knowledge base. One record = one rule.

Field	Meaning
`rule_name`	Unique name (the record's id).
`rule_type`	`Industry` / `Social`.
`enabled`	Disabled rules are skipped by the config builder.
`target_value`	The label a match emits (tech name, network). Ignored for `Industry`.
`industry`	(Industry rules only) the linked `CRM Industry` a match classifies to.
`weight`	Multiplier on the hit count (industry scoring).
`match_scope`	`Headline` / `Full Text` / `HTML` / `Headers` / `URL` — which slice of the page the rule matches.
`patterns` (child)	One or more `pattern` strings; `is_regex` toggles substring vs. regex. Substrings are matched case-insensitively.

3. CRM Enrichment Field Mapping

The single source of result→field truth. One record = one field written.

Field	Meaning
`enabled`	Disabled mappings are skipped.
`source_key`	One of the 13 frozen result keys (below).
`target_doctype`	`CRM Lead` / `CRM Deal` / `CRM Organization`.
`target_fieldname`	The field on that doctype to fill (validated to exist).
`write_policy`	`Fill if empty` / `Always refresh` / `Override defaults` (below).
`default_values`	(Override defaults) newline-separated values treated as "unset".
`create_missing_link`	(Link fields) auto-create the linked master if missing (e.g. a new `CRM Industry`).

The 13 source_keys: company_name, description, logo, industry, primary_email, primary_phone, secondary_phone, linkedin, twitter, github, facebook, instagram, youtube.

Write policies:

Fill if empty — write only when the target field is currently empty.
Always refresh — overwrite (or clear) with the fresh value, even if the field had a previous value. Used for fully engine-derived fields that should always reflect the latest crawl.
Override defaults — treat the listed default_values (plus empty) as "unset" and fill over them, but respect any real user value. Useful for a field that ships with a meaningless placeholder default that enrichment should replace.

Worked examples

Add a new industry. Create a CRM Industry (or let create_missing_link do it), then a CRM Enrichment Rule: rule_type = Industry, industry = <your industry>, match_scope = Headline, and add patterns (e.g. biotech, genomics). Save — the classifier picks it up on the next run.

Add a social network. New CRM Enrichment Rule: rule_type = Social, target_value = "mastodon", match_scope = HTML, and a regex pattern (is_regex = 1) such as @[A-Za-z0-9_]+@mastodon\.social.

Remap a result field / change a write policy. Edit the relevant CRM Enrichment Field Mapping — change target_fieldname, or flip write_policy from Fill if empty to Always refresh. To stop a field being written, untick enabled on its mapping.

Tune crawl limits / SSRF. Raise max_pages/max_depth for deeper crawls; add a skip_patterns row (e.g. /blog/) to skip noisy sections; add a blocked_domains row to hard-block a host, or allowed_domains rows to restrict crawling to an allow-list only.

How a run works (end to end)

Trigger — the SPA EnrichFromWebsite button (or the desk form button) calls the whitelisted api.enrich(reference_doctype, reference_name).
Validate + enqueue — enrich checks the doctype is enabled, the user has write permission, and the record has a website, then frappe.enqueues tasks.run_enrichment on the long queue with a per-doc job_id and deduplicate=True (a second click while one is in-flight is a no-op).
Crawl — pipeline.run builds the session, runs the BFS crawler (SSRF-guarded http.fetch per page), and emits progress callbacks.
Rule execution — the extractors run the config rules: industry classification, social profiles, plus the mechanics (company info, emails, phones).
Map — mapper.apply_to_document writes the mapped fields onto the origin doc per each mapping's write policy; the doc is saved (normal, permission-respecting).
Run record — tasks.write_run persists exactly one CRM Enrichment Run with the summary fields + full raw_json provenance.
Realtime — progress and the terminal result stream to the triggering user over the domain_enrichment_progress event.

Progress event payload (mirrors crm/api/whatsapp.py):

{"reference_doctype": "...", "reference_name": "...", "step": 3, "total": 9,
 "message": "...", "status": "running|completed|error", "payload": {}}

The terminal completed event's payload carries {filled_fields, notes, **result.flat()}.

Run lifecycle (state machine)

        enqueue (dedupe)      worker start          all steps ok
 [Queued] ───────────► [Running] ─────────────────────────────► [Completed]
                           │
                           │  exception / blocked / unreachable
                           ▼
                       [Failed]   (reason in notes; frappe.log_error)

tasks.run_enrichment never raises to the worker: on any exception the Run is written as Failed (with the traceback truncated into notes), the error is logged via frappe.log_error, and an error event is published.

Cross-record behavior

There is exactly one cross-record copy: cross_record.copy_enrichment_from_organization(target_doc). When a Lead or Deal links an already-enriched Organization, the Organization's stored enriched fields are copied onto the empty fields of the Lead/Deal (fill-empty semantics, even for mappings normally configured "Always refresh"), so user-entered data is never touched.

Lead seam: CRMLead.create_organization (existing-org branch).
Deal seam: CRMDeal.validate when organization is set/changed.
An Organization is "enriched" if it has a completed CRM Enrichment Run, or (fallback) any populated native enriched field.

No background fan-out. The copy is synchronous, in-request, Organization → Lead/Deal only. There is no job, no propagation to other linked records, and no write back to the Organization. The field-writing reuses mapper.apply_to_document (the single authority) — no second copy implementation exists.

SSRF Guard

SSRF guard (http.validate_url). Mandatory and not provided by the framework. It rejects non-http(s) schemes, resolves the hostname, and rejects any URL that resolves to a loopback / private / link-local / reserved / multicast / unspecified address — including the cloud-metadata endpoint 169.254.169.254. It honors the blocked_domains / allowed_domains lists and re-validates after every redirect (redirects are followed manually). The private-network rejection is unconditional — there is deliberately no setting to disable it (a crawler that fetches user-supplied URLs must never be turnable into an internal-network probe).
DNS-rebinding TOCTOU — closed by IP pinning. The guard validates the resolved addresses and the connection is then made to one of those exact addresses (http._pinned_get): the URL netloc is rewritten to the validated IP while the Host header, TLS SNI and certificate verification stay on the original hostname (urllib3 server_hostname). A host that alternates DNS answers (short TTL) can no longer pass validation with a public IP and serve the request from a private one. Each redirect hop is re-validated and re-pinned.
Permissions. api.enrich enforces the Settings allow-list and doc.check_permission("write") — a user who cannot edit the record cannot enrich it. The worker save is a normal permission-respecting save. create_missing_link auto-creation is permission-respecting too: _ensure_link_target only creates a master (CRM Industry) when the enriching user has create rights on it, so a scraped value can never escalate into an insert the user couldn't do by hand. The only ignore_permissions write is the engine-owned CRM Enrichment Run audit log — never a cross-record write to data the user can't edit.
Rate limiting. Every entry point that triggers a fetch — enrich and retry — is decorated with rate_limit(limit=ENRICH_RATE_LIMIT, seconds=60) (per user, per route). Both only enqueue a per-doc-deduplicated job.

Extending in code

Add an extractor mechanic (e.g. a new heuristic): add a pure function to extractors.py, call it from pipeline.run, and surface its output on EnrichmentResult (result.py). Keep tunable knowledge in rules, not constants.
Add a new rule_type: add the option to the CRM Enrichment Rule Select, add a generic executor (or reuse apply_keyword_rules) in extractors.py, and wire it into pipeline.run via cfg.rules("<Type>"). Keep the rule vocabulary small (substring/regex + scope + weight) — do not grow a DSL.
Add a new source_key: extend mapper.get_value_for_source_key, add the option to the Field Mapping Select, and (if it needs new data) populate it on EnrichmentResult.

Limitations & future

JS-rendered sites. Enrichment reads server-delivered HTML only (requests + BeautifulSoup), so a fully client-rendered site yields only <head> metadata; the run records the "main content is JavaScript-rendered" note (READABILITY_MESSAGES["empty"]) and fields may be blank. A headless-Chromium render fallback was prototyped but intentionally removed — the per-worker Chromium memory footprint isn't worth it for a fallback. (The implementation is preserved locally in the gitignored CHROMIUM_FALLBACK.local.md if it ever needs to be reintroduced.)
Logo vs image. The logo is the company's link icon (extract_logo): the best declared <link rel=icon> — scalable SVG > largest raster / apple-touch — with /favicon.ico as the fallback. It's the crisp, square brand mark suited to an avatar. The larger social/brand image (extract_image: JSON-LD Organization.logo → og:image/twitter:image) is captured separately as image in the run JSON — it is not used as the logo (og:image is usually a wide banner) and isn't mapped to a record field by default.
Scheduled re-enrichment — future feature, not implemented. Triggering is either manual (Enrich button) or automatic on Lead/Deal/Organization creation (auto_enrich); there is no scheduled re-enrichment of existing records. A future version could enqueue periodic re-enrichment of stale records (e.g. via a scheduled job that re-runs tasks.run_enrichment for records whose last CRM Enrichment Run is older than N days).
TLS fingerprinting. The fetcher uses requests, so its TLS ClientHello fingerprints (JA3/JA4) as a non-browser client. Bot walls that fingerprint TLS (Cloudflare, Akamai, DataDome) can still block it even though http.build_session sends browser-like headers. Real evasion needs a TLS-impersonating client (curl_cffi) or a headless browser — deliberately out of scope here.

Tests

Framework-native tests live in crm/domain_enrichment/tests/ and are discoverable by bench run-tests --app crm:

Pure unit (frappe.tests.UnitTestCase, no DB, offline fixtures): test_crawler.py, test_extractors.py, test_pipeline.py, test_ssrf.py. The rule executors are exercised with in-memory Rule objects (tests/fixtures.py); the SSRF guard mocks socket.getaddrinfo so DNS is deterministic and offline.
Integration (frappe.tests.IntegrationTestCase, real test DB): test_integration.py — seeder idempotency, mapper write-policies, the Run writer, API permission/allow-list enforcement, and the link-time Org→Lead/Deal copy. The crawl is monkeypatched to a canned EnrichmentResult, so the suite is network-free.

bench --site <site> run-tests --app crm --module crm.domain_enrichment.tests.test_extractors
bench --site <site> run-tests --app crm --module crm.domain_enrichment.tests.test_integration

README.md

Domain Enrichment