Skip to content

About

Find which shop platform and which email or SMS marketing tool a website uses, and read its public product feed. No API key.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

store-tech-collector

Two small readers for any website, over plain HTTP, with no API key:

  • Platform reader: which shop platform the site runs on (Shopify, WooCommerce, BigCommerce, Magento, PrestaShop, Squarespace; Wix and Shopline are recognised too), and that platform's public product feed: titles, URLs, prices, compare-at prices, currency, image and, where the platform says it, the catalogue size.
  • Email reader: which email/SMS marketing tool the site uses (Klaviyo, Omnisend, Mailchimp, Attentive, Postscript, Privy, HubSpot and about 20 more), and how many signup forms it shows (counted for Klaviyo shops).

Detection only: the email reader never reads email content and never subscribes anybody.

Install

pip install store-tech-collector

Python 3.9 or newer. One dependency: httpx.

Not on PyPI yet. Until the first release, install from a checkout: pip install .

Command line

store-tech example-shop.com                         # both readers
store-tech -r platform --max-products 20 a.com b.com
store-tech -r email -f domains.txt --format jsonl -o out.jsonl
store-tech [DOMAIN ...] [-f FILE] [-r platform|email|both] [--max-products N] [--no-products]
           [--format json|jsonl] [-o FILE] [--concurrency N] [--timeout S] [--delay S]
           [--proxy URL] [--user-agent UA] [--version]
  • Domains come from arguments and/or UTF-8 files (-f, one per line, # starts a comment; -f - reads stdin). URLs are accepted; their host is used (www. is dropped).
  • --max-products (1-250, default 100) caps the feed; each platform also has its own cap (see Limits). --no-products keeps the counts and leaves out the product list.
  • --proxy http://user:pass@host:port (or the STORE_TECH_PROXY environment variable) is passed to httpx unchanged.
  • --delay is the pause before every request (default 0.25 s). Requests for one domain are sequential; --concurrency (default 4) is the number of domains read in parallel.
  • Error messages name the option, never the value typed.

Python

from store_tech_collector import Settings, collect

for rec in collect(["example-shop.com"], Settings(readers=("platform", "email"), max_products=20)):
    print(rec["domain"], rec["platform"]["platform"], rec["platform"]["feed"])
    print(rec["email"]["primaryProvider"], rec["email"]["signupForms"])

collect_async(domains, settings) is the async version; read_domain(domain, settings) reads one normalised domain. The parsers can be used on HTML you already have:

from store_tech_collector import detect_platform, detect_providers

platform, evidence = detect_platform(html, response_headers)   # ("shopify", ["html:Shopify.shop", ...])
tools = detect_providers(html)                                  # [{"name": "Klaviyo", "kind": "esp", ...}]

The record

One record per input, in input order. A reader that was not asked for has no key.

Field Meaning
input_index position of the input (join key)
domain, finalHost the normalised domain; the host the homepage ended on after redirects
ok, error ok is true when every reader asked for gave a result; error is the first reader's error code
requests, fetchedAt HTTP requests made for this domain; UTC time
platform.platform shopify, woocommerce, bigcommerce, magento, prestashop, squarespace, wix or other
platform.platformEvidence the markers that matched (html:..., header:..., feed:<kind>)
platform.feed {kind, ok, error, productCount, catalogueTotal, currency, products[]} or null
platform.feed.products[] {title, url, price, compareAtPrice, currency, imageUrl}; prices in major units
email.providers every tool found, primary order: email (esp) > SMS > popup > native
email.primaryProvider the first of providers, or null
email.providerEvidence [{name, evidence}]: script_or_app_block, web_pixel, app, native_form
email.signupForms {count, live, types[], source}; source is klaviyo_api (counted), klaviyo_unread (Klaviyo seen, file not read) or detected_only (count null)
*.extractionTier, *.extractionTiers, *.tierAttempts which step answered, and every step tried with its code and time

Feed kind: shopify_products_json, woocommerce_store_api, bigcommerce_graphql, magento_graphql, magento_rest, prestashop_best_sales, squarespace_json, or sitemap_jsonld (a few products read from the sitemap and product pages when the platform has no public feed).

Error codes: blocked (401/403/429/503 or a bot-wall page), http_<status>, empty, invalid_domain, a transport error name (ConnectError, ReadTimeout, ...) or all_tiers_failed. Step codes in tierAttempts include no_feed_route, no_feed, no_klaviyo_id and forms_api_<status>.

How it works

  1. One GET of the homepage per domain, shared by both readers.
  2. Platform: strict markers only (CDN hosts, platform headers, full globals such as Static.SQUARESPACE_CONTEXT); a bare word on the page never counts, and one weak Shopify or Wix reference is not enough. Then the platform's own public endpoint: /products.json (+ /meta.json), the WooCommerce Store API sorted by popularity, BigCommerce's storefront GraphQL bestSellingProducts (with the public storefront token the shop prints in its own homepage for its own scripts, read at run time and never stored or output), Magento GraphQL with the anonymous REST products-render-info as fallback, PrestaShop best-sales JSON, Squarespace ?format=json. Without a feed: the sitemap and up to 8 product pages (JSON-LD or Open Graph product tags).
  3. Email: vendor script hosts, app blocks and init calls in the HTML; Klaviyo also from Shopify's web-pixel list; the native Shopify newsletter form. When the page carries a Klaviyo company id, Klaviyo's public form-definition file (the one its on-site script downloads) gives the form count and types. The id is never output.

Limits

  • HTTP only. No browser is used. A site that blocks plain HTTP clients, or shows a bot wall, is reported as blocked and nothing else is tried. Tools injected later by other scripts (tag managers, app loaders) are not seen; the email reader only finds what the homepage HTML shows.
  • Signup forms are counted only for Klaviyo; for other tools signupForms.count is null (detected only).
  • Feed caps per platform: Shopify 250, WooCommerce 100, BigCommerce 50, Magento 50; PrestaShop and Squarespace return what their first page holds. Prices follow the visitor market the site serves to this client.
  • catalogueTotal is only known where the platform says it (Shopify /meta.json, WooCommerce X-WP-Total, BigCommerce, Magento, PrestaShop); for sitemap_jsonld it is the number of sitemap URLs read.
  • One retry (after 2 s) on a transport error or 502/504; no other retries.

Be polite

Read only sites you are allowed to read, respect their terms and robots rules, keep --concurrency and --delay gentle, and do not use this tool to get past blocks. It sends an identifying User-Agent by default.

License

MIT

About

Find which shop platform and which email or SMS marketing tool a website uses, and read its public product feed. No API key.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages