Proxy Providers & the Provider SDK

Add any proxy vendor to Octoprox as a declarative descriptor - from the admin UI, a mounted YAML file, or a Python plugin.

Overview

A provider is anything a credential’s type can point at. Octoprox ships four code-implemented providers (a static proxy list and the AWS, GCP and Azure cloud providers) and a set of descriptor providers. A descriptor is data, not code: it declares which fields a credential and a connector need, how those fields turn into proxy endpoints, and which vendor API calls discover options (zones, entry nodes, sub-users) or validate a credential.

Because most residential and datacenter vendors expose the same shape - a gateway host, a username or password with encoded targeting parameters, and a REST API for account management - one engine executes every descriptor. Adding a vendor means writing a descriptor, not a provider class.

Shipped descriptors

Licensing: the engine, the SDK and the UI are Apache 2.0. The shipped descriptor files in api/providers/builtin/ are proprietary (all rights reserved) and may only be used as part of an Octoprox installation; see the LICENSE file in that directory. Descriptors you author yourself are your own.

Beta descriptors follow the vendor’s public documentation but have not yet been exercised against a live account. Existing Oxylabs and Bright Data credentials, connectors and proxies created before the SDK are adopted in place: the built-in descriptors keep the same type ids and config keys.

Adding a provider from the UI

Admins open Settings → Providers. Shipped providers are read-only; any of them can be exported as YAML or duplicated as a starting point. Every descriptor provider, shipped or custom, has a Test tab where it can be tried with throwaway credentials before any credential or connector exists.

1. General - id (becomes the credential type), name, description, documentation link, logo.
2. Credential fields - what a user enters once per account. Mark API keys and passwords as secret so they are never written into proxy rows.
3. Connector fields - per-connector settings such as proxy counts, countries or zones. Fields can be shown conditionally (show_when) and selects can load their options from the vendor API.
4. Proxy types - one entry per product line, each with a mode (see below), gateway host and port, and username/password templates.
5. Discovery - credential validation, options sources and two-step auth flows, all expressed as HTTP calls with JMESPath extraction.
6. Test - try the descriptor with throwaway config before saving. The credential and connector forms behave as they will for users: dropdowns load from the vendor, selecting a zone fills the dependent fields, and dependent dropdowns wait for their inputs. Request through a proxy validates the credential, builds one proxy endpoint exactly as a connector would (session id, IP discovery or vendor list included) and fetches a URL through it, reporting the status, latency and exit IP. The other actions exercise the vendor API: credential validation, options sources and the proxy list. Requests are shown with secrets redacted and nothing is persisted; the draft stays invisible to other users until it is saved.
7. YAML - preview the normalised document, paste one to import, or export it.

Saving asks you to confirm the vendor hosts that will receive credential material. Changes take effect immediately on every instance, are versioned, and are written to an audit log with the actor and the host list.

Security model

A descriptor tells Octoprox to send credentials to a URL, so descriptors are treated as sensitive configuration:

• Only admins can create, edit, import or delete descriptors. Editors use them; viewers see them.
• Vendor calls are HTTPS only and every hostname is resolved and checked against private, loopback, link-local and cloud-metadata ranges before connecting. The connection is pinned to the vetted address. Redirects are not followed and pagination may not leave the original host. The test request through a proxy applies the same checks to the gateway host and to the URL it fetches.
• Templates are plain {placeholder} substitution; extraction is JMESPath. Neither can reach anything outside the values Octoprox explicitly provides.
• Every egress host must be confirmed on save, and again whenever a new host is added. The credential form shows users where their secret is sent.
• Every mutation is audited (provider_audit_log) with the actor, action and host list.
• Python is never accepted from the UI. Providers that genuinely need code are installed by the operator (see below).

Response size and time limits are operator settings (provider_http_max_response_bytes, default 50 MB, 0 disables; provider_http_timeout_seconds, default 60).

Backups

Admin-authored descriptors and their audit log are part of the Backup & Migration export and are restored by a replace-import, taking effect on the importing instance immediately. Shipped descriptors, mounted YAML files and Python plugins are part of the deployment, not the database, and are not included.

Operator-installed providers

Two more tiers exist for operators who deploy Octoprox:

• Mounted YAML - set OCTOPROX_PROVIDERS_DIR (or providers.dir in the YAML config) to a directory of descriptor files. They are loaded at startup, appear in the catalog as source file, and are read-only in the UI.
• Python entry points - a package can register providers under the octoprox.providers entry-point group. An entry point may resolve to a ProviderDescriptor, a descriptor dict, a path to a YAML file, or a class implementing the SyncableProvider protocol with a descriptor attribute (for vendors that need code, e.g. signed requests). Plugins run in-process with full trust; install only what you would install as part of the image.

[project.entry-points."octoprox.providers"]
acme = "acme_octoprox:AcmeProvider"

Descriptor reference

A descriptor is a YAML (or JSON) document. Export any shipped provider for a complete example; the Bright Data descriptor exercises almost every feature.

id: acme                      # lowercase letters, digits, - and _; becomes the credential type
name: Acme Proxies
description: Residential proxies
docs_url: https://docs.acme.example
beta: false

credential_fields:
  - key: api_key
    label: API key
    type: password
    secret: true
    required: true
  - key: username
    label: Proxy username
    required: true
  - key: password
    label: Proxy password
    type: password
    secret: true
    required: true

connector_fields:
  - key: num_proxies
    label: Number of proxies
    type: number
    required: true
    default: 1
    min: 1
    max: 1000
  - key: country_code
    label: Country
    type: select
    options_preset: countries   # built-in country list
  - key: region
    label: Region
    type: select
    options_from: regions       # dynamic, see options below
    fill: { region_password: password }   # copy option extras into other fields
    show_when: { field: credential.api_key }

proxy_types:
  - key: residential
    label: Residential
    mode: session
    host: gw.acme.example
    port: 9000
    protocol: http
    username:
      separator: "-"
      parts:
        - text: "{credential.username}"
        - text: "cc-{geo.country|lower}"          # the country this slot or request targets
          when: { field: geo.country }             # or a list: every condition must hold
        - text: "city-{geo.city}"                  # dynamic sessions: what a -city- request asked for
          when: { field: geo.city }
        - text: "sid-{session_id}"
    password: "{credential.password}"
    tags: [acme, residential]
    metadata:
      country_code: "{geo.country}"

validation:
  call:
    url: https://api.acme.example/me
    headers: { Authorization: "Bearer {credential.api_key}" }
  success: "active"
  capture: { account_id: id }
  error_message: Invalid Acme API key

options:
  regions:
    call:
      url: https://api.acme.example/regions
      headers: { Authorization: "Bearer {credential.api_key}" }
    items: regions
    value: code
    label: name
    extra: { password: pass }

Fields

Templates

Templates appear in hosts, ports, usernames, passwords, metadata and every HTTP call. Placeholders: {credential.<key>}, {connector.<key>}, {geo.<level>}, {session_id}, {index}, {port}, {discovered_ip}, {auth.token}, {item.<key>}. Filters: |lower, |upper, |urlencode, |nospace (drops everything but letters and digits: los_angeles becomes losangeles), |or:fallback. A composed template (separator + parts) drops parts whose when condition is false and any part that renders empty.

The geo namespace is the place the proxy being rendered is targeted at, and the only thing a template should read the country from. Octoprox sets it per render: for a pooled connector each slot group is rendered with one of the connector’s listed countries (or the country a client asked for on demand), so {geo.country} is that group’s country; for a dynamic-sessions request it is the country rendered for that request (the -cc- code or the allow-list pick), and {geo.state}, {geo.state_name} and {geo.city} are what the request asked for below it. geo.state is the ISO 3166-2 subdivision part (NY, ENG), geo.state_name that state’s name as a slug (new_york, US states only, empty for any other subdivision), geo.city the city slug (los_angeles). Everything in geo is empty when nothing is targeted, so a part conditioned on it (when: { field: geo.city }) is dropped. The connector’s country field itself is the list the admin configured and is never narrowed: a template that reads {connector.<country field>} gets the whole list, and the descriptor editor warns about it. Descriptors stored before the geo namespace existed were rewritten to it by a migration; nothing needs editing by hand.

Secret fields render as {key} runtime placeholders inside stored proxies; everything else is baked in.

Proxy type modes

host and port are templates too, so a connector field can override the vendor default: host: "{connector.gateway_host|or:proxy.geonode.io}", port: "{connector.gateway_port|or:10000}". A port template must render to a number between 1 and 65535, and in port mode it is the first port of the sequence.

count_field (default connector.num_proxies) names the variable holding the desired slot count; in list mode it is an optional cap. proxy_type_field selects the proxy type from a select field when a descriptor has more than one.

Dynamic sessions. A session type provisions a fixed pool of sessions by default: num_proxies rows, each with a baked-in session id, and a group per country. Residential and mobile products do not need that. Set the connector field named by session_mode_field (default connector.session_mode) to dynamic and the connector holds exactly one gateway row whose credentials are rendered per request instead:

• a client -sessid-<value> is hashed, together with the project id, into a session id of the descriptor’s session_id alphabet and length, so the same value reaches the vendor as the same session on every request and every instance, with no shared state and without exposing the client’s string;
• a request without -sessid- gets a freshly generated session id, so the vendor rotates the exit every request;
• -cc-<code> is rendered as {geo.country} for that request and into the vendor session id, so a client session holds one vendor session per country it asks for and returns to the same exit when it asks for that country again; the connector’s country list, when set, is an allow-list (a request for an unlisted country skips the connector) and a request with no country gets one of the listed countries, the same one for the life of a client session and a random one per session-less request; host and port templates that reference geo.country are rendered per request too, so regional gateways work;
• -st-<code> and -city-<slug> are rendered as {geo.state} (or {geo.state_name}) and {geo.city}, and are part of the vendor session id too. Which levels a type can say is read off its templates: a part referencing geo.state or geo.state_name targets states, one referencing geo.city targets cities. A request naming a level the templates cannot carry skips the connector rather than being widened to the country; a type that only knows the state’s name (geo.state_name) serves US states alone. targeting narrows the combinations the vendor accepts: city_requires_state: true (NetNut’s res_sc-us_texas_dallas) and state_or_city: true (one of the two per request);
• num_proxies is ignored; the shipped descriptors hide it with show_when when dynamic;
• the field named by exit_sample_field (default connector.exit_sample_percent, 0-100, default 5) is the share of requests without a client session that preflight echoes to observe their exit, see IP attribution.

The gateway row is not health-checked: no single exit speaks for a pool that hands out a new one per request, and a probe would only spend vendor traffic on judging a random exit. The health checker marks it healthy without a request, so it is always routable; a dead credential or blocked gateway host shows up as failed client requests on the connector. Metrics, rate limits and quarantine attach to that one row, which is to say to the connector. Routing does not count rows either: the connector’s share of a project’s traffic comes from its weight, so one gateway row next to a fifty-slot pool takes half the requests at equal weights. IP attribution treats the row differently: its exits belong to sessions, not to the row, so nothing is written on it and its sightings come from preflight (see IP attribution). The shipped Oxylabs, Bright Data, Decodo, IPRoyal and NetNut descriptors expose the choice as Sessions: Fixed pool / Dynamic (unlimited) on their session-based types; a custom descriptor adds a select field with values pool and dynamic under the key its session_mode_field names. Switching an existing pool connector to dynamic removes its slot rows on the next sync and creates the gateway row; sticky bindings to the removed rows rebind on the next request.

Countries. A connector field of type country (or a select with options_preset: countries) holds a list of ISO codes; the form offers a multi-select and the API accepts one code, a list or a comma-separated string. num_proxies (the count_field) is always per country. When the proxy type’s username, password, host or port template references {geo.country}, slots are provisioned as one group per listed country, each rendered with geo.country set to that country and tagged with geo in its metadata. With the field left empty the type provisions one ungeo-targeted group and, for session mode, creates a group per country on demand when clients request it with -cc-<code>. When the templates do not reference geo.country but the type is port mode, the list becomes a discovery filter: ports are scanned (or known IPs picked) and only IPs whose reported country (discovery.country_path, known_ips.country) is listed are kept, num_proxies per country, stopping on the usual failure and duplicate limits or after discovery.max_scan_ports ports. In both cases the periodic refresh removes a proxy whose reported location leaves its allowed countries and the sync that follows replaces it. Templates render a list value as its single element, or comma-joined when it holds several.

HTTP calls, auth flows, options and validation

• Call: method, url, headers, params (empty rendered values are dropped), body (JSON, strings templated), auth (name of an auth flow), paginate: { next_url: <JMESPath> }.
• Auth flow (auth.<name>): a call plus token_path; the token is cached for ttl_seconds and exposed as {auth.token}.
• Options source (options.<name>): a call plus items, value, label, description, extra (named JMESPath or value mappings), enrich (per-option follow-up calls with when and merge), filter and cache_seconds. label and description are evaluated over the enriched option, so they can mention values from follow-up calls. group_by_value: true collapses items sharing a value into one option with a count extra - how Bright Data turns a list of exit IPs into countries with IP counts.
• Validation: a call plus optional success predicate, capture (stored on the credential), error_message and when.

Value mappings turn API values into yours: { path: type, map: [{ starts_with: res, to: residential }], default: unknown }.

Configuration

The two egress overrides exist for development against local mocks; leave them off in production.

API

See the Providers section of the API reference.