anuna-research/ar-crawl
1
0
Fork 0
Code Issues 1 Pull requests Projects Packages Wiki Activity Actions

Feature: SOCKS5/HTTP Proxy Support for IP Selection #1

New issue
Open
opened 2025-12-31 00:11:01 +00:00 by hugooconnor · 0 comments
hugooconnor commented 2025-12-31 00:11:01 +00:00 (Migrated from codeberg.org)
Copy link

Summary

Add proxy support to ar-crawl allowing users to specify outbound IP addresses via SOCKS5 or HTTP proxies. This enables IP rotation, geo-targeting, and avoiding rate limits when crawling.

Motivation

  • Avoid IP-based rate limiting and blocks
  • Geo-target requests from specific regions
  • Rotate IPs across crawl sessions or per-request
  • Use residential/datacenter proxy pools for better success rates

Proposed Implementation

1. CLI Flags

Add new flags to crawl and crawl-site commands:

# Single proxy
ar-crawl crawl https://example.com \
  --proxy socks5://proxy.example.com:1080

# With authentication
ar-crawl crawl https://example.com \
  --proxy socks5://proxy.example.com:1080 \
  --proxy-user myuser \
  --proxy-pass mypass \
  --proxy-bypass ".local,.internal"

# Rotating proxies from file
ar-crawl crawl-site https://example.com \
  --proxy-list proxies.txt \
  --proxy-rotation per-request

New flags:

Flag Description
--proxy <url> Proxy URL (socks5://host:port or http://host:port)
--proxy-user <user> Proxy username (or embed in URL)
--proxy-pass <pass> Proxy password (or embed in URL)
--proxy-bypass <domains> Comma-separated domains to bypass
--proxy-list <file> File with proxy URLs (one per line)
--proxy-rotation <mode> per-request, sticky, or round-robin

2. Config File Support

Add proxy section to config JSON:

{
  "proxy": {
    "enabled": true,
    "server": "socks5://${PROXY_HOST}:${PROXY_PORT}",
    "username": "${PROXY_USER}",
    "password": "${PROXY_PASS}",
    "bypass": [".local", ".internal"],
    "rotation": "per-request",
    "pool": [
      "socks5://proxy1.example.com:1080",
      "socks5://proxy2.example.com:1080"
    ]
  },
  "services": {
    "playwright": {
      "use_proxy": true
    },
    "direct": {
      "use_proxy": true
    }
  }
}

3. Environment Variables

export PROXY_SERVER=socks5://proxy.example.com:1080
export PROXY_USER=myuser
export PROXY_PASS=mypass
# Or combined:
export PROXY_URL=socks5://user:pass@proxy.example.com:1080

4. Priority Order

CLI flag > config file > environment variable > disabled

Technical Implementation

Playwright Service (playwright-service/server.js)

Playwright natively supports SOCKS5. Modify to accept proxy in request options:

// In fetchPage() - context creation
const context = await b.newContext({
  viewport,
  userAgent,
  proxy: options.proxy ? {
    server: options.proxy.server,
    username: options.proxy.username,
    password: options.proxy.password,
    bypass: options.proxy.bypass
  } : undefined
});

Update /fetch endpoint to accept proxy config:

POST /fetch
{
  "url": "https://target.com",
  "proxy": {
    "server": "socks5://proxy.example.com:1080",
    "username": "user",
    "password": "pass"
  }
}

Direct HTTP Service (crawl-service-adaptor.rkt)

Two options:

Option A: Use Racket's built-in HTTP proxy support

  • Set current-proxy-servers parameter
  • Works for HTTP/HTTPS proxies, not SOCKS5
  • Limited but requires no new dependencies

Option B: Switch to http-easy + socks5 packages

  • Install: raco pkg install http-easy socks5
  • Full SOCKS5 support with make-socks5-proxy
  • More flexible, per-request proxy control
(require http-easy socks5)

(define (make-proxy-from-config config)
  (cond
    [(string-prefix? (hash-ref config 'server) "socks5://")
     (make-socks5-proxy host port
       #:username-password (cons user pass))]
    [else
     (make-http-proxy (hash-ref config 'server))]))

(define session (make-session #:proxies (list proxy)))
(session-request session url)

CLI Parameters (cli.rkt)

Add near existing parameters (~line 29):

(define proxy-server (make-parameter #f))
(define proxy-user (make-parameter #f))
(define proxy-pass (make-parameter #f))
(define proxy-bypass (make-parameter #f))
(define proxy-list-file (make-parameter #f))
(define proxy-rotation (make-parameter 'per-request))

Add to parse-crawl-args (~line 1070):

[("--proxy") server "Proxy URL (socks5://host:port or http://host:port)"
 (proxy-server server)]
[("--proxy-user") user "Proxy username"
 (proxy-user user)]
[("--proxy-pass") pass "Proxy password"
 (proxy-pass pass)]
[("--proxy-bypass") domains "Comma-separated domains to bypass"
 (proxy-bypass domains)]
[("--proxy-list") file "File with proxy URLs (one per line)"
 (proxy-list-file file)]
[("--proxy-rotation") mode "Rotation mode: per-request, sticky, round-robin"
 (proxy-rotation (string->symbol mode))]

Proxy Pool Manager

For --proxy-list support, implement rotation logic:

(struct proxy-pool (proxies current-index) #:mutable)

(define (load-proxy-pool file)
  (proxy-pool (file->lines file) 0))

(define (next-proxy pool mode)
  (case mode
    [(per-request round-robin)
     (begin0
       (list-ref (proxy-pool-proxies pool) (proxy-pool-current-index pool))
       (set-proxy-pool-current-index! pool
         (modulo (add1 (proxy-pool-current-index pool))
                 (length (proxy-pool-proxies pool)))))]
    [(sticky)
     (list-ref (proxy-pool-proxies pool) (proxy-pool-current-index pool))]
    [(random)
     (list-ref (proxy-pool-proxies pool)
               (random (length (proxy-pool-proxies pool))))]))

Files to Modify

  1. src/cli.rkt - Add CLI flags and parameters
  2. src/config-manager.rkt - Parse proxy config section
  3. src/crawl-service-adaptor.rkt - Add proxy support to direct HTTP
  4. src/production-crawler.rkt - Pass proxy config to services
  5. playwright-service/server.js - Accept proxy in fetch options
  6. config/production.json - Add proxy config template

Dependencies

  • Racket: http-easy, socks5 packages (for SOCKS5 in direct service)
  • Node.js: None (Playwright has native SOCKS5 support)

Install Racket packages:

raco pkg install http-easy socks5

Testing

  1. Test with free SOCKS5 proxy
  2. Test proxy rotation with multiple proxies
  3. Test authentication
  4. Test bypass rules
  5. Test fallback when proxy fails
  6. Integration test with Playwright service

References

Labels

enhancement, networking

## Summary Add proxy support to ar-crawl allowing users to specify outbound IP addresses via SOCKS5 or HTTP proxies. This enables IP rotation, geo-targeting, and avoiding rate limits when crawling. ## Motivation - Avoid IP-based rate limiting and blocks - Geo-target requests from specific regions - Rotate IPs across crawl sessions or per-request - Use residential/datacenter proxy pools for better success rates ## Proposed Implementation ### 1. CLI Flags Add new flags to `crawl` and `crawl-site` commands: ```bash # Single proxy ar-crawl crawl https://example.com \ --proxy socks5://proxy.example.com:1080 # With authentication ar-crawl crawl https://example.com \ --proxy socks5://proxy.example.com:1080 \ --proxy-user myuser \ --proxy-pass mypass \ --proxy-bypass ".local,.internal" # Rotating proxies from file ar-crawl crawl-site https://example.com \ --proxy-list proxies.txt \ --proxy-rotation per-request ``` **New flags:** | Flag | Description | |------|-------------| | `--proxy <url>` | Proxy URL (`socks5://host:port` or `http://host:port`) | | `--proxy-user <user>` | Proxy username (or embed in URL) | | `--proxy-pass <pass>` | Proxy password (or embed in URL) | | `--proxy-bypass <domains>` | Comma-separated domains to bypass | | `--proxy-list <file>` | File with proxy URLs (one per line) | | `--proxy-rotation <mode>` | `per-request`, `sticky`, or `round-robin` | ### 2. Config File Support Add `proxy` section to config JSON: ```json { "proxy": { "enabled": true, "server": "socks5://${PROXY_HOST}:${PROXY_PORT}", "username": "${PROXY_USER}", "password": "${PROXY_PASS}", "bypass": [".local", ".internal"], "rotation": "per-request", "pool": [ "socks5://proxy1.example.com:1080", "socks5://proxy2.example.com:1080" ] }, "services": { "playwright": { "use_proxy": true }, "direct": { "use_proxy": true } } } ``` ### 3. Environment Variables ```bash export PROXY_SERVER=socks5://proxy.example.com:1080 export PROXY_USER=myuser export PROXY_PASS=mypass # Or combined: export PROXY_URL=socks5://user:pass@proxy.example.com:1080 ``` ### 4. Priority Order ``` CLI flag > config file > environment variable > disabled ``` ## Technical Implementation ### Playwright Service (`playwright-service/server.js`) Playwright natively supports SOCKS5. Modify to accept proxy in request options: ```javascript // In fetchPage() - context creation const context = await b.newContext({ viewport, userAgent, proxy: options.proxy ? { server: options.proxy.server, username: options.proxy.username, password: options.proxy.password, bypass: options.proxy.bypass } : undefined }); ``` Update `/fetch` endpoint to accept proxy config: ```json POST /fetch { "url": "https://target.com", "proxy": { "server": "socks5://proxy.example.com:1080", "username": "user", "password": "pass" } } ``` ### Direct HTTP Service (`crawl-service-adaptor.rkt`) Two options: **Option A: Use Racket's built-in HTTP proxy support** - Set `current-proxy-servers` parameter - Works for HTTP/HTTPS proxies, not SOCKS5 - Limited but requires no new dependencies **Option B: Switch to http-easy + socks5 packages** - Install: `raco pkg install http-easy socks5` - Full SOCKS5 support with `make-socks5-proxy` - More flexible, per-request proxy control ```racket (require http-easy socks5) (define (make-proxy-from-config config) (cond [(string-prefix? (hash-ref config 'server) "socks5://") (make-socks5-proxy host port #:username-password (cons user pass))] [else (make-http-proxy (hash-ref config 'server))])) (define session (make-session #:proxies (list proxy))) (session-request session url) ``` ### CLI Parameters (`cli.rkt`) Add near existing parameters (~line 29): ```racket (define proxy-server (make-parameter #f)) (define proxy-user (make-parameter #f)) (define proxy-pass (make-parameter #f)) (define proxy-bypass (make-parameter #f)) (define proxy-list-file (make-parameter #f)) (define proxy-rotation (make-parameter 'per-request)) ``` Add to `parse-crawl-args` (~line 1070): ```racket [("--proxy") server "Proxy URL (socks5://host:port or http://host:port)" (proxy-server server)] [("--proxy-user") user "Proxy username" (proxy-user user)] [("--proxy-pass") pass "Proxy password" (proxy-pass pass)] [("--proxy-bypass") domains "Comma-separated domains to bypass" (proxy-bypass domains)] [("--proxy-list") file "File with proxy URLs (one per line)" (proxy-list-file file)] [("--proxy-rotation") mode "Rotation mode: per-request, sticky, round-robin" (proxy-rotation (string->symbol mode))] ``` ### Proxy Pool Manager For `--proxy-list` support, implement rotation logic: ```racket (struct proxy-pool (proxies current-index) #:mutable) (define (load-proxy-pool file) (proxy-pool (file->lines file) 0)) (define (next-proxy pool mode) (case mode [(per-request round-robin) (begin0 (list-ref (proxy-pool-proxies pool) (proxy-pool-current-index pool)) (set-proxy-pool-current-index! pool (modulo (add1 (proxy-pool-current-index pool)) (length (proxy-pool-proxies pool)))))] [(sticky) (list-ref (proxy-pool-proxies pool) (proxy-pool-current-index pool))] [(random) (list-ref (proxy-pool-proxies pool) (random (length (proxy-pool-proxies pool))))])) ``` ## Files to Modify 1. `src/cli.rkt` - Add CLI flags and parameters 2. `src/config-manager.rkt` - Parse proxy config section 3. `src/crawl-service-adaptor.rkt` - Add proxy support to direct HTTP 4. `src/production-crawler.rkt` - Pass proxy config to services 5. `playwright-service/server.js` - Accept proxy in fetch options 6. `config/production.json` - Add proxy config template ## Dependencies - **Racket**: `http-easy`, `socks5` packages (for SOCKS5 in direct service) - **Node.js**: None (Playwright has native SOCKS5 support) Install Racket packages: ```bash raco pkg install http-easy socks5 ``` ## Testing 1. Test with free SOCKS5 proxy 2. Test proxy rotation with multiple proxies 3. Test authentication 4. Test bypass rules 5. Test fallback when proxy fails 6. Integration test with Playwright service ## References - [Racket socks5 library](https://docs.racket-lang.org/socks5/Provides.html) - [Racket http-easy](https://docs.racket-lang.org/http-easy/index.html) - [Playwright proxy docs](https://playwright.dev/docs/api/class-browsertype) - [SOCKS5 RFC 1928](https://datatracker.ietf.org/doc/html/rfc1928) ## Labels `enhancement`, `networking`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/android-session
v1.0.30
v1.0.29
v1.0.28
v1.0.27
v1.0.26
v1.0.25
v1.0.24
v1.0.23
v1.0.22
v1.0.20
v1.0.19
v1.0.18
v1.0.17
v1.0.16
v1.0.15
v1.0.14
v1.0.13
v1.0.12
v1.0.11
v1.0.10
v1.0.9
v1.0.8
v1.0.7
v1.0.6
v1.0.5
v1.0.4
v1.0.3
v1.0.2
v1.0.1
v0.0.1
v1.0.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
anuna-research/ar-crawl#1
No description provided.