Protection Against Web Scrapers (PAWS)

Web proxy for Mastodon that puts public profiles behind an auth layer.

How it works

PAWS sits between Mastodon and your front-facing web proxy to intercept incoming requests. If a profile, toot, or any related json is requested, it will be blocked unless authenticated. If authenticated fetches on mastodon are disabled, PAWS will check signatures instead

Installation

Python 3.6.0+ (3.8.0 recommended)

python3 -m pip install -r requirements.txt --user

Configuration

data/production.env:

# Path to mastodon instance. Defaults to current working dir
MASTOPATH=/home/mastodon/glitch-soc

# The address to mastodon
MASTOHOST=localhost:3000

# Listen address and port for PAWS. Can safely be ignored if running on same host as web server
PAWS_HOST=127.0.0.1
PAWS_PORT=3001

# Domain to list in the actor. Only needed if you plan on using the whitelist
PAWS_DOMAIN=bappypaws.example.com

Extra environment variables (cannot be put in production.env):

• LOGDATE: boolean (default: yes). Set to no to remove access times from the console log

If Mastodon is on a different host, you'll have to copy Mastodon's .env.production to the working directory or mount the Mastodon directory to make it accessible to PAWS

Caddy

Append this to caddy's mastodon config:

rewrite {
	if_op and
	if {path} starts_with /users
	if {path} not_ends_with inbox
	to {path} /auth/{path}
}

rewrite {
	if_op or
	if {path} starts_with /@
	if {path} starts_with /paws
	to {path} /auth/{path}
}

proxy /auth localhost:3001 {
	without /auth
	transparent
}

Config for PAWS domain:

paws.bappypaws.example.com {
	proxy / localhost:3001 {
		transparent
	}
}

Nginx

Append this to the mastodon config (Thanks to @finkeldoodle@transfur.online for the conversion)

## Note: Order of location blocks is IMPORTANT.
## Nginx matches longest non-regex location block, then **FIRST** matching regex block takes preference.
## (regex blocks are ones with ~ or ~* before the path)

## This file is designed to be dropped into the Mastodon server block, probably before all the location blocks.

## If starts with /users, ends with inbox, bypass filter
location ~* ^/users[\s\S]*inbox {
        try_files $uri @proxy;
}

## If starts with /users, /@, or /paws, enter filter
location ~* ^/(users|@|paws) {
        try_files '' @paws;
}


location @paws {
        proxy_set_header Host                   $host;
        proxy_set_header X-Real-IP              $remote_addr;
        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto      https;
        proxy_set_header Proxy                  "";
        proxy_pass_header                       Server;

        proxy_pass http://localhost:3001;
}

Config for PAWS domain:

server {
	listen 80;
	listen [::]:80;
	server_name paws.bappypaws.example.com;
	root /home/mastodon/live/public;
	location /.well-known/acme-challenge/ { allow all; }
	location / { return 301 https://$host$request_uri; }
}

server {
	listen 443 ssl http2;
	listen [::]:443 ssl http2;
	server_name paws.bappypaws.example.com;

	ssl_protocols TLSv1.2 TLSv1.3;
	ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
	ssl_prefer_server_ciphers on;
	ssl_session_cache shared:SSL:10m;

	# Uncomment these lines once you acquire a certificate:
	# ssl_certificate     /etc/letsencrypt/live/paws.bappypaws.example.com/fullchain.pem;
	# ssl_certificate_key /etc/letsencrypt/live/paws.bappypaws.example.com/privkey.pem;

	keepalive_timeout    70;
	sendfile             on;
	client_max_body_size 80m;

	gzip on;
	gzip_disable "msie6";
	gzip_vary on;
	gzip_proxied any;
	gzip_comp_level 6;
	gzip_buffers 16 8k;
	gzip_http_version 1.1;
	gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	add_header Strict-Transport-Security "max-age=31536000";

	location / {
			proxy_set_header Host                   $host;
			proxy_set_header X-Real-IP              $remote_addr;
			proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto      https;
			proxy_set_header Proxy                  "";
			proxy_pass_header                       Server;

			proxy_pass http://localhost:3001;
	}
}

Mastodon

While it isn't necessary, I highly recommend turning on authorized fetches (v3.0+) to let PAWS pass json requests directly through to mastodon. Also upgrade to at least v3.0 to be able to properly interact with other instances that have auth fetches turned on.

.env.production:

AUTHORIZED_FETCH=true

Usage

Start the server with ./server.py. If you want to edit the config, just run ./server.py edit.

WebUI usage

If your account is an admin account according to Mastodon, you can manage the PAWS whitelist at {domain}/paws. Instances in this whitelist will have their fetches signed if they don't sign them themselves.