Outbound filtering of Web requests using Squid as a Proxy server
Frequently in my line of work I’ll be asked about filtering of outbound traffic from application servers. There are two schools of thought here, one is that an app server can have unfiltered access to the internet, and the other that the app server should have as little access to any resources (both inside and outside of the solution) as needed to preform its role.
This generally isn’t an issue if site to site VPNs, static IPs or similar are being used on the destination side. But what happens if your application requires access to something like Youtube, Facebook or Flickr. As these cloud services are not managed by the customer, we have no idea if they are on static IP addresses (and in the case of flickr, they do seem to change moderately frequently).
With this in mind a traditional Layer3/Layer4 firewall is only going to be able to handle this if it supports DNS resolution in its access-list set, and unfortunately (but for good reason) this is not a common feature. Cisco did introduce this to the ASA firewalls in 8.4, however I personally have not used this, so at the moment its still a bit of an unknown and I can’t recommend it to a customer.
There is however another way of doing this, whilst it might not be a perfect situation, it does at least allow you to filter outbound traffic.
The Squid proxy server has been around for quite some time and is quite a stable product, both in the forward (outbound) and reverse (inbound) HTTP proxy space. We’re going to use this to preform our outbound proxying. It is possible to use commercial products like a BlueCoat proxy, however I’m going to concentrate on the FOSS solution here.
Prerequisites #
Before we start we need to have the following:
- A Linux Server (for this example I’m going to be using CentOS 6.4, however any linux distribution should work)
Installing Squid #
This is a really simple task on most linux distributions, as not only has squid been since the early 90’s, it’s also really popular! You can use the package manager to install squid on most distributions
yum install -y squid
You should get a response similar to below:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package squid.x86_64 7:3.1.10-18.el6_4 will be installed
--> Processing Dependency: perl(DBI) for package: 7:squid-3.1.10-18.el6_4.x86_64
--> Processing Dependency: libltdl.so.7()(64bit) for package: 7:squid-3.1.10-18.el6_4.x86_64
--> Running transaction check
---> Package libtool-ltdl.x86_64 0:2.2.6-15.5.el6 will be installed
---> Package perl-DBI.x86_64 0:1.609-4.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================================================================================
Installing:
squid x86_64 7:3.1.10-18.el6_4 updates 1.7 M
Installing for dependencies:
libtool-ltdl x86_64 2.2.6-15.5.el6 base 44 k
perl-DBI x86_64 1.609-4.el6 base 705 k
Transaction Summary
=====================================================================================================================================================================================================
Install 3 Package(s)
Total download size: 2.5 M
Installed size: 7.5 M
Downloading Packages:
(1/3): libtool-ltdl-2.2.6-15.5.el6.x86_64.rpm | 44 kB 00:00
(2/3): perl-DBI-1.609-4.el6.x86_64.rpm | 705 kB 00:00
(3/3): squid-3.1.10-18.el6_4.x86_64.rpm | 1.7 MB 00:01
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 942 kB/s | 2.5 MB 00:02
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : perl-DBI-1.609-4.el6.x86_64 1/3
Installing : libtool-ltdl-2.2.6-15.5.el6.x86_64 2/3
Installing : 7:squid-3.1.10-18.el6_4.x86_64 3/3
Verifying : 7:squid-3.1.10-18.el6_4.x86_64 1/3
Verifying : libtool-ltdl-2.2.6-15.5.el6.x86_64 2/3
Verifying : perl-DBI-1.609-4.el6.x86_64 3/3
Installed:
squid.x86_64 7:3.1.10-18.el6_4
Dependency Installed:
libtool-ltdl.x86_64 0:2.2.6-15.5.el6 perl-DBI.x86_64 0:1.609-4.el6
Complete!
We now would need to configure squid to start on boot
chkconfig squid on
SSL Proxying #
Squid has a rather nice feature called SSLBump which allows us to preform a Man In the Middle SSL Proxy. Privacy issues aside on this feature (after all we’re using it for servers not for end users) this is going to work for us from the server side of things. One key thing to note is we have to trust the CA, that we’re going to generate, on all applications / servers. I’m not going to cover how to do this in this post.
Normally when we create an SSL certificate we’d do this with a specific domain, however as we’re going to be proxying for all domains we’re going to use a wildcard certificate. For the “Common Name” or Server name, we need to chose “*” as the value.
In order to create the CA you can follow the following post. One point of note is to ensure that you do not do this on the Squid server, as this would mean that should the server be compromised, the CA (which is trusted on multiple servers) is now trusted as well.
We need to create the certificate using the CA script as per the above post. CA -newreq This will look similar to
Generating a 2048 bit RSA private key
...................................+++
.........................+++
writing new private key to 'newkey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:
State or Province Name (full name) [London]:
Locality Name (eg, city) [London]:
Organization Name (eg, company) [AGrainOfSalt]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:*
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Request is in newreq.pem, private key is in newkey.pem
Once this is completed you’ll need to sign this with the CA -sign command
Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for /etc/pki/CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number:
c5:07:3c:dc:c5:8a:cb:ae
Validity
Not Before: Jul 21 16:45:48 2013 GMT
Not After : Jul 21 16:45:48 2014 GMT
Subject:
countryName = GB
localityName = London
organizationName = AGrainOfSalt
commonName = *
X509v3 extensions:
Authority Information Access:
OCSP - URI:http://162.13.47.187:8888
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
E8:30:79:47:75:8D:06:0C:CB:9E:84:47:65:61:D4:27:8D:61:52:D2
X509v3 Authority Key Identifier:
keyid:1F:F8:DB:5F:7B:FE:30:29:6F:E2:E2:C8:23:DE:5B:77:AE:82:2D:F8
Certificate is to be certified until Jul 21 16:45:48 2014 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
c5:07:3c:dc:c5:8a:cb:ae
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=GB, ST=London, O=AGrainOfSalt, CN=ca.test.local
Validity
Not Before: Jul 21 16:45:48 2013 GMT
Not After : Jul 21 16:45:48 2014 GMT
Subject: C=GB, L=London, O=AGrainOfSalt, CN=*
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:e0:e7:17:43:65:55:7c:da:56:46:4c:44:11:2a:
96:8b:b9:c0:d0:55:10:d7:f7:c7:ca:29:3b:2d:1d:
08:46:00:db:23:78:58:04:36:35:79:ca:5a:4f:a3:
81:31:c7:4c:ec:a9:07:46:af:60:98:9d:ff:06:1c:
58:8f:16:53:97:1b:f1:b0:17:b5:9a:5c:eb:eb:7c:
c5:a9:3a:93:e2:8b:23:ad:d9:54:1e:c3:99:2c:8f:
24:1e:b0:0b:d3:3a:2f:b3:72:79:f5:71:d9:3b:52:
de:78:18:c4:41:8e:dc:5c:4c:96:da:90:75:1f:21:
f5:83:91:30:64:11:fe:af:b1:e1:fb:4d:4a:1f:06:
a4:7a:b0:bf:91:bc:74:5b:27:88:e3:0d:2e:1c:3c:
3b:e8:6c:7b:32:90:60:c8:4c:2a:db:84:fc:c2:53:
7a:6c:4b:ae:45:4a:86:4a:c6:a4:88:24:33:a6:f4:
de:8b:56:3b:59:3f🇩🇪92:c4:9d:f2:d4:2b:53:da:
62:05:94:7e:bc:f9:f8:45:53:88:56:13:30:67:ed:
6c:e7:0c:32:f8:53:03:49:3e:c3:c2:b6:06:83:e0:
d1:80:51:f4:86:f7:52:b0:7d:05:34:39:df:3b:46:
29:62:24:43:9a:b9:fe:ac:10:30:17:3a:e3:9a:d5:
ab:69
Exponent: 65537 (0x10001)
X509v3 extensions:
Authority Information Access:
OCSP - URI:http://162.13.47.187:8888
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
E8:30:79:47:75:8D:06:0C:CB:9E:84:47:65:61:D4:27:8D:61:52:D2
X509v3 Authority Key Identifier:
keyid:1F:F8:DB:5F:7B:FE:30:29:6F:E2:E2:C8:23:DE:5B:77:AE:82:2D:F8
Signature Algorithm: sha1WithRSAEncryption
83:2c:00:e4:58:1f:db:02:aa:a8:ff:52:45:d3:63:4f:8f:47:
8f:65:1e:21:c7:d7:c2:76:df:03:af:64:c1:e0:2c:d5:92:44:
35:a4:c3:02:78:0a:43:0f:ed:91:03:2a:f8:00:5c:97:f7:fc:
6f:81:69:96:3d:c3:ce:80:f2:d2:0d🇩🇪5c:2d:f0:27:ca:ba:
1c:b4:09:8a:cc:b6:76:06:9f:a9:ad:a5:bf:9c:7d:9d:c5:f1:
32:d4:d6:30:bf:bc:57:19:76:06:51:ee:e8:8b:f2:a1:4c:f7:
69:ef:d4:96:58:4d:5a🇩🇪98:c5:f4:17:af:b5:d2:cc:26:f0:
69:43:72:77:4a:e7:cf:79:62:b6:a4:47:75:4b:29:dc:2f:6f:
e6:c6:d5:1a:79:e2:1d:bf:f8:82:04:fe:d0:21:7a:8b:4e:1e:
93:10:f4:81:d6:9d:41:9a:70:02:e7:3f:22:1a:d8:a6:2e:21:
8a:b1:34:03:1c:83:ca:8c:19:59:1b:d6:85:f7:eb:e2:a7:32:
d9:61:5c:e6:68:b3:ef:ba:27:4a:3f:ff:5a:6e:d2:60:36:bb:
a2:0a:ba:aa:4f:d9:22:7e🆎7e:78:80:87:6d:51:92:44:61:
e6:aa:63:fa:e5:13:88:c9:f9🇩🇪90:31:9f:28:78:ca:79:74:
ff:0e:a4:0b
-----BEGIN CERTIFICATE-----
MIIDxTCCAq2gAwIBAgIJAMUHPNzFisuuMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV
BAYTAkdCMQ8wDQYDVQQIDAZMb25kb24xFTATBgNVBAoMDEFHcmFpbk9mU2FsdDEW
MBQGA1UEAwwNY2EudGVzdC5sb2NhbDAeFw0xMzA3MjExNjQ1NDhaFw0xNDA3MjEx
NjQ1NDhaMEExCzAJBgNVBAYTAkdCMQ8wDQYDVQQHDAZMb25kb24xFTATBgNVBAoM
DEFHcmFpbk9mU2FsdDEKMAgGA1UEAwwBKjCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAODnF0NlVXzaVkZMRBEqlou5wNBVENf3x8opOy0dCEYA2yN4WAQ2
NXnKWk+jgTHHTOypB0avYJid/wYcWI8WU5cb8bAXtZpc6+t8xak6k+KLI63ZVB7D
mSyPJB6wC9M6L7NyefVx2TtS3ngYxEGO3FxMltqQdR8h9YORMGQR/q+x4ftNSh8G
pHqwv5G8dFsniOMNLhw8O+hsezKQYMhMKtuE/MJTemxLrkVKhkrGpIgkM6b03otW
O1k/3pLEnfLUK1PaYgWUfrz5+EVTiFYTMGftbOcMMvhTA0k+w8K2BoPg0YBR9Ib3
UrB9BTQ53ztGKWIkQ5q5/qwQMBc645rVq2kCAwEAAaOBszCBsDA1BggrBgEFBQcB
AQQpMCcwJQYIKwYBBQUHMAGGGWh0dHA6Ly8xNjIuMTMuNDcuMTg3Ojg4ODgwCQYD
VR0TBAIwADAsBglghkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlm
aWNhdGUwHQYDVR0OBBYEFOgweUd1jQYMy56ER2Vh1CeNYVLSMB8GA1UdIwQYMBaA
FB/42197/jApb+LiyCPeW3eugi34MA0GCSqGSIb3DQEBBQUAA4IBAQCDLADkWB/b
Aqqo/1JF02NPj0ePZR4hx9fCdt8Dr2TB4CzVkkQ1pMMCeApDD+2RAyr4AFyX9/xv
gWmWPcPOgPLSDd5cLfAnyroctAmKzLZ2Bp+praW/nH2dxfEy1NYwv7xXGXYGUe7o
i/KhTPdp79SWWE1a3pjF9BevtdLMJvBpQ3J3SufPeWK2pEd1SyncL2/mxtUaeeId
v/iCBP7QIXqLTh6TEPSB1p1BmnAC5z8iGtimLiGKsTQDHIPKjBlZG9aF9+vipzLZ
YVzmaLPvuidKP/9abtJgNruiCrqqT9kifqt+eICHbVGSRGHmqmP65ROIyfnekDGf
KHjKeXT/DqQL
-----END CERTIFICATE-----
Signed certificate is in newcert.pem
Once this is completed, ensure that newcert.pem and newkey.pem are copied to the squid server. You will then also need to remove the passphrase from the key.
openssl rsa -in newkey.pem -out sslbump.pem
Once this is done, you’ll need to then also copy the cert into the same file
cat newcert.pem >> sslbump.pem
Configuring Squid #
We’re going to make a very simple squid config, allowing access from the App servers to youtube.com, but no other hosts. Replace /etc/squid/squid.conf with the following
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
acl youtube dstdomain www.youtube.com
acl app_server src 192.168.0.3/32
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
ssl_bump allow app_server youtube
http_access allow app_server youtube
http_access deny all
ssl_bump deny all
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid
# Listen on 3128 and do ssl-bump
http_port 3128 ssl-bump cert=/etc/squid/sslbump.pem
# Don't cache / refresh anything
refresh_pattern . 0 0% 0
Testing Squid #
We’re going to use the curl command to test that the ACLs are working
First lets test google, this should fail. We specify the proxy with the -x flag
[root@box1 squid]# curl -I www.google.com -x 192.168.0.2:3128
HTTP/1.0 403 Forbidden
Server: squid/3.1.10
Mime-Version: 1.0
Date: Sun, 21 Jul 2013 17:23:36 GMT
Content-Type: text/html
Content-Length: 3274
X-Squid-Error: ERR_ACCESS_DENIED 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from box1.agrainofsalt.com
X-Cache-Lookup: NONE from box1.agrainofsalt.com:3128
Via: 1.0 box1.agrainofsalt.com (squid/3.1.10)
Connection: keep-alive
As you can see we get a 403 on this from Squid
Lets now try http access to youtube.com
[root@box1 squid]# curl -I www.youtube.com -x 192.168.0.2:3128
HTTP/1.0 200 OK
Date: Sun, 21 Jul 2013 17:25:37 GMT
Server: gwiseguy/2.0
X-Frame-Options: SAMEORIGIN
X-YouTube-Other-Cookies: VISITOR_INFO1_LIVE=A4yQlrbOatM;PREF=f1=50000000
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/bin/answer.py?answer=151657&hl=en-GB for more info."
X-Content-Type-Options: nosniff
Cache-Control: no-cache
Expires: Tue, 27 Apr 1971 19:44:06 EST
Set-Cookie: YSC=zqSQpd08t-o; path=/; domain=.youtube.com; httponly
Set-Cookie: PREF=f1=50000000; path=/; domain=.youtube.com; expires=Sat, 22-Mar-2014 05:18:37 GMT
Set-Cookie: VISITOR_INFO1_LIVE=A4yQlrbOatM; path=/; domain=.youtube.com; expires=Sat, 22-Mar-2014 05:18:36 GMT
Content-Type: text/html; charset=utf-8
X-XSS-Protection: 1; mode=block
X-Cache: MISS from box1.agrainofsalt.com
X-Cache-Lookup: MISS from box1.agrainofsalt.com:3128
Via: 1.0 box1.agrainofsalt.com (squid/3.1.10)
Connection: keep-alive
This works as expected. Lets try https to youtube.com now!
[root@box1 squid]# curl -I https://www.youtube.com/ -x 127.0.0.1:3128
HTTP/1.0 200 Connection established
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
This has failed as we’ve not got the CA certificate in the bundle that curl uses, lets get curl to ignore the SSL certificate
[root@box1 squid]# curl -Ik https://www.youtube.com/ -x 127.0.0.1:3128
HTTP/1.0 200 Connection established
HTTP/1.0 503 Service Unavailable
Server: squid/3.1.10
Mime-Version: 1.0
Date: Sun, 21 Jul 2013 18:36:00 GMT
Content-Type: text/html
Content-Length: 3408
X-Squid-Error: ERR_CANNOT_FORWARD 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from box1.agrainofsalt.com
X-Cache-Lookup: MISS from box1.agrainofsalt.com:3128
Via: 1.0 box1.agrainofsalt.com (squid/3.1.10)
Connection: keep-alive
Now lets just make sure that other https sites don’t work.
[root@box1 squid]# curl -Ik https://www.gmail.com/ -x 127.0.0.1:3128
HTTP/1.0 403 Forbidden
Server: squid/3.1.10
Mime-Version: 1.0
Date: Sun, 21 Jul 2013 18:36:28 GMT
Content-Type: text/html
Content-Length: 3261
X-Squid-Error: ERR_ACCESS_DENIED 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from box1.agrainofsalt.com
X-Cache-Lookup: NONE from box1.agrainofsalt.com:3128
Via: 1.0 box1.agrainofsalt.com (squid/3.1.10)
Connection: keep-alive
curl: (56) Received HTTP code 403 from proxy after CONNECT
Forwarding all traffic via the Proxy server #
Now the way that this is done depends on the firewall or router in use. What we need to achieve is to either D-NAT or redirect all traffic on port 80 / 443 outbound to the Squid server.
For a Cisco ASA there is a guide on how to do this with WCCP
For a Linux based device you would want to have a IPTables rule similar to
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to 192.168.0.2:3128 -s ! 192.168.0.2
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j DNAT --to 192.168.0.2:3128 -s ! 192.168.0.2