I recently setup a small server which is running Debian 9. The purpose of this machine is to run OpenVPN server on port 443 to bypass censorship. It runs the following services and nothing else:
- Squid on private IP belongs to VPN pool (10.8.0.1:3128)
- SSH on private IP belongs to VPN pool (10.8.0.1:22)
- DNS resolver on private IP belongs to VPN pool (10.8.0.1:53)
- OpneVPN on public IP port 443 (server_public_ip_address:443)
After setting up everything, I decided to run Nmap to scan my server. To my surprise, I discovered that port 80 was open:
{macbookpro}$ sudo nmap ln.vpngatway
Sample outputs:
Starting Nmap 7.50 ( https://nmap.org ) at 2017-07-03 00:24 IST
Nmap scan report for ln.vpngatway (xxx.yyy.zzz.123)
Host is up (0.20s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
80/tcp open http
443/tcp closed https
Nmap done: 1 IP address (1 host up) scanned in 19.87 second
I also ran the following nc command to just make sure port 80 is not responding (but it responded):
{macbookpro}$ nc -zv ln.vpngatway 80
Sample outputs:
found 0 associations
found 1 connections:
1: flags=82
outif en6
src 192.168.2.5 port 51462
dst xxx.yyy.zzz.123 port 80
rank info not available
TCP aux info available
Connection to ln.vpngatway port 80 [tcp/http] succeeded!
The nmap says the port is open while netstat says nothing is running on port 80 on the server itself:
{ln.vpngatway}$ netstat -tulpn | grep :80
I got nothing. So I ran the following:
{ln.vpngatway}$ ps aux | egrep -i 'httpd|nginx|apache|lighttpd'
Sample outputs:
root 4257 0.0 0.0 12784 992 pts/0 S+ 19:06 0:00 grep -E -i httpd|nginx|apache|lighttpd
I am baffled by Nmap output. I double checked everything. Same result. Why port 80 (HTTP) reported as open by Nmap? Before I start playing with tcpdump I decided to do old good telnet/nc session:
{macbookpro}$ nc ln.vpngatway 80
I just requested / document:
GET / HTTP/1.1 host: ln.vpngatway
After some time I got this on screen:
HTTP/1.1 503 Service Unavailable Server: squid Mime-Version: 1.0 Date: Sun, 02 Jul 2017 19:16:52 GMT Content-Type: text/html;charset=utf-8 Content-Length: 3406 X-Squid-Error: ERR_CONNECT_FAIL 60 Vary: Accept-Language Content-Language: en X-Cache: MISS from localhost X-Cache-Lookup: MISS from localhost:3128 Connection: keep-alive <html><head> <meta type="copyright" content="Copyright (C) 1996-2017 The Squid Software Foundation and contributors"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <title>ERROR: The requested URL could not be retrieved</title> <style type="text/css"><!-- .... ...... .. <script language="javascript"> if ('[unknown method]' != '[unknown method]') document.getElementById('missing-method').style.display = 'none'; if ('error:invalid-request' != '[no URL]') document.getElementById('missing-url').style.display = 'none'; if ('[unknown protocol]' != '[unknown protocol]') document.getElementById('missing-protocol').style.display = 'none'; </script> <hr> <div id="footer"> <p>Generated Sun, 02 Jul 2017 19:16:52 GMT by localhost (squid)</p> <!-- ERR_INVALID_REQ --> </div> </body></html>
The mystery is solved. I found Interception Caching/Transparent Proxying. Interception Caching is nothing but the process by which HTTP connections coming from remote clients are redirected to a cache server, without their knowledge or explicit configuration. In this case, all HTTP connections were redirected to a proxy server by default using combination of a firewall and squid server.
🐧 4 comments so far... add one ↓
| Category | List of Unix and Linux commands |
|---|---|
| Disk space analyzers | df • duf • ncdu • pydf |
| File Management | cat • cp • mkdir • tree |
| Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
| Modern utilities | bat • exa |
| Network Utilities | NetHogs • dig • host • ip • nmap |
| OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
| Package Manager | apk • apt |
| Processes Management | bg • chroot • cron • disown • fg • glances • gtop • jobs • killall • kill • pidof • pstree • pwdx • time • vtop |
| Searching | ag • grep • whereis • which |
| User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
| WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
How did this even happen? Shouldn’t DNS always point to the same ip regardless? Are you somehow a victim of DNS cache poisoning?
Normally that’d be a misconfiguration.
Sweet find! Excellent troubleshooting to find a resolution to an obfuscation problem.
Thank you for sharing your findings. This will surly help me and many others when they encounter a similar situation.
Interesting find. How could one find that proxy? I presume when you unplug the network and run a local test, the result would be as expected?