Why port 80 (HTTP) reported as open by nmap when it is closed?

Why are some ports reported as open by nmap?
I recently setup a small server which is running Debian 9. The purpose of this machine is to run OpenVPN server on port 443 to bypass censorship. It runs the following services and nothing else:

ADVERTISEMENTS

  1. Squid on private IP belongs to VPN pool (10.8.0.1:3128)
  2. SSH on private IP belongs to VPN pool (10.8.0.1:22)
  3. DNS resolver on private IP belongs to VPN pool (10.8.0.1:53)
  4. OpneVPN on public IP port 443 (server_public_ip_address:443)

After setting up everything, I decided to run Nmap to scan my server. To my surprise, I discovered that port 80 was open:
{macbookpro}$ sudo nmap ln.vpngatway
Sample outputs:

Starting Nmap 7.50 ( https://nmap.org ) at 2017-07-03 00:24 IST
Nmap scan report for ln.vpngatway (xxx.yyy.zzz.123)
Host is up (0.20s latency).
Not shown: 998 filtered ports
PORT    STATE  SERVICE
80/tcp  open   http
443/tcp closed https

Nmap done: 1 IP address (1 host up) scanned in 19.87 second

I also ran the following nc command to just make sure port 80 is not responding (but it responded):
{macbookpro}$ nc -zv ln.vpngatway 80
Sample outputs:

found 0 associations
found 1 connections:
     1:	flags=82
	outif en6
	src 192.168.2.5 port 51462
	dst xxx.yyy.zzz.123 port 80
	rank info not available
	TCP aux info available

Connection to ln.vpngatway port 80 [tcp/http] succeeded!

The nmap says the port is open while netstat says nothing is running on port 80 on the server itself:
{ln.vpngatway}$ netstat -tulpn | grep :80
I got nothing. So I ran the following:
{ln.vpngatway}$ ps aux | egrep -i 'httpd|nginx|apache|lighttpd'
Sample outputs:

root      4257  0.0  0.0  12784   992 pts/0    S+   19:06   0:00 grep -E -i httpd|nginx|apache|lighttpd

I am baffled by Nmap output. I double checked everything. Same result. Why port 80 (HTTP) reported as open by Nmap? Before I start playing with tcpdump I decided to do old good telnet/nc session:
{macbookpro}$ nc ln.vpngatway 80
I just requested / document:

GET / HTTP/1.1
host: ln.vpngatway

After some time I got this on screen:

HTTP/1.1 503 Service Unavailable
Server: squid
Mime-Version: 1.0
Date: Sun, 02 Jul 2017 19:16:52 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3406
X-Squid-Error: ERR_CONNECT_FAIL 60
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Connection: keep-alive
 
 
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2017 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!-- 
....
......
..
<script language="javascript">
if ('[unknown method]' != '[unknown method]') document.getElementById('missing-method').style.display = 'none';
if ('error:invalid-request' != '[no URL]') document.getElementById('missing-url').style.display = 'none';
if ('[unknown protocol]' != '[unknown protocol]') document.getElementById('missing-protocol').style.display = 'none';
</script>
 
<hr>
<div id="footer">
<p>Generated Sun, 02 Jul 2017 19:16:52 GMT by localhost (squid)</p>
<!-- ERR_INVALID_REQ -->
</div>
</body></html>

The mystery is solved. I found Interception Caching/Transparent Proxying. Interception Caching is nothing but the process by which HTTP connections coming from remote clients are redirected to a cache server, without their knowledge or explicit configuration. In this case, all HTTP connections were redirected to a proxy server by default using combination of a firewall and squid server.

🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source & DevOps topics via:
CategoryList of Unix and Linux commands
File Managementcat
FirewallCentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNCentOS 8 Debian 10 Firewall Ubuntu 20.04

ADVERTISEMENTS
4 comments… add one
  • Jose Jul 3, 2017 @ 7:55

    How did this even happen? Shouldn’t DNS always point to the same ip regardless? Are you somehow a victim of DNS cache poisoning?

  • Jouni "rautamiekka" Järvinen Jul 3, 2017 @ 12:05

    Normally that’d be a misconfiguration.

  • Christian Jul 3, 2017 @ 16:08

    Sweet find! Excellent troubleshooting to find a resolution to an obfuscation problem.

    Thank you for sharing your findings. This will surly help me and many others when they encounter a similar situation.

  • Stephan Wissel Sep 16, 2017 @ 4:54

    Interesting find. How could one find that proxy? I presume when you unplug the network and run a local test, the result would be as expected?

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.