interface directive in /etc/ntp.conf

You can prevent ntpd to listen on 0.0.0.0:123 by setting the interface command. The syntax is:
interface listen IPv4|IPv6|all
interface ignore IPv4|IPv6|all
interface drop IPv4|IPv6|all
The above configures which network addresses ntpd listens or dropped without processing any requests. The ignore prevents opening matching addresses, drop causes ntpd to open the address and drop all received packets without examination. For example to ignore listing on all interfaces, add the following in /etc/ntp.conf:
interface ignore wildcard
To listen to only 127.0.0.1 and 192.168.1.1 addresses:
interface listen 127.0.0.1
interface listen 192.168.1.1
Here is my sample /etc/ntp.conf file from FreeBSD cloud server:
$ egrep -v '^#|$^' /etc/ntp.conf
Sample outputs:

tos minclock 3 maxclock 6
pool 0.freebsd.pool.ntp.org iburst
restrict    default limited kod nomodify notrap noquery nopeer
restrict -6 default limited kod nomodify notrap noquery nopeer
restrict    source  limited kod nomodify notrap noquery
restrict 127.0.0.1
restrict -6 ::1
leapfile "/var/db/ntpd.leap-seconds.list"
interface ignore wildcard
interface listen 172.16.3.1
interface listen 10.105.28.1

Restart ntpd

Reload/restart the ntpd on a FreeBSD unix:
$ sudo /etc/rc.d/ntpd restart
OR use the following command on a Debian/Ubuntu Linux:
$ sudo systemctl restart ntp
OR use the following on a CentOS/RHEL 7/Fedora Linux:
$ sudo systemctl restart ntpd

Verification

Use the netstat command/ss command for verification or to make sure ntpd bind to the specific IP address only:
$ netstat -tulpn | grep :123
OR
$ ss -tulpn | grep :123
Sample outputs:

udp        0      0 10.105.28.1:123         0.0.0.0:*                           -               
udp        0      0 172.16.3.1:123          0.0.0.0:*                           -

Use the sockstat command on a FreeBSD Unix server:
$ sudo sockstat
$ sudo sockstat -4
$ sudo sockstat -4 | grep :123
Sample outputs:

root     ntpd       59914 22 udp4   127.0.0.1:123         *:*
root     ntpd       59914 24 udp4   127.0.1.1:123         *:*