MikroTik RouterOS WEB Proxy
Overview
The MikroTik RouterOS has the squid proxy server implementation.
Proxy server features:
* Regular http proxy.
* Transparent proxy. Can be transparent and regular at the same time.
* Access list by source, destination, and URL.
* Cache access list.
Contents of the Manual
The following topics are covered in this manual:
* Installation
* Hardware Resource Usage
* MikroTik Web Proxy Description
* MikroTik Web Proxy Setup
Monitoring the Web Proxy
Access List
Managing the Cache
* Transparent Mode
* Troubleshooting
Installation
The MikroTik Web Proxy feature is included in the 'web-proxy' package. To install the web-proxy package, upload it to the router and reboot. After successful install of the web-proxy package it should be listed under the /system package print list:
[MikroTik] > system package print
Flags: I - invalid
# NAME VERSION BUILD-TIME UNINSTALL
0 system 2.5.2 apr/24/2002 11:52:28 no
1 routing 2.5.2 apr/24/2002 12:04:34 no
2 web-proxy 2.5.2 apr/24/2002 12:02:52 no
3 ppp 2.5.2 apr/24/2002 11:57:03 no
4 pptp 2.5.2 apr/24/2002 11:57:43 no
5 pppoe 2.5.2 apr/24/2002 11:58:46 no
6 ssh 2.5.2 apr/24/2002 11:54:52 no
7 snmp 2.5.2 apr/24/2002 11:53:10 no
[MikroTik] >
Hardware Resource Usage
The proxy cache can use as much disk space as there is allocated for it. When the system allocates the space for the proxy cache, 1/7th of the total partition (disk) size is reserved for the system, but not less than 30MB. The rest is left for the proxy cache. The system RAM size is considered as well when allocating the cache size. The cache size is limited so, that there are at least 11.1MB of RAM per 1GB of cache.
It is recommended, that at least 100MB HDD is used when running web proxy. Do not try to run web-proxy on a 32 or 48 MB FlashDisk!
MikroTik Web Proxy Description
The web proxy can be used as transparent and normal web proxy at the same time. In transparent mode it is possible to use it as standard web proxy, too. However, in this case, proxy users may have trouble to reach web pages which are accessed transparently.
MikroTik Web Proxy Setup
The Web Proxy management can be accessed under the /ip web-proxy submenu:
[MikroTik] ip web-proxy> ?
HTTP proxy
clear-cache Clear http cache
access Access list
cache Cache access list
print Print current configuration and status
get Get value of configuration property
set Change proxy configuration
export Export web proxy settings
[MikroTik] ip web-proxy>
For web proxy setup, do the following:
* Specify at least one dns server for the router:
/ip dns set primary-dns=192.168.1.1
* Set IP address and port on which proxy will listen for requests:
/ip web-proxy set address=0.0.0.0:8080
* If this proxy has to use another proxy, specify it:
/ip web-proxy set parent-proxy=192.168.1.1:8080
otherwise disable it:
/ip web-proxy set parent-proxy=0.0.0.0:0
* Specify cache administrator's e-mail address:
/ip web-proxy set cache-administrator=support@mt.lv
* Specify hostname (DNS or IP address) of the web proxy:
/ip web-proxy set hostname=proxy.mt.lv
* Enable the proxy service:
/ip web-proxy set enabled=yes
Now it is possible to use this proxy, by setting it as proxy for IE or Netscape.
Web proxy will automatically detect any problems with cache and will try to solve them without loosing any cache data. But in case of a heavy damage to the file system, the web proxy can't rebuild cache data. Cache can be deleted and new cache directories created by the command '/ip web-proxy clear-cache'.
Monitoring the Web Proxy
Use the command /ip web-proxy print to see the current web proxy status:
[MikroTik] ip web-proxy> print
enabled: yes
address: 0.0.0.0:8080
transparent-mode: no
parent-proxy: 0.0.0.0:0
cache-administrator: support@mt.lv
hostname: proxy.mt.lv
status: running
reserved-for-cache: 6782 MB
[MikroTik] ip web-proxy>
Description of the status parameter value:
stopped - proxy is disabled and is not running
rebuilding-cache - proxy is enabled and running, existing cache is being verified
running - proxy is enabled and running
stopping - proxy is shutting down (max 10s)
clearing-cache - proxy is stopped, cache files are being removed
creating-cache - proxy is stopped, cache directory structure is being created
dns-missing - proxy is enabled, but not running because of unknown DNS server (please, specify it under /ip dns)
invalid-address - proxy is enabled, but not running because of invalid address (please, change address or port)
invalid-cache-administrator - proxy is enabled, but not running because of invalid cache-administrator's e-mail address
invalid-hostname - proxy is enabled, but not running because of invalid hostname (please, set valid hostname value)
error-logged - proxy is not running because of unknown error. This error is logged as System-Error. Please, send us this error and some description, how it happened.
Access logs are sent to Web-Proxy-Access logging facility. These logs can be disabled, logged locally or sent to remote address. To log locally:
/system logging facility set Web-Proxy-Access logging=local
Logs can be viewed using /log print command.
Access List
Access list is implemented in the same way as MikroTik firewall rules. Rules are processed from top to bottom. First matching rule specifies decision of what to do with this connection. Connections can be matched by its source address, destination address, destination port or substring of requested url. If none of these parameters is specified, every connection will match this rule.
If connection is matched by a rule, action property of this rule specifies whether connection will be allowed or not. If connection does not match any rule, it will be allowed.
For example:
[MikroTik] ip web-proxy access> print
Flags: X - disabled
# SRC-ADDRESS DST-ADDRESS DST-PORT URL ACTION
0 0.0.0.0/0 0.0.0.0/0 0-65535 .mp3 deny
1 10.0.0.1/32 0.0.0.0/0 0-65535 allow
2 0.0.0.0/0 0.0.0.0/0 0-65535 ftp:// deny
3 10.0.0.0/24 10.9.9.128/28 0-65535 allow
4 0.0.0.0/0 0.0.0.0/0 0-65535 deny
[MikroTik] ip web-proxy access>
Argument description:
src-address - source address of the request
dst-address - destination address of the request
dst-port - destination port of the request
url - the URL of the request. Can be regular expression.
action - (allow / deny) action to take.
Access list, shown above, disables access to any mp3 files for everyone.
Local gateway 10.0.0.1 has access to everything else (excluding mp3 files).
All other local network (10.0.0.0/24) users have access to servers located at 10.9.9.128/28, but, ftp protocol is not allowed for them.
Any other request is denied.
Managing the Cache
Cache access list specifies, which requests (domains, servers, pages) have to be cached locally by web proxy, and which not. The Web Proxy cache access list is located under the /ip web-proxy cache submenu.
Access list is implemented exactly the same way as web proxy access list. Default action is to cache object (if no matching rule is found). By default, one cache access rule is already added:
[MikroTik] ip web-proxy cache> print
Flags: X - disabled
# SRC-ADDRESS DST-ADDRESS DST-PORT URL ACTION
0 0.0.0.0/0 0.0.0.0/0 0-65535 cgi-bin \? deny
[MikroTik] ip web-proxy cache>
This rule defines, that all runtime generated pages (which are located within cgi-bin directories or contain '?' in url) has not to be cached.
NOTE: Objects, which are larger than 4MB, are not cached.
Transparent Mode
To enable the transparent mode, firewall rule in destination nat has to be added, specifying which connections (to which ports) should be transparently redirected to the proxy. For example, we have the following web-proxy settings:
[MikroTik] ip web-proxy> print
enabled: yes
address: 0.0.0.0:8080
transparent-mode: yes
parent-proxy: 0.0.0.0:0
cache-administrator: support@mt.lv
hostname: proxy.mt.lv
status: running
reserved-for-cache: 3398 MB
[MikroTik] ip web-proxy>
If we want all connections coming from interface ether1 and going to port 80 to handle with web proxy transparently, and if our web proxy is listening on port 8080, then we add following destination nat rule:
[MikroTik] ip firewall dst-nat> add in-interface=ether1 protocol=tcp \
dst-address=!10.0.0.1/32:80 action=redirect to-dst-port=8080
[MikroTik] ip firewall dst-nat> print
Flags: X - disabled, I - invalid
0 ;;; Transparent proxy
src-address=0.0.0.0/0:0-65535 in-interface=ether1
dst-address=!10.0.0.1/32:80 protocol=tcp icmp-options=any:any flow=""
src-mac-address=00:00:00:00:00:00 limit-count=0 limit-burst=0
limit-time=0s action=redirect to-dst-address=0.0.0.0 to-dst-port=8080
bytes=118949 packets=2260
[MikroTik] ip firewall dst-nat>
Here, the router's address and port 80 (10.0.0.1/32:80) have been excluded from redirection to preserve the winbox functionality which uses TCP port 80 on the router. More than one redirect rule can be added to redirect more than one port.
NOTE: only HTTP traffic is supported by web proxy transparent mode. HTTPS and FTP are not going to work this way!
Troubleshooting
* My web-proxy does not start. There are error messages in the system log, and the status of the web-proxy is 'rebuilding cache ...'
Problem with underscore '_' in the identity name has been fixed (starting with v2.5.2). It is a good idea, to update web-proxy, if there is newer version available.
* The transparent proxy stops working after 5 minutes
See if you have enough disk space (do not run web-proxy on a 48MB FlashDisk!) and sufficient RAM (10MB...20MB on each GB of proxy cache).
* Can I use transparent proxy feature on a MikroTik router with bridged interfaces?
No. Transparent proxy requires redirection of IP packets by firewall destination NAT. Firewall is not involved when packets are passed from one bridged interface to another. But packets have to be translated by firewall destination NAT for transparent web-proxy to work. So, web-proxy is not going to work in transparent mode between bridge interfaces.
* When I turned on transparent proxy and redirected TCP port 80 to it, my WinBox stopped working.
TCP port 80 is used by WinBox when connecting to the router. You should exclude the router's address:80 from redirection by using rule
'/ip firewall src-nat add dst-address=address/32:80 protocol=tcp action=accept'
BEFORE the redirect rule. Alternatively, you can use just one rule
'/ip firewall src-nat add dst-address=!address/32:80 protocol=tcp action=redirect to-dst-port=8080'
* I use firewall to block access to the router from the Internet. My proxy does not work.
Make sure you allow established TCP connections with tcp option 'non-syn-only' to the router before blocking everything else. In v2.5, the rule is like this:
'/ip firewall rule input add protocol=tcp tcp-options=non-syn-only connection-state=established'
The MikroTik RouterOS has the squid proxy server implementation.
Proxy server features:
* Regular http proxy.
* Transparent proxy. Can be transparent and regular at the same time.
* Access list by source, destination, and URL.
* Cache access list.
Contents of the Manual
The following topics are covered in this manual:
* Installation
* Hardware Resource Usage
* MikroTik Web Proxy Description
* MikroTik Web Proxy Setup
Monitoring the Web Proxy
Access List
Managing the Cache
* Transparent Mode
* Troubleshooting
Installation
The MikroTik Web Proxy feature is included in the 'web-proxy' package. To install the web-proxy package, upload it to the router and reboot. After successful install of the web-proxy package it should be listed under the /system package print list:
[MikroTik] > system package print
Flags: I - invalid
# NAME VERSION BUILD-TIME UNINSTALL
0 system 2.5.2 apr/24/2002 11:52:28 no
1 routing 2.5.2 apr/24/2002 12:04:34 no
2 web-proxy 2.5.2 apr/24/2002 12:02:52 no
3 ppp 2.5.2 apr/24/2002 11:57:03 no
4 pptp 2.5.2 apr/24/2002 11:57:43 no
5 pppoe 2.5.2 apr/24/2002 11:58:46 no
6 ssh 2.5.2 apr/24/2002 11:54:52 no
7 snmp 2.5.2 apr/24/2002 11:53:10 no
[MikroTik] >
Hardware Resource Usage
The proxy cache can use as much disk space as there is allocated for it. When the system allocates the space for the proxy cache, 1/7th of the total partition (disk) size is reserved for the system, but not less than 30MB. The rest is left for the proxy cache. The system RAM size is considered as well when allocating the cache size. The cache size is limited so, that there are at least 11.1MB of RAM per 1GB of cache.
It is recommended, that at least 100MB HDD is used when running web proxy. Do not try to run web-proxy on a 32 or 48 MB FlashDisk!
MikroTik Web Proxy Description
The web proxy can be used as transparent and normal web proxy at the same time. In transparent mode it is possible to use it as standard web proxy, too. However, in this case, proxy users may have trouble to reach web pages which are accessed transparently.
MikroTik Web Proxy Setup
The Web Proxy management can be accessed under the /ip web-proxy submenu:
[MikroTik] ip web-proxy> ?
HTTP proxy
clear-cache Clear http cache
access Access list
cache Cache access list
print Print current configuration and status
get Get value of configuration property
set Change proxy configuration
export Export web proxy settings
[MikroTik] ip web-proxy>
For web proxy setup, do the following:
* Specify at least one dns server for the router:
/ip dns set primary-dns=192.168.1.1
* Set IP address and port on which proxy will listen for requests:
/ip web-proxy set address=0.0.0.0:8080
* If this proxy has to use another proxy, specify it:
/ip web-proxy set parent-proxy=192.168.1.1:8080
otherwise disable it:
/ip web-proxy set parent-proxy=0.0.0.0:0
* Specify cache administrator's e-mail address:
/ip web-proxy set cache-administrator=support@mt.lv
* Specify hostname (DNS or IP address) of the web proxy:
/ip web-proxy set hostname=proxy.mt.lv
* Enable the proxy service:
/ip web-proxy set enabled=yes
Now it is possible to use this proxy, by setting it as proxy for IE or Netscape.
Web proxy will automatically detect any problems with cache and will try to solve them without loosing any cache data. But in case of a heavy damage to the file system, the web proxy can't rebuild cache data. Cache can be deleted and new cache directories created by the command '/ip web-proxy clear-cache'.
Monitoring the Web Proxy
Use the command /ip web-proxy print to see the current web proxy status:
[MikroTik] ip web-proxy> print
enabled: yes
address: 0.0.0.0:8080
transparent-mode: no
parent-proxy: 0.0.0.0:0
cache-administrator: support@mt.lv
hostname: proxy.mt.lv
status: running
reserved-for-cache: 6782 MB
[MikroTik] ip web-proxy>
Description of the status parameter value:
stopped - proxy is disabled and is not running
rebuilding-cache - proxy is enabled and running, existing cache is being verified
running - proxy is enabled and running
stopping - proxy is shutting down (max 10s)
clearing-cache - proxy is stopped, cache files are being removed
creating-cache - proxy is stopped, cache directory structure is being created
dns-missing - proxy is enabled, but not running because of unknown DNS server (please, specify it under /ip dns)
invalid-address - proxy is enabled, but not running because of invalid address (please, change address or port)
invalid-cache-administrator - proxy is enabled, but not running because of invalid cache-administrator's e-mail address
invalid-hostname - proxy is enabled, but not running because of invalid hostname (please, set valid hostname value)
error-logged - proxy is not running because of unknown error. This error is logged as System-Error. Please, send us this error and some description, how it happened.
Access logs are sent to Web-Proxy-Access logging facility. These logs can be disabled, logged locally or sent to remote address. To log locally:
/system logging facility set Web-Proxy-Access logging=local
Logs can be viewed using /log print command.
Access List
Access list is implemented in the same way as MikroTik firewall rules. Rules are processed from top to bottom. First matching rule specifies decision of what to do with this connection. Connections can be matched by its source address, destination address, destination port or substring of requested url. If none of these parameters is specified, every connection will match this rule.
If connection is matched by a rule, action property of this rule specifies whether connection will be allowed or not. If connection does not match any rule, it will be allowed.
For example:
[MikroTik] ip web-proxy access> print
Flags: X - disabled
# SRC-ADDRESS DST-ADDRESS DST-PORT URL ACTION
0 0.0.0.0/0 0.0.0.0/0 0-65535 .mp3 deny
1 10.0.0.1/32 0.0.0.0/0 0-65535 allow
2 0.0.0.0/0 0.0.0.0/0 0-65535 ftp:// deny
3 10.0.0.0/24 10.9.9.128/28 0-65535 allow
4 0.0.0.0/0 0.0.0.0/0 0-65535 deny
[MikroTik] ip web-proxy access>
Argument description:
src-address - source address of the request
dst-address - destination address of the request
dst-port - destination port of the request
url - the URL of the request. Can be regular expression.
action - (allow / deny) action to take.
Access list, shown above, disables access to any mp3 files for everyone.
Local gateway 10.0.0.1 has access to everything else (excluding mp3 files).
All other local network (10.0.0.0/24) users have access to servers located at 10.9.9.128/28, but, ftp protocol is not allowed for them.
Any other request is denied.
Managing the Cache
Cache access list specifies, which requests (domains, servers, pages) have to be cached locally by web proxy, and which not. The Web Proxy cache access list is located under the /ip web-proxy cache submenu.
Access list is implemented exactly the same way as web proxy access list. Default action is to cache object (if no matching rule is found). By default, one cache access rule is already added:
[MikroTik] ip web-proxy cache> print
Flags: X - disabled
# SRC-ADDRESS DST-ADDRESS DST-PORT URL ACTION
0 0.0.0.0/0 0.0.0.0/0 0-65535 cgi-bin \? deny
[MikroTik] ip web-proxy cache>
This rule defines, that all runtime generated pages (which are located within cgi-bin directories or contain '?' in url) has not to be cached.
NOTE: Objects, which are larger than 4MB, are not cached.
Transparent Mode
To enable the transparent mode, firewall rule in destination nat has to be added, specifying which connections (to which ports) should be transparently redirected to the proxy. For example, we have the following web-proxy settings:
[MikroTik] ip web-proxy> print
enabled: yes
address: 0.0.0.0:8080
transparent-mode: yes
parent-proxy: 0.0.0.0:0
cache-administrator: support@mt.lv
hostname: proxy.mt.lv
status: running
reserved-for-cache: 3398 MB
[MikroTik] ip web-proxy>
If we want all connections coming from interface ether1 and going to port 80 to handle with web proxy transparently, and if our web proxy is listening on port 8080, then we add following destination nat rule:
[MikroTik] ip firewall dst-nat> add in-interface=ether1 protocol=tcp \
dst-address=!10.0.0.1/32:80 action=redirect to-dst-port=8080
[MikroTik] ip firewall dst-nat> print
Flags: X - disabled, I - invalid
0 ;;; Transparent proxy
src-address=0.0.0.0/0:0-65535 in-interface=ether1
dst-address=!10.0.0.1/32:80 protocol=tcp icmp-options=any:any flow=""
src-mac-address=00:00:00:00:00:00 limit-count=0 limit-burst=0
limit-time=0s action=redirect to-dst-address=0.0.0.0 to-dst-port=8080
bytes=118949 packets=2260
[MikroTik] ip firewall dst-nat>
Here, the router's address and port 80 (10.0.0.1/32:80) have been excluded from redirection to preserve the winbox functionality which uses TCP port 80 on the router. More than one redirect rule can be added to redirect more than one port.
NOTE: only HTTP traffic is supported by web proxy transparent mode. HTTPS and FTP are not going to work this way!
Troubleshooting
* My web-proxy does not start. There are error messages in the system log, and the status of the web-proxy is 'rebuilding cache ...'
Problem with underscore '_' in the identity name has been fixed (starting with v2.5.2). It is a good idea, to update web-proxy, if there is newer version available.
* The transparent proxy stops working after 5 minutes
See if you have enough disk space (do not run web-proxy on a 48MB FlashDisk!) and sufficient RAM (10MB...20MB on each GB of proxy cache).
* Can I use transparent proxy feature on a MikroTik router with bridged interfaces?
No. Transparent proxy requires redirection of IP packets by firewall destination NAT. Firewall is not involved when packets are passed from one bridged interface to another. But packets have to be translated by firewall destination NAT for transparent web-proxy to work. So, web-proxy is not going to work in transparent mode between bridge interfaces.
* When I turned on transparent proxy and redirected TCP port 80 to it, my WinBox stopped working.
TCP port 80 is used by WinBox when connecting to the router. You should exclude the router's address:80 from redirection by using rule
'/ip firewall src-nat add dst-address=address/32:80 protocol=tcp action=accept'
BEFORE the redirect rule. Alternatively, you can use just one rule
'/ip firewall src-nat add dst-address=!address/32:80 protocol=tcp action=redirect to-dst-port=8080'
* I use firewall to block access to the router from the Internet. My proxy does not work.
Make sure you allow established TCP connections with tcp option 'non-syn-only' to the router before blocking everything else. In v2.5, the rule is like this:
'/ip firewall rule input add protocol=tcp tcp-options=non-syn-only connection-state=established'