Personal tools

Web Filter

From UntangleWiki

Jump to: navigation, search
Image:WebFilter_128x128.png     Web Filter
Other Links:
Web Filter Description Page
Web Filter Video Demo
Web Filter Screenshots
Web Filter Forums
Web Filter FAQs




Contents

About Web Filter

Web Filter monitors HTTP traffic on your network to monitor user behavior and block inappropriate content. It offers everything that Web Filter Lite does and more. Web Filter appeals to customers who require an added level of protection or are subject to regulations, for example Web Filter helps libraries comply with the Children's Internet Protection Act). Need to block Pornography or Hate Speech on your network? Web Filter is your answer.


Web Filter improves upon Web Filter Lite in the following areas:

  • Real-time classification and updates: When your users visit a site, Untangle sends the URL to the cloud to be categorized. When the data is returned, Untangle keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to block or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at zVelo and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or blocked according to your settings without additional intervention, plus you have the option of requesting recategorization of sites.
  • HTTPS Filtering: Web Filter has multiple techniques to deal with HTTPS filtering. HTTPS traffic is encryted so only some information is visible and this information is used to categorize the session. More information on how this is down below.
  • Detailed categorization: Web Filter Lite does a good job categorizing, but Web Filter offers over 140 categories and over 450 million categorized sites. The abundance of categories means that you can narrow your scope - maybe you want to block websites related to Sex, but allow sites dealing with Sexual Education or Pregnancy. With Web Filter Lite you would have to block 'Pornography' category and allow any specifics sites through, while with Web Filter you could set the separate categories appropriately and not worry about manually unblocking anything.
  • Additional features: Youtube for Schools support, the ability to force SafeSearch on search engines that support it, and more!


Settings

This section reviews the different settings and configuration options available for Web Filter.


Block Lists

Block Lists are used to block content that you do not want users to have access to.


  • Categories: The Categories section allows you to customize which categories of sites will be blocked or flagged. Categories that are blocked will display a block page to the user; categories that are flagged will allow the user to access the site, but will be silently logged to the Violation Events tab of Reports. These block/flag actions operate the same way for all of the different Web Filter options.


  • Categorize HTTPS traffic by IP address: If this option is enabled, HTTPS traffic will be categorized using the IP address when the session is initiated. More details in #HTTPS Details.


  • Categorize HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in #HTTPS Details.


  • Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.


  • Blocked Sites: In the Sites section you can add individual domain names you want to be blocked or flagged - just enter the domain name (e.g. youtube.com) and specify your chosen action. More specific information on how matching works is available on our Rule Syntax page.


A few sites entered into the Block List


  • File Types: The File Types section allows you to block files by file extension - just select (or add) your chosen file extension, check your preferred action, and save.


The File Types Block List


  • MIME Types: The MIME Types section allows you to block files by MIME types - just select (or add) your chosen file extension, check your preferred action, and save.


The MIME Types Block List


  • Enforce safe search on popular search engines: When this option is enabled, safe search will be enforced on all searches using supported search engines (Google, Yahoo, etc).


  • Block pages from IP only hosts: When this option is enabled, users entering an IP address rather than domain name will be blocked.


  • Youtube for Schools: If enabled, this option will inject your youtube indentifier into all youtube web traffic so youtube will enforce the appropriate policy for computers on your network.
After creating an account a unique identifier will be supplied (Example: Jvagw05BzSxAntTLKwUw1w). Take the supplied youtube identifier and save it in your settings and Web Filter will rewrite all youtube URLs with this identifier.
After doing this you will need to configure your desired settings on the Youtube for Schools configuration page under "Account->Settings->Manage School" on youtube.com. This pages allows you to create a list of blessed videos and blessed "teacher accounts" and more to enforce proper youtube usage on your network.
If this is configured it may also be necessary to block all HTTPS youtube traffic using Application Control to prevent access to youtube from an encrypted channel where the URL can not be rewritten.
You can read more about how to setup Youtube for Schools here.


  • Unblock: This section can be used to add a button to allow users to bypass restrictions on a case-by-case basis.
If Unblock is set to None no users will be allowed to bypass the block page. If Unblock is set to Temporary users will be allowed to visit the site for one hour from the time it is unblocked. If Unblock is set to Permanent and Global then users will be allowed to visit the site and unblocked sites will be added to the permanent global pass list so it will always be allowed in the future.
You also have the option of setting a password to Unblock; it can either be the existing Administrator password for the Untangle or you can set a new, separate password only for the Unblock feature.




Pass Lists

Pass Lists are used to pass content that would have otherwise been blocked. This can be useful for "unblocking" sites that you don't want blocked or allowing certain users special privileges.


  • Passed Sites: Any domains you add to the Passed Sites list will be allowed, even if blocked by category or by individual URL - just add the domain and save. Unchecking the pass option will allow the site to be blocked as if the entry was not present.


A few sites entered into the Pass List


  • Passed Client IPs: If you add an IP to this list, Web Filter will not block any traffic from that IP regardless of the blocked categories or sites. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list.
If you have a few users that need to completely bypass Web Filter controls, consider using pass lists. If you have users that simply need different Web Filter settings, you should set up a separate rack using Policy Manager. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.


A few different entries in the Pass Listed Client IPs list


Event Log

Use the following terms and definitions to understand the Event Log:


Name Description
Timestamp The time the event took place.
Client The IP address of the client that made the request.
Username The username of the client that made the request, if available.
Host The Host portion of the request.
URI The URI portion of the request.
Blocked True is the site was blocked, false if it was not.
Flagged True if the site was flagged, false if it was not.
Reason For Action The reason the action was taken.
Category The category of the site visited.
Server The IP address of the server that received the request.

HTTPS Details

As described briefly above, there are two HTTPS processing engines.

  • Categorize HTTPS traffic by IP Address
  • Categorize HTTPS traffic by SNI (Server Name Indication) if present.

The two engines operate independently.

The IP-address categorization happens first at session initiation time. The IP-address categorization attempts to categorize the site being visited by the unencryped server IP-address of the HTTPS session. For example, for HTTPS traffic to 1.2.3.4 it will attempt to categorize "https://1.2.3.4". If a category is successfully determined and that category is blocked according to the settings, the session is reset and no more processing of this session will be done. If the IP-based categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on its IP is logged ("https://1.2.3.4").

The SNI categorization happens later when the data is sent to the server. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication." This is a recent standard added to HTTPS to make virtual hosting easier on the server-side. Web Filter will parse the data looking for the SNI information in the clear. If found it will use the hostname to categorize the session. If the SNI-based categorization determines the page should be blocked the session is reset. If the SNI-based categorization deteremines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on the SNI information is logged ("https://example.com/").

If both engines are enabled you may see two HTTPS events per HTTPS web hit, one using the IP and one using the SNI information.

Note: When blocking HTTPS traffic, block pages can not be shown. The HTTPS encryption prevents man-in-the-middle spoofing of data required to display the block page. The connection will simply be reset and the browser will display an error.

Note: Neither HTTPS engine (IP-based nor SNI-based) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating block/pass rules.

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Related Topics


Web Filter FAQs

How do Web Filter and Web Filter Lite work?

Web Filter and Web Filter Lite both transparently scans HTTP traffic in order to block or log specific activity. Websites can be blocked or logged based on Category (Pornography, Social Networking, etc.), URL (facebook.com, youtube.com, etc.), MIME Type or File Type (.exe, .mp3, etc..). Web Filter has a more robust feature set that is explained in a FAQ entry below.


Can I use both Web Filter and Web Filter Lite?

We do not recommend running both Web Filter and Web Filter Lite at the same time - if you have access to the trial or full version of Web Filter, we recommend using it rather than Web Filter Lite.


Is Web Filter really better than Web Filter Lite?

Web Filter is the same as Web Filter Lite except it is based on SiteFilter technology. Web Filter is better than Web Filter Lite in many ways:

  • HTTPS Filtering
  • Many More categories (141 vs. 15)
  • Larger database (450+ million URLs vs ~1 million)
  • Dynamic categorization of new sites
  • Youtube for Schools support
  • SafeSearch enforcement
  • Password option for the Unblock feature


Can I install Web Filter/Web Filter Lite on a single computer to use as Parental Control software?

No - Untangle is designed to operate as a gateway or transparent bridge for an entire network and is not meant to filter the computer it is installed on. Installing Untangle will wipe out all existing data on the PC it is installed to. For filtering a single PC, other Internet filter/Parental Control software can be used.


Why is a site not being properly displayed even though I added it to the Pass List?

It's common for a web site to display links, banners and content from other web sites as part of their web pages. There are two easy methods to re-integrate the content while maintaining your access controls. A good example is Facebook - when you go to 'facebook.com', much of the site is loaded from 'fbcdn.net', which also must be put on the pass list for it to display properly. To fix these problems, just look in the Event Log for domains that are still being blocked when you load the site.


Can I block all web sites except certain ones?

Yes, simply block all categories (including "Uncategorized"). Then add whatever sites you'd like to pass to the Pass List. Please be aware that the complex nature of the web and the fact that many applications communicate over HTTP can make this approach difficult.


Why block both MIME Type and File Types?

In an ideal world, both pieces of information would always be present for every web request. However, some sites use incorrect content types or extensions. The behavior of operating systems (Windows vs. Mac) is also different when given only file extension or content type. To be safe, both lists should be used.


What kind of reporting features do Web Filter and Web Filter Lite offer?

Web Filter and Web Filter Lite provide network- and user-based reporting. Data from these apps is fed into Reports to show high level trends such as peak network usage hours as well as allowing administrators to drill down to the individual user level for activity monitoring.


Can I grant privileged access to some users while still blocking sites for everyone else?

There are several ways to accomplish this:

  • Policy Manager can be used to create multiple racks, which allows you to have separate filtering settings for individuals or groups of users. The easiest example is a school, where you would want Teachers to have more relaxed internet filter settings than the students. Different settings can be applied to any individual or group in your organization such as CEOs, Administrative Assistants or Accounting Departments.
  • The Passed Client IPs List allows you to exempt specific users from all filtering inside the Web Filter/Web Filter Lite applications.
  • The Unblock option displays a button that, when clicked, will allow users to bypass the block page. Web Filter has an additional option to require a password for this.


Can I let users access certain sites during lunch?

You can leverage Policy Manager to set up specific filtering settings for different days or time periods, such as allowing Facebook during breaks or after work hours.

How do I submit a mis-categorized or uncategorized site?

You can go to zvelo and submit the correct (or new) categorization. It will be reviewed immediately by a human. Once the new categorization takes effect you may need to flush your category cache in Web Filter to see the new categorization.


Does Web Filter use a lot of memory and CPU?

If your Untangle Server is operating well without Web Filter, then you won't see much of a difference if you run Web Filter. Web Filter doesn't use much memory, and its cloud-based architecture adds very little to CPU utilization.


How do real-time updates work?

When a client first vists a site, Web Filter accesses the zveloDB to get the categories the site is under to make a decision to block or pass based on your configuration. The category information is also written to a local cache so it doesn't have to be checked the next time a user visits that site.


How long does Web Filter cache category information for sites?

Several days. Web Filter flushes non-frequently used cache. The website that you visit daily will not be cleared from cache.


Can I add additional categories?

Custom categories are not available, however we provide over 140 categories for granular control over what your clients can access. If you feel there are categories that we can add to make it even better, just let us know.


How should I handle false positives?

While the fastest way to allow clients to access a site that is currently blocked is to add the site to your pass list, you can request recategorization of sites here - the turnaround time is usually less than two days.


Can I use Web Filter to block HTTPS/SSL sites?

Yes - because Web Filter has access to a separate database of IP addresses, it can categorize HTTPS traffic based on the destination IP address. This is not done by individual domain, but by category - for example, if you simply block 'facebook.com'. Please note that this does not mean Web Filter can parse HTTPS as it is encrypted. Categorization is done via IP address. This means other forms of blocking like URL, file-type, mime-type, etc can not be done on HTTPS as the stream is encrypted and these require parsing of the HTTP protocol.


Why can i access a site using HTTPS when I've added it to the block list?

Web Filter scans and categorizes HTTPS traffic by IP address because the session itself is encrypted and cannot be scanned. As a result, if you add "example.com" to the block list and go to "https://example.com" it will not be blocked because Untangle can only see the IP address. However, if you block the category "example.com" is in, then go to "https://example.com" it will not connect and you will see a block event in the Event Log.


Why is Web Filter still blocking an HTTPS site even after I added it to the pass list?

This should only be a problem with older browsers that do not provide SNI information in the HTTPS stream - if your browser provides SNI information, adding the domain to the pass list should allow the site to load. Older browsers that do not provide SNI information can run into this problem, however. If this is the case, it is because Web Filter does categorization of HTTPS traffic by IP address. HTTPS encrypts the hostname and request, so all we can see is the destination IP. This means if https://example.com/ is getting blocked, adding "example.com" to the passlist will have no effect because HTTPS is categorized by IP address. If you add the IP address of example.com to the passlist then HTTPS traffic to example.com will be allowed.