UA POLICE Lite (security)

[Peter]

This plugin hardens a site's security by enforcing the usage of valid UA (User Agent) header.

A typical UA of a human visitor looks like:

Code:

Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; en) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/4.1.3 Safari/533.19.4

HTTP requests with invalid UA (User Agent) header will be served an error page.

This plugin is a valuable security tool while blocking hackers and malicious robots.

This is a simplified version of "UA Police Pro". The Pro version offers advanced functions, user-defined spoofed UA detection options and detailed logging of events. The Pro version is not yet available. It will take some effort to finish the development work -- I estimate about 4-8 weeks from now.

Download at from Jamit Market http://market.jamit.com/. Everyone should download and use this plugin!

What is User Agent? https://secure.wikimedia.org/wikipedia/en/wiki/User_agent

Examples of invalid User Agents (UA):

[empty] ......... Empty or non-existent UA header
Mozilla/4.0 ..... This UA belongs to a rogue robot
Java/1.6.0_04 ... This UA belongs to a rogue robot
xpymep.exe ...... This UA belongs to a rogue robot
example/1.0 ..... This UA belongs to a rogue robot


Detecting spoofed, forged and invalid (according to some rules) User Agent and blocking such requests is one of several ways to harden a site's security. Forcing the use of a valid User Agent is used on many famous and big sites, such as:

• wikimedia.org
• wikipedia.org
• arxiv.org Cornell University
• real.com Requires non-empty UA

[wclang]

Hi Peter,

Thanks for the great freebie

Quick question regarding the plugin. I installed it and it seems to be working well (Blocked 1096 requests with invalid UA so far) but my question is what is the recommended Minimum UA length number I should have entered? The default after the install is 10 but I wasn't sure if I should leave it there or change it?

Thanks,

-wclang

[Peter]

.... I installed it and it seems to be working well (Blocked 1096 requests with invalid UA so far) but my question is what is the recommended Minimum UA length number I should have entered? The default after the install is 10 but I wasn't sure if I should leave it there or change it?.....

Yes, sites with high traffic volumes will block many robots with invalid UA. Thanks for the feedback. I am happy to know that my plugin has been very effective on your site.

Here is the reason why I made this plugin --
- When there is an invalid UA (User Agent) header detected, the UA POLICE plugin will give the visitor (or robot) a clear warning and status about what is wrong (the error page). Some human visitors deliberately forge their UA in order to protect their privacy. When they see the error page, they can make switch to using a valid UA and visit your site as any other normal visitor. Unlike in Traffic Cop, the blocking in UA POLICE is NOT permanent! Also, UA POLICE is lightweight and uses less overhead than Traffic Cop (for validation of UA).

On my non-JobBoard sites, I use a setting of 34 characters minimum length. This higher value may potentially block some robots that you'd want to access your site. In my case, I am very, very restrictive about who can access my site.

Once the PRO version is out, you will be able to study the detailed logs and see if there were any false positives and adjust your settings accordingly. The PRO version will also have more advanced settings to better detect forged UA.

As I wrote, you may try increasing the minimum length setting up to 34, however, this higher setting will block requests from robots with the following UA:

Code:

Mozilla/5.0
Mozilla/5.0 (compatible)

The above UAs may be used by some legitimate robots. I recently discovered a visit from CAREERJET.NET robot which used UA

Code:

Mozilla/5.0 (compatible)

Best wishes,
Peter

How to forge UA (User Agent):

• Use the Firefox browser.
• Install an add-on that allows you to change (forge) your UA, such as "UA Control" (https://addons.mozilla.org/en-US/firefox/search/?q=uacontrol&cat=1%2C0&x=21&y=12) or "User Agent Switcher" (https://addons.mozilla.org/en-US/firefox/addon/user-agent-switcher/). I recommend the first one!

[wclang]

Hello Peter,

It seems UA Police is blocking my Cron job from running. I checked the email logs I receive from cron and this is what they say:

Code:

<HTML>
<HEAD>
<TITLE>User Agent Required</TITLE>
</HEAD>
<BODY>
<H1>User Agent Required</H1>
This site enforces the use of a valid User Agent. Please contact your system
administrator for assistance.
<P>
<HR>
<ADDRESS>
Web Server at
</ADDRESS>
</BODY>
</HTML>

<!--
   - Unfortunately, Microsoft has added a clever new
   - "feature" to Internet Explorer. If the text of
   - an error's message is "too small", specifically
   - less than 512 bytes, Internet Explorer returns
   - its own error message. You can turn that off,
   - but it's pretty tricky to find switch called
   - "smart error messages". That means, of course,
   - that short error messages are censored by default.
   - IIS always returns error messages that are long
   - enough to make Internet Explorer happy. The
   - workaround is pretty simple: pad the error
   - message with a big comment like this to push it
   - over the five hundred and twelve bytes minimum.
   - Of course, that's exactly what you're reading
   - right now.
   -->

Any idea how to fix this so my cron jobs run?

[Peter]

Thanks for pointing this out!

The bug is now fixed and new version 1.06 is available on Jamit Market.

[Peter]

There are some limitations with UA Police Lite.

UA Police Lite tries to detect an invalid User Agent (UA) by measuring the length of the UA string and comparing it to minimum and maximum length.

I have recently encountered a robot by Twitter having this UA:

Twitterbot/0.1

So if you use UA Police Lite (and want to allow the Twitter robot), make sure that the minimum length is 14 (or more).

You don't have this limitation if you are using UA Police Pro. This is what you do if you have UA Police Pro:

• You white list the Twitter UA string
• You can have the minimum length any number, such as 10. No problem! Just make sure that you white list robots that would otherwise be blocked by your black list rules.

Those who want to switch from UA Police Lite to UA Police Pro, you can trade in your Lite version and then the Pro version would cost you less. Just contact me if you have any questions.