19 Jul

Zabbix template to monitor ESVA via SNMP

A ZABBIX TEMPLATE

We use Zabbix to monitor our ESVA instances. The monitoring happens only via SNMP, no agent is installed on ESVA appliances.

Below you find the link to download a Zabbix template to monitor your ESVA instances.

Here is a description of what this template provides.

Items avaiable in the template

  1. ESVA version number
  2. Uptime
  3. Hostname
  4. Incoming queue size
  5. Outgoing queue size
  6. Incoming queue delta (calulated)
  7. Outgoing queue delta (calculated)
  8. RAM total
  9. RAM free
  10. RAM cached
  11. RAM buffered
  12. RAM used (calculated)
  13. SWAP total available space
  14. SWAP free space
  15. SWAP used space (calculated)
  16. SWAP used percentage (calculated)
  17. Quarantine total space (/var)
  18. Quarantine used space
  19. Quarantine used percentage (calculated)
  20. Quarantine free space (calculated)
  21. Load average 15m
  22. SMTP service status
  23. HTTPS service status
  24. SSH service status

Triggers available in the template

Level: Warning
  1. SMTP down (unavailable for 3 consecutive minutes)
  2. SSH service up (SSH service is on, this should happen only during maintenance)
  3. HTTP service down (unavaileble for 3 consecutive minutes)
Level: Average
  1. Incoming queue growing (has constantly been growing for 3 consecutive hours)
  2. Outgoing queue growing (has constantly been growing for 3 consecutive hours)
  3. Quarantine filesystem > 79%
  4. Swap filesystem > 50%
Level: High (you should set-up a notification here)
  1. Incoming queue too high (queue size has been over 4 times the last week’s average for 30 consecutive minutes)
  2. Outgoing queue too high (queue size has been over 6 times the last week’s average for 30 consecutive minutes)
  3. HTTP service down (unavailable for 10 consecutive minutes)
  4. SNMP down (no data for 15 consecutive minutes)
  5. SWAP filesystem high (over 78% for 10 consecutive minutes)
  6. SSH service on (has been on for 3 consecutive minutes)
  7. Quarantine disk high (over 94%)
Level: Disaster (of course you should get a notification here)
  1. SMTP down (has been unavailable for 10 consecutive minutes)

Graphs available in the template

  1. System load history
  2. Queues size history
  3. RAM history
  4. SMTP and SSH history
  5. Swap history
  6. Swap current status
  7. Quarantine (var) history
  8. Quarantine (var) current status

Screens available in the template

  1. Summary history: load, queues, swap, RAM

Get the template

Download the template here

08 May

Recipients receive a winmail.dat attachment

PROBLEM

For some emails, the recipient receives a text message with a winmail.dat attachment, no HTML part.

SOLUTION

Some emails, very few recently, are sent in an old legacy and proprietary TNEF format which is not supported by all clients. Moreover, security checks and html disarming may interfere with such format.

One work-around is to disable such format on the sending side. Here is a Microsoft document that explains how to do it:
https://support.office.com/en-us/article/Recipients-receive-a-winmail-dat-attachment-1735BA97-39B8-40D4-BA17-0E0150EF87A8

 

17 Mar

How to test the email delivery before going to production

Going to production

You have installed Libra ESVA and followed the first deployment guide, you are now ready to go to production.

Going to production usually means changing the MX record of the domain to point to Libra ESVA or changing a firewall rule to route the email traffic to Libra ESVA.

You want to make sure that your configuration is correct before going to production, how to do it?

 

Testing the email delivery

Testing that the email delivery through ESVA is very easy. You just need to deliver an email to ESVA and verify that the email is correctly delivered to your mail server. When the email you sent to ESVA reached the inbox on your mail server you can safely go to production.

This simple test can be performed via command line using telnet or netcat.

Of course you can also use a mail client and configure an account that delivers email directly to ESVA but it is slower, so the example I provide below is a test via command line using netcat. If you use telnet just change nc with telnet and everything else is the same.

 

Preparation

I suggest to temporarily add the ip address from which you will perform the test to the smtp check override table so that the smtp checks are skipped. We are not testing smtp validation now and we don’t want interference.

Just enter the ip address and save, that’s it. You will remove it after the test.

 

Execution

Look at the transcript below. The part in red is the one you have to enter. Change the hostnames and email addresses appropriately.

In this example we are testing the delivery to a mailbox on the domain newdomain.com.

$ nc esvagw.domain.ext 25
220 esvagw.domain.ext ESMTP Postfix
ehlo test.libraesva.com
250-esvagw.domain.ext
250-PIPELINING
250-SIZE 50000000
250-ETRN
250-AUTH LOGIN GSSAPI DIGEST-MD5 PLAIN CRAM-MD5
250-AUTH=LOGIN GSSAPI DIGEST-MD5 PLAIN CRAM-MD5
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: rodolfo.saccani@libraesva.com
250 2.1.0 Ok
rcpt to: rodolfo.saccani@newdomain.com
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
Subject: this is a test
From: Rodolfo Saccani <rodolfo.saccani@libraesva.com>
To: Rodolfo Saccani <rodolfo.saccani@newdomain.com>
Hi,
this is a test message
Bye
.
250 2.0.0 Ok: queued as 71EDC4031F
quit
221 2.0.0 Bye

Now check the inbox of the recipient. If the email is there you’re all set and can go to production.

If the email is not there check the ESVA interface. The delivery error for this message will tell you what to fix.

 

23 Feb

Notes on sizing and monitoring the performance of Libra ESVA

HOW MANY CPUs?

It really depends on what CPUs you are using, there are huge differences among different CPUs. The performance of a CPU released three years ago it drastically different than a recent CPU. So, monitor the load and increase the CPUs accordingly.
Please note that increasing CPUs may not provide much more computing power if the hypervisor is overloaded. If your VM has 2 CPUs, the hypervisor waits for 2 cores to have a free clock cycle in order to assign it to the VM. With 4 CPUs assigned to the VM the hypervisor has to wait for 4 cores to have a spare CPU cycle and if the machine is overloaded it may wait longer before a clock cycle can be assigned to the VM. So, increasing CPUs works fine unless the hypervisor is overloaded.

THE MEMORY IS ALWAYS FULL!

The OS tries to use all the available memory and uses all the spare memory for caches and buffers, so it is normal to have always the memory usage close to 100%. You should remove the memory assigned to cache and buffers from the total amount of memory in use.
More easily, you can just have a look at the swap usage: if it’s heavily used then the memory is not sufficient or you have configured too many mailscanning processes. If the swap usage is low (20-25%) then it’s fine.

DISK PERFORMANCE

The number of processed messages per time unit is is also affected by disk speed. If you log into the console with the admin user, you can execute a disk speed test to check if it’s fast enough.

MONITORING THE PERFORMANCE

All the standard linux SNMP OIDs are available, plus the esva spcific OIDs.
To keep an eye on the perfomance, you can monitor the following parameters:
  • load avg 15m (1.3.6.1.4.1.2021.10.1.3.3)
  • free swap space (1.3.6.1.4.1.2021.4.4.0) over total swap space (1.3.6.1.2.1.25.2.3.1.5.37)
  • incoming mail queues (1.3.6.1.4.1.41091.1.1.8.0).

These are the important ones for performance monitoring.

The outgoing queues (1.3.6.1.4.1.41091.1.1.9.0) grow only in case of problems on your mail server, it is not affected by ESVA performance.

MONITORING VIA ZABBIX

A Zabbix template is available, you can download it here

07 Feb

How the Libra ESVA QuickSand file sandbox works

What is the Libra ESVA QuickSand file sandbox

It is a service that protects from malicious active content in Microsoft Office and PDF files.

Active content is any executable code embedded in the document like macros, javascript code,  ActiveX applications.

The QuickSand sandbox runs on the gateway, which means that the files never leave ESVA.

As the name suggests, it is a very quick sandbox: the attachments are analyzed in the same pipeline of the email analysis without additional delays, it is not vulnerable to the sandbox evasion techniques.

The QuickSand sandbox identifies active content inside documents and classifies it based on the behavior. The possible categories are:

  • safe: active content is present and it does not perform any critical operation in respect to security
  • suspicious: potentially critical actions are performed by the active content like downloading data from the internet, launching programs, performing actions on the filesystem and so on
  • indeterminate: active content is present but for technical reasons it’s behavior cannot be categorized with sufficient accuracy
  • encrypted: the document is encrypted and therefore it is not possible to tell whether there is active content inside

 

For each of these categories, you can choose what to do with the file:

  • deliver: deliver the file as is
  • sanitize and deliver: disarm the active content and deliver the disarmed document
  • block: do not deliver the file, it will be removed from the email

Not all of these actions are available for all the the categories, for safety reasons. You can also define fallback actions in case the document cannot be sanitized for technical reasons. In this case you can either fallback to deliver or block.

The default actions are what we suggest as the best compromise.

The attached documents are analyzed and cleaned/removed also if they are contained in archives, even if the archives are nested inside other archives.

When a file is either sanitized or blocked, the entire email message is quarantined so that the original version of the file remains available for a release should it be needed.

30 Sep

The quarantine disk is filling-up, what should I do?

Quarantine disk

The quarantine disk stores the messages that are quarantined, including their attachments.

Depending on your spam actions configuration, the quarantine can contain also clean messages.

The messages are quarantined for the number of days defined in the quarantine settings.

 

What to do if the quarantine disk is filling-up

You can choose to store less data in your quarantine or to enlarge the quarantine disk.

Here are instructions for both of these options.

 

How to reduce the quarantine usage by storing less data

There are a few configurations that you can do in order to reduce the quarantine disk usage.

If you are currently storing clean messages in quarantine (i.e. you have the action “store” for the clean messages in your anti-spam settings) you can reduce the retention days for such messages (this can be done on version 4 and up) or decide not to store the clean messages at all. You can do this from the quarantine settings.
Keeping clean messages in quarantine is useful for many reasons: being able to recover messages that the recipient accidentally lost and being able to report false negatives to esva labs are a couple of good reasons to store clean messages. Reducing the retention time for clean messages is therefore the first step that you might want to take before deciding not to store them at all.

If you are currently storing both spam and hi-spam messages in quarantine, you may want to decide not to store hi-spam messages. These are messages that have have been classified as spam with a high confidence so the chances that you will need to release them are slim. You can make this configuration from the anti-spam action settings.

Of course you can always reduce the quarantine retention days for all spam messages.
This can be done from the quarantine settings. Based on your current quarantine disk usage and current retention time, you can estimate your average daily disk usage and adjust the retention time accordingly.

 

How to increase the quarantine disk size

The quarantine disk can be easily extended in size. It requires a brief shutdown, though.

The full procedure is documented here.

15 Sep

How the Libra ESVA URLSand sandboxing service works

What is the Libra ESVA URLSand sandboxing service

It is a service available on all Libra ESVA appliances starting with version 4.0. You can enable it in System -> Content Analysis -> Sandbox Filters  by checking the “Enable URI Sanbox” checkbox.

This option can be customized for each domain, which means that you can enable it for the whole appliance and disable it for some domain or keep it disabled by default and enable it only for some domain.

 

How it works

If the option is enabled for the recipient domain, the Libra ESVA appliance rewrites the URIs it finds inside emails so that when the final recipient clicks on the link it doesn’t go to the original URI but, instead, to the EsvaLabs URI Sanbox service.

Here is an example:

Original URI:
http://www.fivl.it

Rewritten URI:
https://urlsand.esvalabs.com/?u=http%3A%2F%2Fwww.fivl.it&e=366181f3&h=6c12b0dd

When the user clicks on the link, the EsvaLabs URI Sanbox will analyze the target URI in real time by performing lookups on known malware/phishing URI lists and by actively analyzing the contents of the page looking for malicious behavior.

If the URI has recently been analyzed, the response of the Sandbox will be immediate and, if classified as “clean”, and immediate redirect is performed.

If the page has not been recently analyzed, it will be retrieved and scanned, if redirects are found the checks are repeated for all the intermediary URIs. This can take up to a few minutes depending on the number of intermediary pages and the speed of the servers serving those pages.

The user is allowed to skip the checks but warned about it, and the complete URI is shown to allow the user to decide whether to trust it or not.

If the URI is classified as “dangerous” a blocking page is displayed.

The option “I accept the risk and want to follow this dangerous link” can be disabled with the ESVA configuration flag “Do not allow users to skip URI Sandbox checks”.

If the URI is classified as “suspect” a warning page with the website screenshot preview is displayed to allow visual checks of the requested website.

The option to show suspect website preview to the user can be disabled with the ESVA configuration flag “Show preview for suspicious pages”.

Privacy

We gather the absolute minimum amount of information we need to provide the service. In the rewritten URI you can see that there are only three parameters:

  • The original URI
  • A unique ID of the Libra ESVA appliance that has rewritten the URI
  • A checksum that guarantees the integrity of the data

The last two parameters are required to verify that only legit URIs are processed by the service (i.e. URIs rewritten by Libra ESVA appliances) and that the URI has not been tampered with.

The identity of the recipient of the email is not provided to the Sandbox. Of course the original URI may contain parameters that could identify the recipient, this is inevitable. For example, a URI to unsubscribe from a mailing list might contain the email address of the recipient.

The Sandbox service is accessed via HTTPS which protects the whole conversation between the user’s browser and the sandboxing service.

 

Exceptions

Libra ESVA provides and maintains a list of exceptions via it’s usual update service. This list instructs the Libra ESVA appliance not to rewrite URIs that match these exception list. Only highly reliable services where no user content is available are included in such list.

The administrator of the Libra ESVA appliance can add exceptions via System -> Content Analysis -> Phishing Highlight. All URIs for the sites added as “safe” to the “Phishing Sites List” are not rewritten.

26 Jul

How to delist a blacklisted IP address

PROBLEM

My IP address is blacklisted by some sender reputation RBL and emails are not delivered.
How can I remove it form the blacklists?

 

RBLs

RBLs are blacklists of IP addresses. One IP address enters into a blacklist for spamming activity.
Here you can find instructions on what to do once the IP address entered into one or more blacklists.

Why did you get into the blacklist in the first place?

First of all, make sure you identified the reason for the blacklisting.
If you didn’t identify and resolve it, you will just make things worse by asking for delisting. The IP will be quickly re-blacklisted and it will be harder to delist it.
So, check why the IP has been blacklisted and fix the source of the problem before going ahead.
Once you resolved all the problems (if it is a new IP address there is no problem to resolve, of course), you can go ahead with the delisting with the following instructions.

First time cleanup

If you just acquired this IP address, make sure that the dns reverse lookup is set before requesting removal from blacklists.
When asked for a reason for requesting delisting, tell that you just acquired the IP address.

Delisting

First of all, let Valli check a bunch of RBLs for you: http://multirbl.valli.org/lookup
For each RBL where the IP is listed, follow the link and read the instructions on how to delist.
Each RBL has it’s own rules, some of them require an email verification in order to delist, some of them require that you explain the reasons of the listing (if it’s a new ip address just tell them that this ip address has just been assigned to you), some of them don’t allow delisting at all (it’s automatic after some time).

In the Valli list of RBLs, some are more important than others.
Some important blacklists are Spamhaus, Barracuda, SORBS, V4BL.
Start requesting the delisting from the important ones and then proceed until you requested delisting from all of the blacklists that allow it. The ones that don’t accept delisting requests will delist automatically after some time (like rbldns.ru).

There are some important blacklists that are not checked by Valli, make sure you remove the IP address also from those:

Outlook (Microsoft): https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&locale=en-us&ccsid=636014233369251686
This list doesn’t provide a test to check if the IP is blacklisted, you can easily test by sending an email from the ESVA IP address to a hotmail email address. If the email does go through then the IP address is not blacklisted, if the email isn’t delivered you can read in the SMTP response message the reason. You can find the SMTP response message in the maillog. Copy the text because it is needed in the removal request.

Microsoft Office: https://sender.office.com/Delist
This service offers a reputation check. You will be required to enter the ip address and your email address where a confirmation link will be sent.

Trend Micro: https://ers.trendmicro.com/reputations
Reputation check available.

Sophos: https://www.sophos.com/en-us/threat-center/ip-lookup.aspx
Reputation check available.

Symantec: http://ipremoval.sms.symantec.com/lookup/
Reputation check available.

Yahoo: http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html
This service is not just for bulk mailers, it is also used for delisting requests. Just like the Outlook service above, it does not offer a reputation lookup service so you should test for blacklisting trying to send an email to a yahoo email address and checking the result as described above for Outlook. In order to request delisting you must have a yahoo account, which you can create for free.

AT&T: http://rbl.att.net/cgi-bin/rbl/block_admin.cgi
It is not possible to check whether an ip address is listed or not. If your ip address is listed your messages will be bounced with an SMTP clause similar to this:
553 5.3.0 alph155 DNSBL:ATTRBL 521< 199.169.39.199 >_is_blocked.For assistance forward this email to abuse_rbl@abuse-att.net>

Whitelisting

It is a good thing also to enter into white lists:
https://www.dnswl.org/selfservice/

IP Reputation Monitoring

There are some services that allow you to monitor your reputation and check it over time. You can take advantage of this services if you send a few thousand emails on a daily basis to domains like gmail or hotmail. For small amounts of email traffic they don’t provide feedback.

Google: https://postmaster.google.com/
Microsoft: https://postmaster.live.com/snds/JMRP.aspxhttps://postmaster.live.com/snds/JMRP.aspx

04 Jul

Blocking Office Macros with Libra Esva

PROBLEM:

How can I block office macros?

SCENARIO 1: Treat Office Macros as Viruses

In the clamav antivirus status page you can enable the “Block Office Macros” option.

This is a system wide setting that will treat documents with office macros as viruses. Emails with attachments containing office macros will be flagged as virus and quarantined. No exception is allowed.

SCENARIO 2: Treat Office Macros as spam

If you disable the “Block Office Macros” option in the clamav antivirus status page, then the office macros will still be detected by the spam filters.

When an office macro is detected, the rule ESVA_OLE2MACRO kicks-in and the spam score of the message is incremented by 5. If you want you can change this score through the spam score override function of Libra Esva.

Exceptions in this scenario can be handled through the whitelists  or through custom spam policies.