Chapter 19 Throttling Incoming Connections Using MeterMaid

MeterMaid is a server that can provide centralized metering and management of connections and transactions, including through monitoring IP addresses SMTP envelope addresses. Functionally, MeterMaid can be used to limit how often a particular IP address can connect to the MTA. Limiting connections by particular IP addresses is useful for preventing excessive connections used in denial-of-service attacks. MeterMaid supplants conn_throttle.so by providing similar functionality, but extending it across the Messaging Server installation. No new enhancements are planned for conn_throttle.so and MeterMaid is its more effective replacement.

This section consists of the following subsections:

• 19.1 Technical Overview

• 19.2 Theory of Operations

• 19.3 Configutil Parameters for MeterMaid

• 19.4 Limit Excessive IP Address Connections Using Metermaid—Example

19.1 Technical Overview

conn_throttle.so is a shared library used as a callout from the MTA's mapping table that uses an in-memory table of incoming connections to determine when a particular IP address has recently connected too often and should be turned away for awhile. While having an in-memory table is good for performance, its largest cost is that each individual process on each server maintains its own table.

In most cases, the conn_throttle.so callout is done in the PORT_ACCESS mapping that is accessed by the Dispatcher, a single process on each system. The only cost is that there is a separate table per server.

The primary improvement by MeterMaid is that it maintains a single repository of the throttling information that can be accessed by all systems and processes within the Messaging Server environment. It continues to maintain an in-memory database to store this data to maximize performance. Restarting MeterMaid will lose all information previously stored, but since the data is typically very short lived, the cost of such a restart (done infrequently) is very low.

19.2 Theory of Operations

MeterMaid's configuration is stored in msg.conf and is maintained by configutil.

MeterMaid is accessed from the MTA through a mapping table callout using check_metermaid.so. It can be called from any of the *_ACCESS tables. When called from the PORT_ACCESS table, it can be used to check limits based on the IP address of the connection which will be the most common way to implement MeterMaid as a replacement for the older conn_throttle.so. If called from other *_ACCESS tables, MeterMaid can also be used to establish limits on other data such as the envelope from or envelope to addresses as well as IP addresses.

Only one entry point in check_metermaid.so is defined. The throttle routine contacts MeterMaid providing two subsequent arguments separated by commas. The first is the name of the table against which the data will be checked, and the second is the data to be checked.

If the result from the probe is that the particular data being checked has exceeded its quota in that table, check_metermaid.so returns "success" so that the mapping engine will continue processing this entry. The remainder of the entry would then be used to handle this connection that has exceeded its quota.

PORT_ACCESS

  *|*|*|*|* $C$|INTERNAL_IP;$3|$Y$E
  *|*|*|*|* $C$:A$[/opt/SUNWmsgsr/lib/check_metermaid.so,throttle,tablename,$3]$N421$ \
Connection$ declined$ at$ this$ time$E
  *          $YEXTERNAL

Note the $:A flag test in the mapping table entry before the call to check_metermaid.so. This is to ensure that we only do the MeterMaid probe when PORT_ACCESS is being checked by the dispatcher as it will set the A flag for its probe.

19.3 Configutil Parameters for MeterMaid

MeterMaid's configuration is stored in msg.conf and is maintained by configutil. Here are some of the settings currently supported by MeterMaid. Defaults are in parenthesis. See configutil Parameters in Sun Java System Messaging Server 6.3 Administration Reference for complete list of MeterMaid parameters.

Chapter 19, Throttling Incoming Connections Using MeterMaid

• local.metermaid.enable This setting must be set to yes on the system that will run the MeterMaid daemon so that the Watcher will start and control MeterMaid.

• logfile.metermaid.* These settings are the same as those used by imap, pop, and other services. By default MeterMaid writes its log file into msg-svr-base/data/log/metermaid

• metermaid.config.listenaddr (INADDR_ANY) The address to which MeterMaid should bind. On most systems, the default would not need to be changed, but for multi-homed or HA systems, specifying the appropriate address here is recommended.

• metermaid.config.maxthreads (20) The MeterMaid server is multithreaded and maintains a pool of threads onto which its tasks are scheduled. This value sets the maximum number of threads that will be used by MeterMaid. On systems with more than 4 CPUs, increasing this value may increase overall throughput.

• metermaid.config.port (63837) This is the port to which MeterMaid listens for connections and to which MeterMaid clients will connect.

• metermaid.config.secret (No default; value must be supplied) In order to authenticate incoming connections, MeterMaid uses a shared secret that the clients send once they connect to MeterMaid.

• metermaid.config.serverhost (No default; value must be supplied) This is the host name or IP address to which the clients will connect. It may be the same as metermaid.config.listenaddr but will most likely have a particular value to direct clients to one system in particular in the Messaging Server environment.

These settings are used by the check_metermaid client:

• metermaid.mtaclient.connectfrequency (15) Attempt a connection every connectfrequency seconds. When the client needs to connect to MeterMaid, it uses this as an internal throttle to prevent constant connection attempts when MeterMaid isn't available. During the time that the client is unable to communicate with MeterMaid, it will return a "fail" status to the MTA mapping engine indicating that MeterMaid has not blocked this connection.

  For example, if check_metermaid.so attempts to connect to MeterMaid, but it fails for some reason, during the next N seconds as specified by metermaid.mtaclient.connectfrequency, no additional attempts will be attempted. It prevents check_metermaid.so from trying to connect to MeterMaid too frequently if it is not working.

• metermaid.mtaclient.connectwait (5) When the client is waiting for a connection to MeterMaid (either an initial connection or to reuse another already established connection), it will wait for connectwait seconds before returning a fail status and allowing this connection to continue.

• metermaid.mtaclient.debug (no) If this option is enabled, debugging information from the client will be printed into either the server or thread-specific log file for the SMTP server.

• metermaid.mtaclient.maxconns (3) In order to support multithreaded servers, the client can maintain a pool of connections to MeterMaid. By doing this, there can be increased concurrency during communications. However, due to internal locking done by MeterMaid, access to a particular table is limited to one request at a time, so multiple connections from a single process may provide limited benefit.

• metermaid.mtaclient.readwait (10) When communicating with MeterMaid, the client will wait readwait seconds before returning a fail status and allowing this connection to continue.

Lastly, the throttling tables are also defined in msg.conf as shown here. The * in each configuration parameter is the name of the particular table being defined. For example, for a table called internal, the first parameter would be called metermaid.table.internal.data_type.

• metermaid.table.*.data_type (string) MeterMaid can support two kinds of data in its tables, string and ipv4. string data is limited to 255 bytes per entry and can be compared using case-sensitive or case-insensitive functions (see metermaid.table.*.options below).

• metermaid.table.*.max_entries (1000) When MeterMaid initializes each table, it pre-allocates this many entries. MeterMaid automatically recycles old entries, even if they haven't yet expired. When a new connection is received, MeterMaid will reuse the least recently accessed entry. A site should specify a value high enough to cache the connections received during quota_time.

• metermaid.table.*.options is a comma-separated list of keywords that defines behavior or characteristics for the table. Valid keywords are:

  • nocase — When working with the data, all comparisons are done using a case-insensitive comparison function. (This option is valid only for string data.)

  • penalize — After quota_time seconds, throttle will normally reset the connection count to 0, but if the penalize option is enabled, throttle will decrement the connection count by quota (but not less than 0) so that additional connection attempts will penalize future quota_time periods. For example, if quota were 5 with a quota_time of 60, and the system received 12 connection attempts during the first minute, the first 5 connections would be accepted and the remaining 7 would be declined. After 60 seconds has passed, the number of connections counted against the particular address would be reduced to 7, still keeping it above quota and declining connection attempts. Assuming no additional connection attempts were made, after another 60 second period, the number of connections would be further reduced down to 2, and MeterMaid would permit connection attempts again.

• metermaid.table.*.quota (100) When a connection is received, it is counted against quota. If the number of connections received in quota_time seconds exceeds this value, MeterMaid will decline the connection. (The actual effect on the incoming connection is controlled by the mapping table and could result in additional scrutiny, a delay, or denying the connection.)

• metermaid.table.*.quota_time (60) This specifies the number of seconds during which connections will be counted against quota. After this many seconds, the number of connections counted against the incoming address will be reduced depending on the type of this table.

• metermaid.table.*storage (hash) MeterMaid can use two different storage methods, hash and splay. The default hash table method is recommended, but under some circumstances a splay tree may provide faster lookups.

• metermaid.table.*.type (throttle) Currently, the only table type supported by MeterMaid is throttle. This type of table keeps track of the data, typically IP addresses, and will throttle the incoming connections to quota connections during a period of quota_time seconds.

Metermaid was coded to be as efficient as possible, but may not match your expectations as it uses weighted averages to limit the connection rate rather than maintaining a list of just how many connections there have been over the previous quota_time period.

For example, let's say you are trying to limit connections to 1250 mails per hour (metermaid.table.tcp_auth_msg_throttle.quota = 1250). This limits connections to an average rate of 1250/hour. In other words, if all 1250 connections are used up in the first second, then half an hour later, an additional 625 are provided. After another 15 minutes, you'd get another 362, and so on.

A common misconception is that one hour after the initial connection attempt, all will be forgiven. But Metermaid looks at an average connection rate rather than a specific count. The net effect is the same in the long run (over a 24 hour period, a total of 1250 * 24 would be allowed), but it's not mathematically precise in a short period.

A future goals for Metermaid is that instead of offering an average throughput rate, it would maintain a distinct list of connection attempts, and be able to expire each attempt after the precise time. This has not yet been done because it's a much more computationally expensive operation to maintain than computing an average rate.

19.4 Limit Excessive IP Address Connections Using Metermaid—Example

This example uses MeterMaid to throttle IP addresses at 10 connections/minute. For reference, the equivalent conn_throttle.so setup in the mappings file would be as follows:

PORT_ACCESS
   *|*|*|*|*  $C$|INTERNAL_IP;$3|$Y$E
   *|*|*|*|*  $C$[/opt/SUNWmsgsr/lib/conn_throttle.so,throttle,$3,10]\
$N421$ Connection$ declined$ at$ this$ time$E
   *          $YEXTERNAL

This PORT_ACCESS mapping table implements conn_throttle.so to restrict connections to a rate of no more than 10 connections per minute for non-INTERNAL connections.

One fundamental difference between the two technologies is that instead of configuring details such as the rate-limit for throttling directly into the mapping table, MeterMaid uses configutil parameters for these settings. This example is described below.

1. Designate one of your systems to be the MeterMaid server host.

   On this system, set the following configutil parameter:

   local.metermaid.enable -v TRUE

   Set an authentication password used to verify communications between the client and MeterMaid server:

   configutil -o metermaid.config.secret -v password

2. Define a throttling table.

   MeterMaid's throttling behavior is determined by the use of named throttling tables that define operating characteristics. To define a table that throttles at a rate of 10 connections per minute, set the following parameters:

   configutil -o metermaid.table.ext_throttle.data_type -v ipv4 configutil -o metermaid.table.ext_throttle.quota -v 10

   ext_throttle is the name of the throttling table. ipv4 is the data type Internet Protocol version 4 address representation. 10 is the quota (connection limit).

3. On the MeterMaid system, start MeterMaid.

   # start-msg metermaid

4. On systems where the MTA will use MeterMaid to do throttling, specify the MeterMaid host and password.

   These are required:

   configutil -o metermaid.config.secret -v MeterMaid_Password configutil -o metermaid.config.serverhost -v name_or_ipaddress_of_MetermaidHost

5. Set up the MeterMaid PORT_ACCESS table.

   This table is similar to the equivalent conn_throttle.so setup:

   PORT_ACCESS *|*|*|*|* $C$|INTERNAL_IP;$3|$Y$E *|*|*|*|* $C$:A$[/opt/SUNWmsgsr/lib/check_metermaid.so,throttle,\ ext_throttle,$3] $N421$ Connection$ declined$ at$ this$ time$E * $YEXTERNAL

   The first line checks to see if the IP address attempting a connection is internal. If it is, it allows the connection. The second line runs the IP address through MeterMaid and if it has connected too frequently, it declines the connection. The third line allows any other connections through, but flagged as EXTERNAL.

   Note that this call to check_metermaid.so is very similar to the callout to conn_throttle.so. The function in check_metermaid.so is the same. throttle and its arguments are simply the table name as configured using metermaid.table.tablename and the IP address to check ($3). Like conn_throttle.so, this function returns success when the limit (as specified in metermaid.table.ext_throttle.quota) has been reached. This allows the remainder of the mapping entry line is processed, which sends a message (421 SMTP code, transient negative completion, Connection not accepted at this time) to the remote SMTP client, and tells the Dispatcher to close the connection.

   Note also that $:A ensures that this line will only be processed when being called from the Dispatcher. Without this, the call to check_metermaid.so would also happen in the context of the tcp_smtp_server processes which also probes the PORT_ACCESS mapping table. This would cause MeterMaid to count each incoming connection twice.

This is the basic configuration to set up MeterMaid as a conn_throttle.so replacement. See 10.3.2 Mapping Operations and 18.3.4 PORT_ACCESS Mapping Table for information on these topics.

19.4.1 Additional Useful MeterMaid Options

Two additional configuration options may be useful in some circumstances. The conn_throttle.so shared library also had a throttle_p function that would penalize connections that exceeded the limit by applying a consequence for an extended period beyond the basic 60 seconds. This same behavior is available in MeterMaid by configuring the following option on the MeterMaid server system:

configutil -o metermaid.table.ext_throttle.options -v penalize

This changes the behavior for the ext_throttle table so that connections may be penalized for connection attempts greater than the value set for metermaid.table.ext_throttle.quota.

The other option is relevant for systems that receive a large number of connections. Because MeterMaid is able to keep track of connections throughout the distributed MTA environment, it's possible that the limit of the number of connections being retained in MeterMaid's internal in-memory database may be insufficient for the overall volume of the MTA environment. The default is 1000 entries per table, but if you anticipate having more than 1000 connections per minute throughout your MTA environment, you can increase this number through this configuration option:

configutil -o metermaid.table.ext_throttle.max_entries -v max_entries

Note that even if the max_entries is reached during the 60 second period, MeterMaid will automatically discard the oldest and least frequently used entries. Thus, the more frequently connecting systems will remain in MeterMaid's table to be counted, keeping enough information to provide effective throttling.