Domino allows X-Fields on email processed by its servers. These fields are designed for interoperability of mail systems, allowing the passing of more data than just “To” “From” “Subject” and “Body”. In theory, these give a great deal of flexibility to different email systems. In practice, the large variety of these and the fact that spammers use them, too, means that a Domino server quickly hits its limit of 3,000 fields on a server. Even the limit of allowed fields in the mail.box files can be hit fairly quickly despite the ‘allow more fields’ property which is set by default when Domino creates the file.
What are X-Fields?
X-fields (or X-headers) are a way to transmit non-standard information between SMTP servers, allowing the passing of more data than just To, From, Subject etc. Typically an SMTP server will strip off any headers that it does not understand, while those preceded by ‘X-‘ are normally passed on.
How does Domino use them?
They can be used for a variety of reasons. For example, tagging an email with a spam score (so that the receiving client can move it to a different folder).
How does this create problems?
By default, the maximum number of uniquely named fields in a database is approximately 3,000 (limited to ~ 64K total length for all field names), although you can enable the database property “Allow more fields in database” to get up to 22,893 uniquely-named fields. Even with this extended limit, the mail boxes and mail databases on a busy server can quickly hit this number as the field names are still stored in the database even after the document has been moved/deleted. Spam emails coming in to your server often have a lot of randomly named X-fields – some are obviously designed to attempt to bypass filters (such as X-SpamChecked) while others seem to serve no useful purpose at all.
What is the error you will see in Domino?
The server can throw this error:
Cannot store document – database has too many unique field names.
This means that the document is stuck and cannot move to your server or to your mail database. As a user, it is often invisible, so you don’t know your mail is “stuck” somewhere.
And here is the fix:
Please set the ‘Allow more fields in database’ option or ask your administrator to compact the database.
Domino admins are in the habit of deleting mail.box or compacting it to have these fields “disappear”. But busy servers don’t always allow this to occur.
So, “X Field Eliminator” puts all of these into a single field (or simply deletes them) to avoid hitting this problem. There is also an “X-Field” whitelist, which means that field, which may contain important data from an external system, is not lost.
So this utility saves some admin time. What’s the big deal?
The big deal is that these fields roll through to all end user mail files. My personal mail file is 3 years old, and it has over 3,640 of these fields, which means a pain in the a** an administrative headache for me and someone else who is unlucky to get the “help!!” call that day. They also store extra data and therefore increase the mail file size.
Here are just a few of the 3,640 X-Fields in my personal mailbox.
How does XFieldEliminator work exactly?
It runs as a server add-in task (on Windows servers). The best place is on the SMTP gateway, but it can run on any Domino server and it will clean the X Fields.
The XFieldEliminator.dll (add-in task) has four options:
– Delete X-Fields
– Rename X-Fields
– Rollup X-Fields into RFC822 text field
– Rollup X-Fields into a rich text field.
Delete X-Fields will simply delete all X-Fields on the document.
Rename X-Fields will change the Notes field name to X-Field1, X-Field2, X-Fieldx . Although the Notes name is changed, the RFC822 Header Name is preserved.
Rollup X-Fields into an RFC822 text field
X-Fields will be combined into one RFC822 text field. X-Fields are truncated at 2K. The rollup field is named X-FieldRollup1, X-FieldRollup2, X-FieldRollup3, etc.
Rollup X-Fields into a rich text field
X-Fields will be combined into one easy-to read rich text field. X-Fields are not truncated. The new rich text field name is XFieldRollup.
Can I whitelist certain X-Fields?
Yes, you can also white list, or skip X-Fields so that they are not removed. This is necessary to use the X-Fields as they are intended, for cross-system data communication.
In my tests, the typical email database only has the original 1,002 fields that come with the database. With XFieldEliminator, there would only ever be 1,003 fields and no need to set the “allow more fields in database” option.
Where can I get this software?
You can download the XFieldEliminator and try it for free on any Windows based server here: