[BUG] importing emails fails and increases duplicated messages counter

[ManfredWisniewski]

I am not sure this is a bug since noone else is reporting it, but also the debug does not give me any relevant information about what is wrong.
I have tested this with two different mail servers and different mailboxes and I always get the same errors. There is no indication on why it fails to import.

Describe the bug
when I run pilerimport I only get errors like this:
for imap:

ERROR: error importing: './1.eml'
failed to import: ./5.eml (id: 500000006924436d0725eee4000d1a74e9bd)

for pop:
failed to import: pop3-tmp-21752-16.txt (id: 50000000692443b512874364007da5382d39)

To Reproduce
Steps to reproduce the behavior:

1. import emails via pilerimport:
   docker exec piler sh -c "cd /tmp && pilerimport -i imap://mail.server -u mail@box -pPassword -x drafts,spam,trash"
docker exec piler sh -c "cd /tmp && pilerimport -K mail.server -u mail.box -p 'password' -x drafts,spam,trash"

Expected behavior
import the emails

Piler version:

• 1.4.7

Additional context
piler is running in docker (sutoj/piler:1.4.7)

[ManfredWisniewski]

I am even getting this error when importing emails directly from a directory:
ERROR: error importing: '/tmp/mails/testmail.eml'

[ManfredWisniewski]

Ok... so I upgraded to 1.4.8 which introduces new bugs for importing (key file missing in configuration)

Then I tried to downgrade to 1.4.5 which seemed the latest stable version. Here we have a php-fpm version mismatch. 8.1 is installed but nginx expects 8.3.

Is this public version abandoned? None of these "releases" seem usable...

[jsuto]

No, they are not abandoned. What key file is missing exactly? Also try running pilertest on the test email. It shall print how the parser sees the email.

[ManfredWisniewski]

Thanks for your reply!
The key file issue was from upgrading the from the 1.4.7 docker image to 1.4.8. I did a fresh install of 1.4.8 and the configuration and the key were created correctly.

I found out that piler was actually downloading emails and that the emails are in /var/piler/imap (in the container)

the import job however is stuck at 0 / 3933
This can't be as the mailbox barely has 50 emails in total - including Spam and Trash directories.

I am not sure if the imported emails are from the import job that I created in the UI or from running the pilerimport.

Pilerimport (running manually) is still giving me tons of error messages like this and then it just keeps looping:

failed to import: ./36.eml (id: 500000006924882f31941da400987906f55a)
ERROR: error importing: './36.eml'

PS. pilertest for the messages in /var/piler/imap works fine, but the emails are not showing up in the UI. Even the health monitor still shows 0 for everything (0 processed, recieved etc.)

[ManfredWisniewski]

pilertest result example: (I think this looks fine)

locale: C
version: 1.4.8-2e863b0
parsing...
post parsing...
message-id: <20250507030706> / 3b5c1b72776f0d8f6f723bdc2dcf
from: **
sender: **
to: **
journal recipients: **
reference: **
subject: *Subject*
body: *body*
sent: 1746587226, delivered-date: 0
hdr len: 1076
body digest: 0b12123530143359aba380d41e5c5891
rules check: (null)
folder: 0
retention period: 1967512026 (2557 days)
attachments:
direction: 1
spam: 0
1st received line: Received: by mail.server.de (Postfix, from userid 999)   id DE384; Wed,  7 May 2025 05:07:06 +0200 (CEST)

[ManfredWisniewski]

Any idea on why the import is failing?
The import job was now at 0 / 308765 and still running. I am not getting any reasons from the logs except the "failed to import" mentioned above.

I have tested this with two different mailboxes via POP3 and IMAP (dovcot and a stalwart mailserver) and the behaviour is the same for both.

[ManfredWisniewski]

@jsuto is there any way to get verbose output for pilerimport or more debug information?

When I test the eml I am trying to import manually pilertest confirms it is a RFC822 compliant mail (normal headers, MIME, body). So pilerimport should be able to handle it. Yet I am still getting the failed to import: errors with no explanation.

[jsuto]

pilerimport syslogs if it encounters an issue, check the mail log. Also can you try running pilerimport directly on your imap server? The problem with the import job setup on the UI is that there's not much feedback from pilerimport compared to when you run it manually from the terminal.

[github-actions]

This issue is stale because it has been open for 14 days with no activity.