PDF xmp timestamps aren't extracted with timezone info

[patrickdalla]

xmp metadatas dates, like xmp:MetadataDate and xmp:ModifyDate, are being extracted without the zone info, and so are put as UTC but without the correct adjustment.

[lfcnassif]

Could you test standalone Tika or PDFBox versions used by us on your sample(s) to see if it is a dependency issue? Last year I fixed a similar issue with Exif dates into Tika:
https://issues.apache.org/jira/browse/TIKA-3815

[patrickdalla]

right Em qua., 6 de set. de 2023 15:17, Luis Filipe Nassif < ***@***.***> escreveu:

…

Could you test standalone Tika or PDFBox versions used by us on your sample(s) to see if it is a dependency issue? Last year I fixed a similar issue with Exif dates into Tika: https://issues.apache.org/jira/browse/TIKA-3815 — Reply to this email directly, view it on GitHub <#1868 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG247SYSEI2DOZE3CPZ4OP3XZDD5NANCNFSM6AAAAAA4NYS5OM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[patrickdalla]

I've run:
[root@localhost Downloads]# java -jar tika-app-2.9.0.jar sobreavisoEditado3.pdf | grep xmp
that returned

WARN  [main] 07:42:34,238 org.apache.pdfbox.pdmodel.font.PDType1Font Using fallback font LiberationSans for base font Symbol
WARN  [main] 07:42:34,241 org.apache.pdfbox.pdmodel.font.PDType1Font Using fallback font LiberationSans for base font ZapfDingbats
<meta name="xmp:ModifyDate" content="2023-09-06T13:35:38Z"/>
<meta name="xmp:MetadataDate" content="2023-09-06T13:35:38Z"/>
<meta name="xmpTPg:NPages" content="11"/>

The IPED hexviewer showed the second ModifyDate tag as:

So, this is an issue of the custom PDF Tika Parser. I will inform there. Should I close it here?

[lfcnassif]

You can report on https://issues.apache.org/jira/projects/TIKA/issues

(But maybe it comes from PDFBox Tika dependency...)

We should keep this open until it is fixed in upstream library and we update Tika. Although coming from a dependency, it is a bug affecting the full software.

[patrickdalla]

The command:
java -jar pdfbox-app-2.0.29.jar ExtractXMP -console sobreavisoEditado3.pdf

Returned the correct timestamp with zone info.
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?><x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><rdf:Description rdf:about="" xmp:ModifyDate="2023-09-06T13:35:38-04:00" xmlns:xmp="http://ns.adobe.com/xap/1.0/"><xmp:MetadataDate>2023-09-06T13:35:38-04:00</xmp:MetadataDate></rdf:Description></rdf:RDF></x:xmpmeta><?xpacket end="w"?>

So it seems a TIKA parser issue.

[lfcnassif]

I would appreciate if you could report to them when you get permission, since I'm going to travel on vacation now and I'll be back on the weekend, thanks.

[patrickdalla]

I have already got the permission an registered the issue Em seg., 11 de set. de 2023 11:08, Luis Filipe Nassif < ***@***.***> escreveu:

…

I would appreciate if you could report to them when you get permission, since I'm going to travel on vacation now and I'll be back on the weekend, thanks. — Reply to this email directly, view it on GitHub <#1868 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG247S6PQ363VW7JR5CUSXLXZ4SODANCNFSM6AAAAAA4NYS5OM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[patrickdalla]

I found the problem. When setting Calendar object timezone, it does not sync remainder internal fields immediatelly, being delayed. Only whe other methods are called, like date.get(Calendar.HOUR_OF_DAY), the object identifies that its internal state is out of sync and updates them.

So, when the info is read from PDF XMP metadata, the original instance timezone is UTC, and it sets to the informed timezone (-4 in my case) but does not change (again) any other internal fields.

When formatDate method of DateUtils is called, it sets again the timezone of the Calendar object, this time with UTC. Only after, in doFormatDate method, when the get method of Calendar is called, it sync all other fields state, but as the timezone is in UTC, it does not make the HOUR field shift as it thinks this field is uptodate.

I could overcome this just calling "date.get(Calendar.HOUR_OF_DAY);" right before "showed date.setTimeZone(UTC);" in DateUtils, as the get method forces the first timezone setting done when metadata info was read to be applied, so latter timezone modification also shifts correctly the hour_of_day field.

I could do this as there is the iped.engine.tika.SyncMetadata class that we use, overriding the Metadata set method. I have done just to test my hipothesis, what have shown was true.

So we can implement this way, or wait a definitive correction from TIKA. What do you think @lfcnassif?

[lfcnassif]

So we can implement this way, or wait a definitive correction from TIKA. What do you think @lfcnassif?

Hi @patrickdalla. Seems Tim Allison fixed the bug you reported on https://issues.apache.org/jira/browse/TIKA-4126, right?

But not sure when next Tika version will be released. By the way, we already have a ticket to upgrade Tika (#1744), but that is a big change and needs lots of testing... If we can fix this issue on our side, that would be great! I just ask you to put a comment next to Tika version in our pom.xml stating we should revert your fixing changes when Tika is upgraded.

[patrickdalla]

Right, I will commit in same branch of the PDF carve improvement branch PR.

[patrickdalla]

Let me explain better the problem, as it seems a Calendar usage problem.
When the fields of the calendar are set (timezone included), the other possible affected fields are not immediately changed. There is a "complete" method in Calendar class, but it protected. This method calls the other Calendar protected method "computeFields" if the internal fields state have changed, state controlled by the areAllFieldsSet boolean field.
When setTimeZone method is called, the areAllFieldsSet turns false, marking the field in a inconsistent state.
The consequent computation that should be done (shift the fields value based on timezone offset) is relative to UTC.

So, for example, when you set a HOUR OF DAY field to 13, latter the timezone to -4, the field HOUR OF DAY still remains with the 13 value. The Metadata class implementation normalizes all internal date values to UTC, so it call again setTimeZone, but still not making any computation in HOUR OF DAY field that remains with the 13 value.

When the date is formated to string, the method calendar.get(Calendar.HOUR_OF_DAY) is called, that identifies the inconsistent state (areAllFieldsSet = false) and call computeFields. But as the zone is set to UTC, this method does not change the value of the HOUR OF DAY field.

So, to solve this problem, I put the command calendar.get(Calendar.HOUR_OF_DAY) before the setTimeZone(UTC), to force the computeFields call with the zone already configured (loaded from XMP PDF metadata in our case) before changing it to UTC.

That method call was done in SyncMetadata class, method "void set(Property property, Calendar date)".

Tim Allison used another approach to solve it. His approach converts the calendar to Instant class, without setting the UTC normalized timezone as the used Instant toString method already formats it in ISO-8601 (DateTimeFormatter.ISO_INSTANT). But I could not find a easy way to use his approach without changing TIKA code directly.

[patrickdalla]

Pushed to PDF_Carve_Improvement

[lfcnassif]

Right, I will commit in same branch of the PDF carve improvement branch PR.

Please could you revert and push to another branch and PR?

[patrickdalla]

Done

[lfcnassif]

Thank you! And sorry for the additional effort from your side...