[madbrain]

XMP files are what is needed to keep metadata across software. That is really their purpose.

Unfortunately, depending on the software you use, XMP files fail at this task. For example, Corel Aftershot (which is available on Linux) names the XMP files *.jpg.xmp . But Lightroom will name them *.xmp .
But that is only the start. Renaming the files doesn't help, even if you don't have name collisions (I do), because the format of the metadata inside those XMP files differs.
The keywords are written in different XML subsections. Aftershot has an option to export its catalog to so-called "standard" XMP files, but it's a one-way trip only.
I'm looking to move away from Aftershot, but I also don't want my metadata storage to depend on XMP files.
Lightroom can write the metadata into DNG and JPG files, though unfortunately, by default, it still uses XMP files for the metadata. You have to force metadata export to the JPG/DNG files.
Real PITA.

Do any of the open-source Linux image management programs allow reading/writing the keywords to DNG & JPG files, without requiring XMP files ?

[BigMackCam]

Do any of the open-source Linux image management programs allow reading/writing the keywords to DNG & JPG files, without requiring XMP files ?

DigiKam allows you to optionally write metadata to the RAW file. I haven't tried it, but the option is there. I think I'd feel a bit limited using DigiKam's "Image Editor" as my only RAW editor... It's actually pretty good, but certainly not at the level of Darktable or Lightroom...

[newmikey]

Unfortunately, depending on the software you use, XMP files fail at this task. For example, Corel Aftershot (which is available on Linux) names the XMP files *.jpg.xmp . But Lightroom will name them *.xmp .
But that is only the start. Renaming the files doesn't help, even if you don't have name collisions (I do), because the format of the metadata inside those XMP files differs.
The keywords are written in different XML subsections.

You have basically pointed out the same flaw with XMP as with DNG : both are developments which are not so much targeted at ease of use, reliability or merit but purely on vendor lock-in by the creator of both "standards" (quotation marks as most other companies quickly realized what Adobe was attempting to do and refused to play that game).

You do not need either DNG or XMP formats to maintain a decent raw workflow. Actually, saving and resaving data in persistant memory in a file format is a rather wasteful and silly way to do things.

[BigMackCam]

You have basically pointed out the same flaw with XMP as with DNG : both are developments which are not so much targeted at ease of use, reliability or merit but purely on vendor lock-in by the creator of both "standards" (quotation marks as most other companies quickly realized what Adobe was attempting to do and refused to play that game).

You do not need either DNG or XMP formats to maintain a decent raw workflow. Actually, saving and resaving data in persistant memory in a file format is a rather wasteful and silly way to do things.

I'm open-minded to DNG and XMP not being the right approach, but what's the alternative? To use the native RAW files instead of DNG? That's not always possible...

An example: my Hasselblad HV is essentially a Sony A99 internally, but the manufacturer EXIF data is different, and it seems there are other subtle differences too regarding white balance and other information. Darktable fails when trying to load the native ARW files produced by the camera. But if I convert them to DNG, it loads them just fine (and everything looks good). In fact, the Darktable developers make a point of saying that even with unsupported cameras, it should load a DNG file. You might have to select a base curve manually, but otherwise it's unlikely to be a problem.

DNG might be Adobe's creation (and patented as such), but at least its an open standard - albeit a potentially flawed one. It doesn't seem to limit anything should another software vendor or camera manufacturer choose to support it...

[newmikey]

I
An example: my Hasselblad HV is essentially a Sony A99 internally, but the manufacturer EXIF data is different, and it seems there are other subtle differences too regarding white balance and other information.

Oh, I not bashing DNG/XMP for such niche-uses, far from it, just not as a viable 80/20 solution.

[emalvick]

Unfortunately, depending on the software you use, XMP files fail at this task. For example, Corel Aftershot (which is available on Linux) names the XMP files *.jpg.xmp . But Lightroom will name them *.xmp .
But that is only the start. Renaming the files doesn't help, even if you don't have name collisions (I do), because the format of the metadata inside those XMP files differs.
The keywords are written in different XML subsections. Aftershot has an option to export its catalog to so-called "standard" XMP files, but it's a one-way trip only.
I'm looking to move away from Aftershot, but I also don't want my metadata storage to depend on XMP files.
Lightroom can write the metadata into DNG and JPG files, though unfortunately, by default, it still uses XMP files for the metadata. You have to force metadata export to the JPG/DNG files.
Real PITA.

Do any of the open-source Linux image management programs allow reading/writing the keywords to DNG & JPG files, without requiring XMP files ?

So the XMP record (which can be in the file or as an external XMP file) is standardized through what I believe is called the Metadata Working Group.

So stepping back a bit, a good program for cataloging files should adhere to that standard. Now I don't know if that standard gets specific with regard to DNG files and whether metadata should be embedded in the file or not. I know that generally XMP is supposed to be stored externally for RAW files, but DNG is a gray area.

Now, in the case you discussed with Aftershot; technically, there should never be a .jpg.xmp file. But, I do think a few of the raw processor files think they are doing a favor creating an XMP record like that, which may not be a real XMP file? (I've never used Aftershot). Lightroom, as far as I know, generally conforms to the MWG standard, and will store metadata in the appropriate location in an XMP record be it in the file or separate. As far as it writing XMP files for JPG and DNG, I didn't think LR gave an option to do that. I thought by default it stored that information within the file? I rarely did it as I use a completely separate software for cataloging, but at one point I had wanted to see if I could make it do external XMP files, and I couldn't. This was back with LR 5 and 6.

The other thing with XMP files is that it is an XML format, and it can be extendable. So Adobe for instance does add its own spaces to store the edits made in LR or ACR. The information can't be used by any other software specifically, but it is backed up in the files metadata (XMP record, wherever that is stored, in the file or as a separate XMP file). A few other software will do the same thing or attempt to, like your Aftershot example.

But, how well the software takes care of standard metadata depends on the software. I don't recall the software, but one would write standard metadata in its own XMP namespace, which was useless for any other software. STandard metadata needs to be transferable and usable to almost any software (in my opinion). What good or keywords if they are only usable by one program, for instance, or ratings or color labels.

Back to the use of .jpg.xmp or similar... even though that isn't compliant with standards, the benefit is that if you have similar files (let's say a RAW + JPG) situation, at least you would have separate xmp files. But other software can't handle it, so while it may be a benefit, it is more of a hinderence unless you have software that can handle that quirk; the cataloging software I use does give flexibility for those situations, but it requires a lot of settings changes to get there.

Finally, writing xmp data to DNG in other software. As far as I've found, only Adobe and a few large commercial cataloging software support this. Digikam doesn't support nor do a few other programs I've tried. I recently tested out 6 to 10 programs with little to no success in finding those with the setting.

Like you, I've shot with DNG for years and the software I catalog with (IMatch) and LR appear to be two of the only programs that handle DNG files in this way. The problem with switching to external XMP files for DNG is that you would have to wipe out the XMP record within existing DNG files, otherwise there is a conflict. It's not an impossible task and perhaps not even "dangerous", but I'm uneasy doing that unless I really have to (and I don't).

As I mentioned before; external XMP files can be a blessing or a curse. They really streamline a backup process and the compliment a workflow where people want to do nothing to their raw files. Technically, any time you edit metadata within an image file, you risk a possibility of corrupting the file; this is a common argument from those that want RAW files to remain completely untouched. I've not had the issue myself, but it doesn't mean I won't. In that workflow, RAW files get backed up one time initially, and then the backup process is through XMP files only, which is quick because they are so small.

The curse side of XMP files is that you essentially end up with 2 files for every image. They have to remain paired with the image in your storage scheme, and if you intend on sending raw files around, copying, moving, you have to consider whether you need the metadata with the file. A good cataloging / DAM software will handle this situation easily, but it becomes more difficult if you are using basic file managers; and the XMP files make your folders look huge.

Ultimately, I'm not sure. In retrospect, I wish I would have just shot PEF files from the beginning and kept external XMP records because my backup process would be so much faster now than it is. But it sure is clean having all the DNG files which are more compatible with software. I've never had problems using DNGs in software that support them, even if they don't formally support my camera. They may be missing profiles, but at least I can open the files.

Enough rambling... Back to the question of writing to DNG files in Linux; the answer I think is no. JPG files should work, however. It's really a bad situation if software is writing XMPs to external files for JPG. Another thing to look for is a software that uses ExifTool as its underlying mechanism for metadata. It can handle almost anything, which would suggest you might be able to tweak settings to write to DNG if the software doesn't by default. I've not looked that closely at the Linux software out there.

[madbrain]

Now, in the case you discussed with Aftershot; technically, there should never be a .jpg.xmp file.

Unfortunately, there is ... The XMP stores not just the keywords but also all other non-destructive image processing directives that Aftershot can apply.

Lightroom, as far as I know, generally conforms to the MWG standard, and will store metadata in the appropriate location in an XMP record be it in the file or separate.

Lightroom definitely writes the keywords in the JPG, and there is no option to override that.

As far as it writing XMP files for JPG and DNG, I didn't think LR gave an option to do that. I thought by default it stored that information within the file? I rarely did it as I use a completely separate software for cataloging, but at one point I had wanted to see if I could make it do external XMP files, and I couldn't. This was back with LR 5 and 6.

For DNG, Lightroom Classic CC (current version) stores the metadata in XMP by default, but gives you an option to write it to the DNG file. The later has to be triggered manually (save metadata to file).

But, how well the software takes care of standard metadata depends on the software. I don't recall the software, but one would write standard metadata in its own XMP namespace, which was useless for any other software.

That's exactly what Aftershot does, unfortunately. Aftershot offers an option to write a "standard" XMP also, with standard namespace and filename. But that has to be triggered manually, also. And then you end up with 2 XMPs for each image, the Aftershot version (.JPG.XMP) and the standard version (.XMP).

STandard metadata needs to be transferable and usable to almost any software (in my opinion). What good or keywords if they are only usable by one program, for instance, or ratings or color labels.

I absolutely agree. That is why I think storing the metadata in DNGs themselves is attractive. I would like to find an open-source program that can do that, and does that by default.
I'm currently using Windows and don't anticipate changing that near-term. But many open-source programs have Windows versions, too.
Lightroom allows storing keywords in XMP, but it's still a manual process ...

Back to the use of .jpg.xmp or similar... even though that isn't compliant with standards, the benefit is that if you have similar files (let's say a RAW + JPG) situation, at least you would have separate xmp files. But other software can't handle it, so while it may be a benefit, it is more of a hinderence unless you have software that can handle that quirk; the cataloging software I use does give flexibility for those situations, but it requires a lot of settings changes to get there.

Yes, I'm aware of the name collision issue. For me, it is much more of a drawback. I really wish all programs could agree on one naming convention for XMP, and also use a common namespace for the keywords.

Finally, writing xmp data to DNG in other software. As far as I've found, only Adobe and a few large commercial cataloging software support this.

Yes, but even with Adobe, it is not automatic. It has to be triggered manually. By default, data goes to the LR catalog and/or XMP still.

Digikam doesn't support nor do a few other programs I've tried. I recently tested out 6 to 10 programs with little to no success in finding those with the setting.

That is unfortunate.

Like you, I've shot with DNG for years and the software I catalog with (IMatch) and LR appear to be two of the only programs that handle DNG files in this way. The problem with switching to external XMP files for DNG is that you would have to wipe out the XMP record within existing DNG files, otherwise there is a conflict. It's not an impossible task and perhaps not even "dangerous", but I'm uneasy doing that unless I really have to (and I don't).

Ah. I didn't think of this problem. That is bad indeed.

As I mentioned before; external XMP files can be a blessing or a curse. They really streamline a backup process and the compliment a workflow where people want to do nothing to their raw files. Technically, any time you edit metadata within an image file, you risk a possibility of corrupting the file; this is a common argument from those that want RAW files to remain completely untouched. I've not had the issue myself, but it doesn't mean I won't. In that workflow, RAW files get backed up one time initially, and then the backup process is through XMP files only, which is quick because they are so small.

I don't really mind the longer backup times, I'm not usually sitting there when they happen. But the possibility of getting metadata disassociated is a bigger problem. Especially if one is using more than one program. Realistically, it is not possible to use one program for everything.

Ultimately, I'm not sure. In retrospect, I wish I would have just shot PEF files from the beginning and kept external XMP records because my backup process would be so much faster now than it is. But it sure is clean having all the DNG files which are more compatible with software. I've never had problems using DNGs in software that support them, even if they don't formally support my camera. They may be missing profiles, but at least I can open the files.

Sadly, when it comes to Aftershot, and several other programs, like DXO, they will still not open a DNG file if they lack a camera profile. I was able to create a profile for my K-1 II DNG for Aftershot myself, fortunately. But with DXO, I'm still stuck with not being able to open them. Lightroom will open K-1 II DNGs, but uses default settings, with no PixelShift support. It will not open K-1 II PEFs. I'm assuming LR will come up with a profile with the K-1 II soon. Not sure about DXO.

Enough rambling... Back to the question of writing to DNG files in Linux; the answer I think is no. JPG files should work, however. It's really a bad situation if software is writing XMPs to external files for JPG. Another thing to look for is a software that uses ExifTool as its underlying mechanism for metadata. It can handle almost anything, which would suggest you might be able to tweak settings to write to DNG if the software doesn't by default. I've not looked that closely at the Linux software out there.

Exiftool is a good tool, but it is tedious to write scripts to do things the DAMs really should be doing ...

[derekkite]

The reason you don't want to write to the dng file is that you risk losing the image file if there is an error.