[house]

dcrawps is a modified version of dcraw that enables demosaicing of photographs taken using Pentax pixelshift. dcraw is software library and commandline tool for demosaicing or "developing" digital camera raw files.

On GNU/Linux the software can be compiled to a dcrawps executable file using the following flags

Code:

gcc -o dcrawps -O4 dcraw.c -lm -ljasper -ljpeg -llcms2

Basic information about how to use the software can be found at dcrawps and dcraw. The software has many options and this thread is intended as a place to discuss how dcrawps can fit into a workflow. Dcrawps is mentioned in other threads but no dedicated thread exists as far as i know.

---------- Post added 07-23-16 at 11:57 AM ----------

I fail to get results comparable to the in camera jpgs when using dcrawps and I'm curious how others have fared. A full commandline of mine might look as follows

Code:

dcrawps -v -w +M -6 -T -q 3 -m 3 -H 2 -G 20 $DNG

The resulting 16bit tiff is then imported into Darktable for further tweaking and sharpening. A breakdown of the options above

-v verbose
-w camera whitebalance
+M camera colour matrix (default with -w)
-6 16-bit output
-T write tiff files
-q 3 ahd interpolation (demosaicing)
-m 3 three pass cleaning of colour artefacts
-H 2 Highlights clipped
-G 20 pixelshift motion detection level

The reason for wanting to use dcrawps is to have the full 16-bit information to work with when for example raising the shadows. I do see colour improvements but the level of detail in my processed tiff's do not match the in camera jpgs.

Will post some photo examples later.

---

Last edited by house; 07-23-2016 at 12:00 PM. Reason: added

[jbinpg]

I posted a bit on dcrawps here: https://www.pentaxforums.com/forums/190-pentax-k-1/322677-can-you-extract-non...ml#post3684609

I find I get better results using darktable if I export from dcrawps using the -4 -T option which guarantees no gamma correction nor brighening is applied to the tiff output. Then when I bring it into darktable, I use the linear REC2020RGB input profile which gets it close to where it should be from linear format. Then I sharpen using the Equalizer module with good results.

Jack

[house]

Thanks for the link. Have done some more tests but still not quite there.

A "theoretical" question:
Should the files be demosaiced before merging?

Seems to me that they ideally shouldn't? You want clean bayer pixels from all images.

Dcrawps interpolate the images before merging according to the -v output. A reason for loss of quality?

[vonBaloney]

Is there a way to get raw raw pixel values from a RAW image? (Put aside pixel shift images for a minute -- just a single normal RAW image out of camera.) Meaning the actual luminance values from the sensor for each of the color channels, i.e. the non-interpolated data? When I looked into this previously, I found a few people asking the same question (they wanted to play with raw sensor data) and were always asked "why would anyone want to do that?" and told basically no, it's very complicated, just use the full-image demosaiced data. And I guess there are complicating factors to using the raw raw data -- I forget the details -- but was surprised that no one seemed to even know of a way to get at it. (Even with dcraw.)

I was interested in experimenting with altering the raw sensor data BEFORE it was demosaiced and was frustrated that there didn't seem to be any tools available to make that possible, or even any guidance as to what could be done. (And only a couple of people seemed to think that there could possibly be anything interesting to do there, but maybe I was looking in the wrong circles.) I mean, if I wanted to make my own RAW converter from scratch, where would I start? The possibility of working with a set of pixel shifted raw datasets makes it even more intriguing.

[wombat2go]

Meaning the actual luminance values from the sensor for each of the color channels, i.e. the non-interpolated data?

Von, Yes, here is how I do it:

https://www.pentaxforums.com/forums/groups/197-pentax-linux/3948-view-pentax-...ray-linux.html

[stevebrot]

A "theoretical" question:
Should the files be demosaiced before merging?

If you don't want lines, yes. Good question...and I suspect the answer is not simple. Constructing pixel shift is not a simple merge in that the "real" recorded values are used rather than interpolation to "fill in the blanks". That is where it differs from so-called super-resolution process.

Is there a way to get raw raw pixel values from a RAW image?

If you mean the sensor site voltages...no.

If you mean the 14 bit/channel output from the camera's image processor expanded to 16 bits...yes. This may be done with the dcraw or dcrawps with appropriate options.


Steve

[stevebrot]

I mean, if I wanted to make my own RAW converter from scratch, where would I start?

I would look at the source for dcraw/dcrawps and fork the project for your use case.


Steve

---

Last edited by stevebrot; 07-26-2016 at 05:58 PM.

[vonBaloney]

What I would MOST like to do is alter raw color data as if it had been captured that way originally, then be able to write a new valid RAW file with that altered data included (a DNG, presumably), and then import that into Lightroom (or other) just like any other file out of the camera. I made some headway with Adobe's DNG SDK, but the docs were not up-to-date with the actual version (it won't compile even following their own instructions, it was a bit of a mess), plus it sounded like there might be a checksum involved so if you altered data it would flag the file as invalid. So it was just getting over my head, but it is still a project I'd like to pursue at some point. Using pixel shifted images I'd like to do the same thing -- draw data from each to make a new normal (Bayer-type) valid RAW file.

[wombat2go]

If you mean the sensor site voltages...no.

From what I read, up until the latest pre-pixel-shift cameras, the raw files contain the well voltages measured by the adc's.
The nexus for the various computer based raw converters has been only the 3 * 3 array of Bayer coefficients, either inferred or obtained from the camera manufacturers.

If / when that is no longer true ( ie, when the cameras start to pre-process the raw files) the raw converters are going to need the additional info from each manufacturer so that users can display and edit the various raw files.

[stevebrot]

So it was just getting over my head, but it is still a project I'd like to pursue at some point. Using pixel shifted images I'd like to do the same thing -- draw data from each to make a new normal (Bayer-type) valid RAW file.

You are correct about the checksum. I will go out on a limb here and suggest that creating a new single-image DNG/PEF from a PxS DNG or PEF might not be worth the trouble or even be representative of what you are wanting. A TIFF constructed from the PxS data using that algorithm should.


Steve

[wombat2go]

What I would MOST like to do is alter raw color data as if it had been captured that way originally, then be able to write a new valid RAW file with that altered data included (a DNG, presumably), .

If you follow the steps of my link, you will end up with a .txt file containing the array of well voltages.
There are various ways you could read and do number crunching on that array, although I have not tried it.

However this has already been done in the photo processors, it would really be re-inventing the wheel.
For example the latest Darktable on linux uses opencl to do advanced number crunching, fast on the gpu.
Darktable is difficult to learn , but it does some real fancy stuff.
Of interest is the parametric selection of pixels to act on.

[vonBaloney]

Basically I'd want the step before interpolation, although after thinking it about a bit decided the experiments I most want to do could probably be done in a form even with interpolated data and I'd just use RGB pixels values or whatever like a normal person who might be writing a image processing plugin. But I kinda also wanted to see the effect ON THE INTERPOLATION of altering the raw Bayer data in certain ways. I'd have a lot to learn in any case to do any of it. I'm a programmer, but not really my area, and I've always avoided real low-level stuff so when I get too into the weeds with compilers and such I know I will have to take a month or two just to get my bearings. Anyway, thanks for the guidance...

[stevebrot]

From what I read, up until the latest pre-pixel-shift cameras, the raw files contain the well voltages measured by the adc's.

Not my understanding, but also not my area of particular interest or expertise.


Steve

(...yes, I did read your linked post and yes, I do understand what you did...)

[house]

Was distracted* for a bit but I've finally done a comparison between dcrawps+darktable and DCU 5.6.1

Below are 100% 500x500px crops taken with the following settings

• Tripod
• K-3 II
• 1/125s
• f/5.6
• iso 100
• DA SMC 70mm limited
• A distance of approx 25m

If the order is maintained after uploading the first image is processed using Dcrawps+Darktable the second using pixelshift dcu and the third pixelshift off dcu. I used the following dcrapws command

Code:

dcrawps -v -w +M -6 -T -q 3 -m 3 -H 2 -G 20 $filename.DNG

My initial impression is that you can actually get better results using Darktable and Dcrawps than you can using DCU. I am however used a DCU newbie and rather experienced using Darktable. Note that the DCU files are over sharpened but still lack detail. I've applied some noise reduction in Darktable despite shooting iso 100




*Had to install a windows virtual machine which wasn't much fun and made me put the whole thing off.

[stevebrot]

Was distracted* for a bit but I've finally done a comparison between dcrawps+darktable and DCU 5.6.1

Below are 100% 500x500px crops taken with the following settings

• Tripod
• K-3 II
• 1/125s
• f/5.6
• iso 100
• DA SMC 70mm limited
• A distance of approx 25m

If the order is maintained after uploading the first image is processed using Dcrawps+Darktable the second using pixelshift dcu and the third pixelshift off dcu. I used the following dcrapws command

Code:

dcrawps -v -w +M -6 -T -q 3 -m 3 -H 2 -G 20 $filename.DNG

My initial impression is that you can actually get better results using Darktable and Dcrawps than you can using DCU. I am however used a DCU newbie and rather experienced using Darktable. Note that the DCU files are over sharpened but still lack detail. I've applied some noise reduction in Darktable despite shooting iso 100




*Had to install a windows virtual machine which wasn't much fun and made me put the whole thing off.

Thanks! This is useful information.


Steve