RAW Technical Question

[baby duck monge]

Here's a question for all you highly technical digital photogs.

My understanding is that, if you shoot in RAW, you end up with a file that cannot in and of itself be changed. What I mean is that you can edit the file, but you cannot save your changes without creating a new file (so if you shoot in RAW, it is impossible to edit the image, save your edits, and end up with only one file). Is my understanding correct?

For bonus points, can you direct me to any technical documents (or I guess any reliable and cite-able documents) that state my understanding? You can get even more bonus points if your document explains why RAW files work this way and/or explicitly state that the underlying RAW file cannot be altered.

Any guidance in any of these directions would be awesome. Thanks!

 

[ChrisA]

My understanding is that, if you shoot in RAW, you end up with a file that cannot in and of itself be changed. What I mean is that you can edit the file, but you cannot save your changes without creating a new file (so if you shoot in RAW, it is impossible to edit the image, save your edits, and end up with only one file). Is my understanding correct?

That is certainly NOT true. Yes there are some non-destructive RAW editors but they are only non-destructive because that is what users want now. So in practice if you use Aperture your raw files don't get changed.

 

[baby duck monge]

That is certainly NOT true. Yes there are some non-destructive RAW editors but they are only non-destructive because that is what users want now. So in practice if you use Aperture your raw files don't get changed.

Are there currently any destructive editors? As a follow-up question: if there are currently any destructive editors, is there any way of knowing whether the original RAW file has been edited?

 

[robbieduncan]

It would be technically possible to write software to edit the file and write the data back in the same format, but the question would be why would you? The file would no longer be the RAW sensor data, but edited data and the format itself is significantly sub-optimal for storing data!

To write back you'd have to re-mosaic the colour data, undo the effects of the whitebalance shift, undo any default sharpening applied (there is always some to get around the inherent unsharpness after de-mosaicing)... So lets say your only edit was to add some sharpening: this would be lost be re-saving as RAW.

 

[snickelfritz]

RAW files are simply the unprocessed "raw" image data taken directly from the camera CMOS or CCD.
This allows you to make adjustments to the image that would be impossible with a composited and processed JPEG or TIFF.

RAW files essentially allow you to do in your computer, that which would normally be done within the limited capabilities of the camera processor.
The result is much finer control over the quality of your photographs.
ie: a JPEG generated from a RAW file is potentially much higher quality than a JPEG generated automatically by the camera.

The downside is that you have store and manually process the RAW files.

 

[baby duck monge]

Ok. I appreciate the general descriptions of what RAW is and the questions as to why one would want to directly edit RAW files, but those are not really what I'm after here. I know what RAW files do, I know their benefits, and I know their downsides.

I am not wondering if I should switch to shooting in RAW (which I do about half the time). I am not asking how you edit your RAW files after you take your shots (I use Photoshop). I am not trying to find some way to edit RAW images without creating a second file (actually, I kinda am, but not for any sort of personal use). At this point what I need to know is if there are any programs available right now to edit a RAW file directly, and to save those changes over the original file. And if so, if there would be any good way to detect such changes.

 

[robbieduncan]

I am not trying to find some way to edit RAW images without creating a second file (actually, I kinda am, but not for any sort of personal use). At this point what I need to know is if there are any programs available right now to edit a RAW file directly, and to save those changes over the original file. And if so, if there would be any good way to detect such changes.

As I pointed out above: you would not want to do this. The file format is not pixel level as such and requires significant "interpretation" to turn it into a visible image. Saving your edited files back to this format would be a bad idea which is why nothing does it. If you are really that fussed just turn everything into a TIFF and throw the RAW away.

 

[nutmac]

In film photography, cameras capture images onto a strip of film. At the lab, the film is developed into negatives. Print positives are then created from the negatives.

In digital photography, RAW is the negative. You do not tinker with the original negatives. Many news organizations accept only the RAW as the format is difficult to manipulate (many RAW formats are difficult to reverse engineer and encrypted). You tinker with the TIFF/PSD/PNG/JPEG file, the "positives" of digital photography.

As robbieduncan pointed out, if you prefer to keep only the highest quality processed images, create 12- or 14-bit per channel (whichever duplicates your camera's RAW capability) PSD or 16-bit per channel PNG out of RAW and throw out the RAW. (I don't think TIFF is capable of 12- or 14-bit per channel.)

 

[snickelfritz]

Changes to the data in a RAW file are stored externally; not within the file itself.

What is the basis for your question?
If you're looking for a way to archive your RAW images within a single file (no sidecar or database files), DNG is probably the way to go.
Changes to the RAW file data can be embedded in a DNG, and extracted/edited later using the ADOBE DNG Converter.

If the purpose of this information is to support or refute the validity of RAW versus composite digital images in a court of law, you probably should hire an accredited expert on this subject.

 

[robbieduncan]

(I don't think TIFF is capable of 12- or 14-bit per channel.)

TIFF is capable of 16-bits per channel.

 

[ChrisA]

At this point what I need to know is if there are any programs available right now to edit a RAW file directly, and to save those changes over the original file. And if so, if there would be any good way to detect such changes.

The trouble is that "RAW" refers to a large class of file formats. Each camera manufacturer has it's own format and they are not at all alike. Even worst the exact format of the file depends on the camera model. So we can't give a general answer. It might be possible to edit and save over a Nikon D80 NEF file but not a Nikon D300 file now but in two weeks this might change when a new software revision is released.

Technically I could always use a binary file editor and change any byte in any file. These binary editors don't know a RAW image file from an MS Word document. They work at the byte level. So yes, you can change any computer file but it would be "way hard" to make anything but a random change that way

Nikon sells image editing software and it uses the nikon "NEF" rwa file as it's native file format. You make changes and then sae those change back to the raw NEF file. So at least with Nikon. You can change the files. But, and I think this is what yu are asking, The NEF files written out are not the same as the ones the camera writes out. The edits are stored in such a way that they can ba un-done. You can download a 30 day tial version from the nikon web site

Adobe has a raw format too. They call it DNG. I'm pretty sure photoshop can save to DNG format. So if your camera saves files in DNG format Photoshop could be used as a destructive RAW editor. I think there are at least a half dozen camera companies that use DNG.

One last comment. Some raw conversion programs might allow you to over write the raw file. You would have to look at each program. there are quite a few to look at. there is an in-complete list of raw conversion programs on the dcraw site. Look here: http://cybercom.net/~dcoffin/dcraw/ scroll down to "other raw converters" for a list of about 30 programs. But the list is not complete. It leaves out major players like Lightroomand Bibble and the software that ships with Nikon and canon cameras

 

[ghostchild]

adobe lightroom photoshop or iphoto

 

[baby duck monge]

I DO NOT WANT TO KNOW WHAT RAW DOES, WHAT PROGRAMS THERE ARE TO WORK WITH RAW FILES, OR WHY EDITING OVER A RAW FILE WOULD BE A BAD IDEA! All I want to know is (1) if there is any technical limitation to prevent you from writing over a RAW file with an edit, (2) if there are currently any programs that would allow you to do this, and (3) if there would be a way to detect any such edits to the RAW file.

My interest in this is an academic one. I am contemplating writing an article, and this information would be relevant to that article. I have no desire to do this with any of my own images, I just want to know whether or not it can be done.

Let me reiterate this. I want to know this information for purely academic purposes. I do not care if you think editing RAW images and saving them over the original file is a bad idea or if it defeats the purpose of creating a RAW image in the first place. I just need to know if there are any technical limitations, and if there are currently any programs that would allow you to do it.

 

[snickelfritz]

Hmm...

Your question, as I understand it:

"is it possible to alter the data in a camera RAW file, then pass it off as an original, without being detected?"

Why don't you contact Thomas Knoll about this?
He's probably the leading authority on the subject.
tknoll@adobe.com

 

[compuwar]

I DO NOT WANT TO KNOW WHAT RAW DOES, WHAT PROGRAMS THERE ARE TO WORK WITH RAW FILES, OR WHY EDITING OVER A RAW FILE WOULD BE A BAD IDEA! All I want to know is (1) if there is any technical limitation to prevent you from writing over a RAW file with an edit,

No, there are no technical limitations. I perform computer forensic analysis of drives and data for part of my living.

(2) if there are currently any programs that would allow you to do this, and

Yes, there are programs that allow you to do this. Nikon's CaptureNX will allow edits and saves of .NEF files, for example.

(3) if there would be a way to detect any such edits to the RAW file.

Not easily, the same sort of sampling variation found in the recent (last Aug/Sep) JPEG alteration discovery processes may be applicable, though the quantization table stuff obviously doesn't apply. Metadata analysis may also help show modification, assuming the alteration wasn't actively malicious (edit: or the malicious actor didn't know what they were doing or have low-level access to the storage device.) Of course, it depends on the editing done, and the tools used to edit. There's a huge difference between "could this file have been altered" and "is there a reasonable doubt that this file contains and unedited image." for instance.

My interest in this is an academic one. I am contemplating writing an article, and this information would be relevant to that article. I have no desire to do this with any of my own images, I just want to know whether or not it can be done.

Of course it can be done, it's only data in a file.

Let me reiterate this. I want to know this information for purely academic purposes. I do not care if you think editing RAW images and saving them over the original file is a bad idea or if it defeats the purpose of creating a RAW image in the first place. I just need to know if there are any technical limitations, and if there are currently any programs that would allow you to do it.

You can do it, and it's also probably not all that difficult to use the dcraw source to remap back if you have to do it for multiple raw file formats.

I'd say that David Coffin knows more about raw files than Thomas Knoll, I've been told Adobe has used dcraw in Photoshop.

 

[npederse]

I guess I don't know the technical aspects on how an editor, such as CaptureNX saves the files and if they are "easily" detecable as to the differences. My thought is that most any program that is normally used to edit photos would leave some traces that are detecable, though would involve some delving into the file itself.

I do know that Canon (and Nikon, ??I'd assume, but don't know) sell data verification modules that work with their pro series bodies. This add-on allows data verification for court cases, etc where it is required that there is a chain of custody/tampering needed.

Hope that may be helpful.

 

[biffcliff]

Like "compuwar" wrote before, a data is a data is a data. Of course can you manipulate any digital data.
But who knows about how ?

All I want to know is (1) if there is any technical limitation to prevent you from writing over a RAW file with an edit, ..

(1) I've read the hole thread above, but there is an hidden point the earlier post which is important. Any RAW File contains, linear data taken from the camera's chip, likely optimised, such as noise reduction. I guess otherwise Canon would not be able to show such good results which CMOS based chips.
Anyway, the point is, that a RAW file literally doesn't contain an image at all !
If you would see the result without processes called:
e.g.
- demosaicing (demosaiking)
- gammacorrection
- noise reduction
- colorspace transformation
you wouldn't call it an image.

So, what kind of changes do you want to make ?
For me, as an developer on RAW Images, I would be cool to have a tool like that. Believe me, for an regular use, there is no use. For me I would be great to manipulate the input data of my own algorithm (my own RAW Converter).
That's actually the point, you need a programm, an converter, the create an image out of the data you get from the camera!

(2) if there are currently any programs that would allow you to do this, and

(2) I am sure that there is no open-program like that, perhaps Nikon, Canon ect. have something in their labs to use it.

(3) if there would be a way to detect any such edits to the RAW file.

If you want to detect changes on a file, use MD5 (see wikipedia).

My interest in this is an academic one. I am contemplating writing an article, and this information would be relevant to that article. I have no desire to do this with any of my own images, I just want to know whether or not it can be done.

Let me reiterate this. I want to know this information for purely academic purposes. I do not care if you think editing RAW images and saving them over the original file is a bad idea or if it defeats the purpose of creating a RAW image in the first place. I just need to know if there are any technical limitations, and if there are currently any programs that would allow you to do it.

 

[baby duck monge]

Why don't you contact Thomas Knoll about this?
He's probably the leading authority on the subject.
tknoll@adobe.com

<lots of stuff>

I do know that Canon (and Nikon, ??I'd assume, but don't know) sell data verification modules that work with their pro series bodies. This add-on allows data verification for court cases, etc where it is required that there is a chain of custody/tampering needed.

Hope that may be helpful.

Thank you, thank you, and thank you. This is exactly the sort of thing I was looking for. I will definitely check into the leads you have provided.

 

[ChrisA]

Let me reiterate this. I want to know this information for purely academic purposes.

If this is purely academic then we can settle it quickly. Send me a raw file and I will modify it and send it back to you. The file will look exactly the same except that one pixel will have a slight different color, so slight your eye would not notice. It's easy to do this. I'd use a binary editor to change one bit in one pixel.

Now if the question were practical not academic then I'd say that a normal photographer in the course of any reasonable work flow would have a hard time accidentally modifying his raw file in an un-detectable way.

If you want files (of any file type, images, text, spreadsheets,..) were modification is detectable then you will have to append a "digital signature" (aka "cryptographic hash") to the file and then the file will become secure from that point forward as long as the cryptographic key used to compute the hash is not disclosed. People who do this then have the burden of forever needing to keep to key under their exclusive control so they can be sure it is not disclosed. If compromise of the key is ever in question then the files also can be questioned.

Seeing as your question was academic, the technically correct answer is that ANY file lacking a good cryptographic hash can never be proven "unmodified". Resaid without the multiple negatives: "Only with a hash can you proove the file is unmodified" RAW files lack this hash threrfore ...... QED.

 

[compuwar]

Seeing as your question was academic, the technically correct answer is that ANY file lacking a good cryptographic hash can never be proven "unmodified". Resaid without the multiple negatives: "Only with a hash can you proove the file is unmodified" RAW files lack this hash threrfore ...... QED.

Academically, this simply isn't true. First of all, even with a sound cryptographic hash, all you can "prove" is that the file matches the hash. This doesn't protect you from "proving" the file isn't a collision , that collides on the given hash with another image (the ranges of inputs are unlimited (anything in any file) the range of outputs are limited to the output size of the hash, so there are multiple possible inputs for a given output hash value- these "collisions" are inherent in any fixed-length hash.)

Also, a digital signature is a hash protected by encryption, not simply a hash itself, otherwise you can not detect tampering of the hash value.

Lastly, you can "prove" a file is unmodified in other ways- such as escrow and for certain types of modification, alternative images.

I'm doing computer forensics on a criminal case now where one of my tools didn't produce the same hash value as a second tool, and it's going to take me three days to figure out where the problem is and validate everything- but I can't proceed with my analysis until I get matching hash values. While this stuff isn't exactly rocket science, trying to give a technical answer means getting it technically correct, and lots of people understand the theory, but don't know how that translates into practice.

 

[baby duck monge]

Now if the question were practical not academic then I'd say that a normal photographer in the course of any reasonable work flow would have a hard time accidentally modifying his raw file in an un-detectable way.

I was mainly shooting for a determined photographer in the course of a work flow intending to make specific changes to a RAW image (as you might do with Photoshop, but not necessarily using Photoshop). Apparently there are editors that will allow you to directly modify a RAW image, so now the question comes down to what the telltale signs of such manipulation would be (if any). If there are no signs other than the typical signs of editing a JPEG or similar, my inquiry ends there. If there are other signs, I'm interested. I plan on trying to contact Mr. Knoll to see what he has to say.

 

[compuwar]

I plan on trying to contact Mr. Knoll to see what he has to say.

Keep in mind that Thomas Knoll isn't the best person to ask- he's focused on editing images. David Coffin has deconstructed more raw files than probably anyone- but the research in modification detection isn't his either.

When I keynoted the RCFG/HTCIA computer forensics conference at GMU last year, I cited the work of Neal Krawetz. You should really start by reading his published work from last year, even though the quantization table part isn't directly relevant to raw files, the other analysis methods seem as if they might be.

Steganography researchers do more analysis than most folks, Neil Johnson comes to mind as someone who might be able to give an authoritative answer, the folks at SARC might be helpful too. I'd follow up Krawetz's work with reading some of Johnson's stego overview material.