The previous article, DNG Workflow / Part I outlined the basic use of Adobe DNG Converter for converting proprietary Raw files into DNG files. While useful for a variety of reasons, it does indeed add a step in a Raw processing workflow.
So, what if you used Adobe DNG Converter right from the very beginning of your Raw processing workflow–Image Ingestion?
As most experts will tell you, trying to download camera files directly from the camera isn’t ideal. Aside from communication issues between cameras and operating systems, a camera mounted as a storage device is rather limited in both functionality and speed. My personal preference is to download images directly from the compact flash card via a FireWire card reader. Most people will make a copy of either the DCIM folder containing the captures or go to some length to set up a file copying system to avoid issues of file over-writing. Often this has been a procedure of creating folders in advance of actually copying the Raw files.
This stage–where your only image copy is on the compact flash card is hyper-critical.
Bruce Fraser in his new Real World Camera Raw with Adobe Photoshop CS2 edition has named this the “Image Ingestion” stage. It’s the first of four stages in a Raw processing workflow. Bruce says, “If you screw up during the ingestion phase, you run the significant risk of losing images, because at this stage, they only exist on the camera media”.
Because of the limits in the formatting of compact flash cards and microdrives in the FAT-based format, the enclosing DCIM folder contains subfolders where your images are written. With today’s cards, it’s not at all unusual to have multiple subfolders inside your DCIM enclosing folder depending on camera make and file format. Organizing these folders and subfolders for secure copying to your hard drive can be tedious and error prone. However, it’s mission critical if you are to secure the image ingestion stage.
For that very reason I have started working with Adobe DNG Converter from the very beginning-the point of first ingestion.
Looking again at the DNG Converter main dialog, you can point the DNG Converter directly at the CF card at step 1. For ease of editing and consolidating a shoot, I suggest NOT to preserve these tedious subfolder directories in Step 2. Additionally, at the ingestion stage you can set up a file naming and numbering scheme. This will help organize a multi-card shoot.
It should be noted that the naming and numbering scheme is sticky between DNG Converter processes but will auto-increase the numbering. So, if the first card had 200 images processed in a batch, a second card will automatically be numbered starting with the next number in the sequence, or number 201. It should also be noted that by default, DNG Converter will auto-resolve naming conflicts. So, rather than actually overwrite one file with another file, DNG Converter will append a number at the end of the second file with the same name to avoid a naming conflict. This is a major safety valve for a busy shooter who might otherwise make a mistake that could cause lost images during an image ingestion.
You are limited to either drop down presets or custom text strings in the renaming structure. However, DNG Converter’s renaming is far superior to that of basic finder copying. Between the elimination of the subfolders and the ability to rename upon ingestion, several critical aspects of the ingestion stage can be, in essence, automated. It is far easier and potentially safer to do the ingestion while combining the DNG conversion in one simple process. A finder copy procedure is complicated and potentially much riskier.
Compare the folder structure of the original CF card’s DCIM folder and subfolder above with the better organized folder structure and file naming and numbering shown below.
In addition to a better ingestion organization, you gain the benefit of any file compression savings and start off with files that will accept metadata and Camera Raw settings without resorting to XMP side car files.
The original DCIM folder contained 70 files for a total of 492 MBs.
The copied and converted DNG files are 393.9 MBs. A savings of about 20%. Note, your image will vary based upon the camera and type of raw files it produces. With some cameras that use lossy compression (such as the Nikon D70) you may see little reduction or even potentially larger files. Also note that if you choose to embed the original raw file inside the DNG, the file will be larger than the original file size.
Another little known benefit of using the Adobe DNG Converter is the ability to do a simple verification that the raw file you are converting is not corrupted. If the DNG Converter encounters a file that has problems, it will throw an alert error in the Conversion Status dialog as shown above.
While rare, it’s an early warning sign that there may be problems with either that file or the directory structure of your compact flash card. Generally a simple finder copy will not show an error or problem when copying a corrupted image file-it merely copies the corrupted file and you won’t discover the problem until later when trying to open the file. Note that this warning is not a complete fail safe mechanism. It’s merely intended to alert you to a potential problem and indicates which file or files are involved. This will give you a warning that perhaps stronger recovery may be needed before removing the card and reformatting it.
The same file that popped the error in the DNG Converter returned this error when opening in Camera Raw.
The problem file did however produce both a thumbnail and a preview in Bridge. It gave every indication that the file could be read, but when trying to open the file in Camera Raw, it failed.
And that brings up the next stage that Bruce writes about, Image Verification. If anything happens at image ingestion that causes a problem with the process of copying from the CF card to your local hard drive, you need to know about it before you pull the CF card and reformat it. While not foolproof, as indicated in the case of the file above, anything you can do to help ensure you’ve successfully verified an image ingestion is very useful.
As you can see, there are various potential advantages to using the Adobe DNG Converter at the very beginning of a raw processing workflow. The big question, is what does it cost in time?
Working on a dual 2GHZ G5 Macintosh with 6 gigs of ram and a FireWire connected CF card reader, I ran timing tests on the card containing the images discussed above. The card was a Lexar 2GB 80X card with images shot by a Canon EOS 1D MII. There were 70 files shot in Raw only for a total of 492.4 MBs.
Making a finder copy via FireWare to the desktop took about 38 seconds. Fairly quick actually due in large part to the faster FireWire connection of the card reader. A USB 2.0 card reader would be about the same.
Taking the same card and doing a DNG conversion to lossless compression with no embedded DNG took 2:27 seconds. About a two minute penalty for the copy and convert to DNG.
Calculated out, that was about 2 seconds/image for the copy and conversion to DNG. Compare that to a simple finder copy which took just over .5 seconds/image and you might conclude that based only on speed, the DNG Converter takes too long. But let’s examine what you get with the DNG file vs the proprietary raw file. First off, you get a fully publicly documented DNG file that can safely take file renaming, metadata embedding and is likely smaller in file size than the original raw file.
Secondly, you easily solve the problems caused by CF card file organization structures and can use the DNG Converter as a preliminary image verification file check. Since the renaming and renumbering can help organize a shoot better, you’ll also potentially save time and help minimize the risk of overwriting files when ingesting. How much time will you save? That all depends upon how much time you would otherwise spend on file organization and renaming, but potentially, you save up to equal the time spent on the initial DNG Conversion. That’s a tough call because photographers have evolved all sorts of different file naming conventions and methods of safe image ingestion. So your milage may vary, a lot.
Was the free Adobe DNG Converter designed as a pro level image ingestion tool?
Not really. It was only designed to allow photographers to explore the potential of using DNGs. As a free software release from Adobe, it doesn’t at this point have a lot of bells and whistles built in. However, it could become the backbone of a pro level ingestion tool with only a little bit of work. At this point, DNG Converter is neither scriptable nor controllable by command line. So, the ability to automatically control or build scripts around DNG Converter isn’t there yet. Thomas Knoll has stated that command line control or scriptability is an often requested feature and it’s a request that several alpha and beta testers of Photoshop CS2 and Camera Raw 3.x have made. If enough people start adopting a DNG workflow and more people ask for it, I suspect the odds of getting scripting or command line control will increase considerably. (hint, hint)
The relative merits of adopting a DNG workflow really go beyond what it may have in store for you personally. As I’ve outline in Part I and now in Part II, there are some tangible benefits to adopting this workflow. But there are some real intangible benefits as well. There are potential benefits to the entire digital photography community.
It’s no surprise that the issue of undocumented and proprietary raw file formats has received a lot of attention of late. As more and more photographers go digital, more people are discovering some of the, uh, well, thornier sides of shooting digital. A proprietary raw workflow is something that complicates a photographer’s daily workload and as I’ve written in my Digital Preservation article here on PSN, the long term viability and sustainability of undocumented and proprietary raw file formats puts all of our work at risk.
If the digital photographic community adopted a useful and beneficial DNG workflow, I suspect that the odds of the camera companies looking harder at DNG as a viable solution would go up–potentially a lot. I would also suspect that if DNG evolved as the standard for storing raw image files, the long term prospects of DNG surviving as a sustainable format would also go up.
I’m weighing the risks involved. I do honestly believe that at some point in the future (3-5 years at the longest) the camera companies will have no choice but to adopt standard raw file formats. Will it be DNG or some flavor they concoct themselves? I don’t know, but I do think the greater the adoption of DNG now the more likely the camera companies will be encouraged to adopt DNG later.
The more photographers adopt DNG, the greater the pressure will be to have mainstream adoption of DNG industry wide. In that regard, I see individual photographers having a great deal of influence over their own future. I choose to cast my vote for DNG. In my mind, there is currently no reason not to technically and plenty of reasons to do so politically. . .but it’s up to each of you to make your own decisions.
For more information about DNG, see the PSN article Information About Adobe DNG
Additional DNG resources:
The DNG Home Page
The DNG User to User Forum
(free registration required to post)
rawformat.com‘s recent listing of 3rd party support for DNG
The OpenRAW.org Working Group
The Raw Flaw article by Michael Reichmann and Juergen Specht
Understanding Digital Raw Capture by Bruce Fraser (direct 1000K PDF link)
DNG reviews and awards
Adobe Camera Raw Home Page