PhotoshopNews.com
May 25, 2005

DNG Workflow / Part II

 

The previous article, DNG Workflow / Part I outlined the basic use of Adobe DNG Converter for converting proprietary Raw files into DNG files. While useful for a variety of reasons, it does indeed add a step in a Raw processing workflow.

So, what if you used Adobe DNG Converter right from the very beginning of your Raw processing workflow–Image Ingestion?

As most experts will tell you, trying to download camera files directly from the camera isn’t ideal. Aside from communication issues between cameras and operating systems, a camera mounted as a storage device is rather limited in both functionality and speed. My personal preference is to download images directly from the compact flash card via a FireWire card reader. Most people will make a copy of either the DCIM folder containing the captures or go to some length to set up a file copying system to avoid issues of file over-writing. Often this has been a procedure of creating folders in advance of actually copying the Raw files.

This stage–where your only image copy is on the compact flash card is hyper-critical.

Bruce Fraser in his new Real World Camera Raw with Adobe Photoshop CS2 edition has named this the “Image Ingestion” stage. It’s the first of four stages in a Raw processing workflow. Bruce says, “If you screw up during the ingestion phase, you run the significant risk of losing images, because at this stage, they only exist on the camera media”.

Because of the limits in the formatting of compact flash cards and microdrives in the FAT-based format, the enclosing DCIM folder contains subfolders where your images are written. With today’s cards, it’s not at all unusual to have multiple subfolders inside your DCIM enclosing folder depending on camera make and file format. Organizing these folders and subfolders for secure copying to your hard drive can be tedious and error prone. However, it’s mission critical if you are to secure the image ingestion stage.

For that very reason I have started working with Adobe DNG Converter from the very beginning-the point of first ingestion.


Click on the image to see larger sized image in a new window.

Looking again at the DNG Converter main dialog, you can point the DNG Converter directly at the CF card at step 1. For ease of editing and consolidating a shoot, I suggest NOT to preserve these tedious subfolder directories in Step 2. Additionally, at the ingestion stage you can set up a file naming and numbering scheme. This will help organize a multi-card shoot.

It should be noted that the naming and numbering scheme is sticky between DNG Converter processes but will auto-increase the numbering. So, if the first card had 200 images processed in a batch, a second card will automatically be numbered starting with the next number in the sequence, or number 201. It should also be noted that by default, DNG Converter will auto-resolve naming conflicts. So, rather than actually overwrite one file with another file, DNG Converter will append a number at the end of the second file with the same name to avoid a naming conflict. This is a major safety valve for a busy shooter who might otherwise make a mistake that could cause lost images during an image ingestion.


You are limited to either drop down presets or custom text strings in the renaming structure. However, DNG Converter’s renaming is far superior to that of basic finder copying. Between the elimination of the subfolders and the ability to rename upon ingestion, several critical aspects of the ingestion stage can be, in essence, automated. It is far easier and potentially safer to do the ingestion while combining the DNG conversion in one simple process. A finder copy procedure is complicated and potentially much riskier.


Click on the image to see larger sized image in a new window.

Compare the folder structure of the original CF card’s DCIM folder and subfolder above with the better organized folder structure and file naming and numbering shown below.


Click on the image to see larger sized image in a new window.

In addition to a better ingestion organization, you gain the benefit of any file compression savings and start off with files that will accept metadata and Camera Raw settings without resorting to XMP side car files.


The original DCIM folder contained 70 files for a total of 492 MBs.


The copied and converted DNG files are 393.9 MBs. A savings of about 20%. Note, your image will vary based upon the camera and type of raw files it produces. With some cameras that use lossy compression (such as the Nikon D70) you may see little reduction or even potentially larger files. Also note that if you choose to embed the original raw file inside the DNG, the file will be larger than the original file size.


Click on the image to see larger sized image in a new window.

Another little known benefit of using the Adobe DNG Converter is the ability to do a simple verification that the raw file you are converting is not corrupted. If the DNG Converter encounters a file that has problems, it will throw an alert error in the Conversion Status dialog as shown above.

While rare, it’s an early warning sign that there may be problems with either that file or the directory structure of your compact flash card. Generally a simple finder copy will not show an error or problem when copying a corrupted image file-it merely copies the corrupted file and you won’t discover the problem until later when trying to open the file. Note that this warning is not a complete fail safe mechanism. It’s merely intended to alert you to a potential problem and indicates which file or files are involved. This will give you a warning that perhaps stronger recovery may be needed before removing the card and reformatting it.

The same file that popped the error in the DNG Converter returned this error when opening in Camera Raw.

The problem file did however produce both a thumbnail and a preview in Bridge. It gave every indication that the file could be read, but when trying to open the file in Camera Raw, it failed.

And that brings up the next stage that Bruce writes about, Image Verification. If anything happens at image ingestion that causes a problem with the process of copying from the CF card to your local hard drive, you need to know about it before you pull the CF card and reformat it. While not foolproof, as indicated in the case of the file above, anything you can do to help ensure you’ve successfully verified an image ingestion is very useful.

As you can see, there are various potential advantages to using the Adobe DNG Converter at the very beginning of a raw processing workflow. The big question, is what does it cost in time?

Working on a dual 2GHZ G5 Macintosh with 6 gigs of ram and a FireWire connected CF card reader, I ran timing tests on the card containing the images discussed above. The card was a Lexar 2GB 80X card with images shot by a Canon EOS 1D MII. There were 70 files shot in Raw only for a total of 492.4 MBs.

Making a finder copy via FireWare to the desktop took about 38 seconds. Fairly quick actually due in large part to the faster FireWire connection of the card reader. A USB 2.0 card reader would be about the same.

Taking the same card and doing a DNG conversion to lossless compression with no embedded DNG took 2:27 seconds. About a two minute penalty for the copy and convert to DNG.

Calculated out, that was about 2 seconds/image for the copy and conversion to DNG. Compare that to a simple finder copy which took just over .5 seconds/image and you might conclude that based only on speed, the DNG Converter takes too long. But let’s examine what you get with the DNG file vs the proprietary raw file. First off, you get a fully publicly documented DNG file that can safely take file renaming, metadata embedding and is likely smaller in file size than the original raw file.

Secondly, you easily solve the problems caused by CF card file organization structures and can use the DNG Converter as a preliminary image verification file check. Since the renaming and renumbering can help organize a shoot better, you’ll also potentially save time and help minimize the risk of overwriting files when ingesting. How much time will you save? That all depends upon how much time you would otherwise spend on file organization and renaming, but potentially, you save up to equal the time spent on the initial DNG Conversion. That’s a tough call because photographers have evolved all sorts of different file naming conventions and methods of safe image ingestion. So your milage may vary, a lot.

Was the free Adobe DNG Converter designed as a pro level image ingestion tool?

Not really. It was only designed to allow photographers to explore the potential of using DNGs. As a free software release from Adobe, it doesn’t at this point have a lot of bells and whistles built in. However, it could become the backbone of a pro level ingestion tool with only a little bit of work. At this point, DNG Converter is neither scriptable nor controllable by command line. So, the ability to automatically control or build scripts around DNG Converter isn’t there yet. Thomas Knoll has stated that command line control or scriptability is an often requested feature and it’s a request that several alpha and beta testers of Photoshop CS2 and Camera Raw 3.x have made. If enough people start adopting a DNG workflow and more people ask for it, I suspect the odds of getting scripting or command line control will increase considerably. (hint, hint)


Click on the image to see larger sized image in a new window.

The relative merits of adopting a DNG workflow really go beyond what it may have in store for you personally. As I’ve outline in Part I and now in Part II, there are some tangible benefits to adopting this workflow. But there are some real intangible benefits as well. There are potential benefits to the entire digital photography community.

It’s no surprise that the issue of undocumented and proprietary raw file formats has received a lot of attention of late. As more and more photographers go digital, more people are discovering some of the, uh, well, thornier sides of shooting digital. A proprietary raw workflow is something that complicates a photographer’s daily workload and as I’ve written in my Digital Preservation article here on PSN, the long term viability and sustainability of undocumented and proprietary raw file formats puts all of our work at risk.

If the digital photographic community adopted a useful and beneficial DNG workflow, I suspect that the odds of the camera companies looking harder at DNG as a viable solution would go up–potentially a lot. I would also suspect that if DNG evolved as the standard for storing raw image files, the long term prospects of DNG surviving as a sustainable format would also go up.

I’m weighing the risks involved. I do honestly believe that at some point in the future (3-5 years at the longest) the camera companies will have no choice but to adopt standard raw file formats. Will it be DNG or some flavor they concoct themselves? I don’t know, but I do think the greater the adoption of DNG now the more likely the camera companies will be encouraged to adopt DNG later.

The more photographers adopt DNG, the greater the pressure will be to have mainstream adoption of DNG industry wide. In that regard, I see individual photographers having a great deal of influence over their own future. I choose to cast my vote for DNG. In my mind, there is currently no reason not to technically and plenty of reasons to do so politically. . .but it’s up to each of you to make your own decisions.

For more information about DNG, see the PSN article Information About Adobe DNG

Additional DNG resources:
The DNG Home Page
The DNG User to User Forum
(free registration required to post)
rawformat.com‘s recent listing of 3rd party support for DNG
The OpenRAW.org Working Group
The Raw Flaw article by Michael Reichmann and Juergen Specht
Understanding Digital Raw Capture by Bruce Fraser (direct 1000K PDF link)
DNG reviews and awards
Adobe Camera Raw Home Page

11 Responses to “DNG Workflow / Part II”

  1. David Marx Says:

    Dear Mr. Jeff Schewe,
    I have been following your excellent articles on the Adobe DNG and wish to thank you for throughly explaining the uses for this valuable tool. One danger, though, that this article fails to mention in using the DNG as a “image ingestion” tool is that it ignores JPEG files. If you use this as a method for empyting a memory card rather than doing a copy using the operating system you will miss any images that were captured in JPEG.
    I think that you are on the right track with this suggestion but that the DNG needs to include some of the features that Adobe already includes in its “import images from memory card” wizard in the PC version of Elements 3.0. (On a tanget Elements 3.0 has a ton of cool features like the side-by-side slideshow feature, and the backup utility that should be expanded and included in other programs.)
    I fully support using the DNG to ingest images and love the way that it solves the file naming and sub-folders annoyance but this somewhat dangerous advice for photographers who shoot with a mix of RAW and JPEG on the same card. Thanks again for the great info on this tool.

    –David Marx

  2. Philip L Says:

    The issue of corrupted files, Jeff, does cause some concern and make one wonder whether converting right at the point of download is the best idea.

    Although the pop-up warning dialog that the DNG Converter provides is no doubt very handy, and would likely allow for less chance that a corrupt file will go unnoticed, what if a file is corrupt and not discovered by the DNG Converter, only to be discovered later, after the file is converted to the DNG file format?

    My question is, would you lose the ability of 3rd-party recovery software to get back the file. (I’m thinking in terms of if you were to do as you have outlined above, as opposed to archiving the *entire* contents of each card)

    Thanks

  3. Nick Rains Says:

    I am pleased to see this workflow put forward by one of the main protagonists in the digital world. I already use this workflow myself, almost exactly as described, and even went so far as to write a major feature for one of the camera magazines here in Australia about it, due out in the next edition.

    You missed one benefit, if you use DNG as your main archive format, you only need to preview in Bridge once. All RAW edits are refected in the JPG previews embedded in the DNG file so you have a fully up-to-date previewed archive perfect for burning as-is to DVD.

    DNG Converter is turning into Photo Mechanic! If you combine this workflow with the slideshow features out of Elements3, you have a pretty good clone of PM.

    So, the wishlist:

    1. Ingestion into at least 2 different folders at the same time.

    2. IPTC templates to add copyright etc.

    Wishlist for Bridge:

    1. Option to turn off ‘previews’ and use embedded jpegs only. Editing is vastly too slow to make Bridge a professional tool for this task.

    Cheers

    Nick Rains

  4. Thom Hogan Says:

    When I talk about necessary software for digital cameras (most of which doesn’t exist yet), I talk about “hubs.” You need three: an ingestion hub, a processing hub, and an output hub. In a perfect world, they would be completely integrated; but in an imperfect world, photographers would tolerate them at least being aligned enough so that they were scriptable (on a Mac) or could be run sequentially (on Windows). (And if you haven’t already figured it out, the use of the word “hub” instead of “program” is intentional and implies something that can direct sub-programs into more complex and customized workflows.)

    The problem with DNG Converter as an ingestion hub is that it fails Bruce Fraser’s (and my, and that of a lot of other photographers’) need for not screwing up the moving of data from the camera to computer. It lacks ability to deal with JPEGs, extract preview images, add IPTC data, identify sub-jobs, create automatic backups, and much, much more. Depending upon the client I’m shooting for, my image ingestion changes a bit, so it also fails in its ability to have me “mass configure” it to a particular client setting.

  5. Simon Iannelli Says:

    thanks a lot for the great articles.

    i agree with you that it is a great feature to convert directly from the CF. but i got a little problem. after the conversion and numbering i’ll maybe delete some shots an so i got gaps in my numbering. what now? renumber or leave the gaps? (or maybe i could explore my CF first with bridge, delete some shots and import and convert afterwards[this takes time i maybe don't have while])

    uhm.. i’m not shure wich is the best solution. any suggestions?

  6. Scott Jones Says:

    Great articles. Only problem for me in using this as my “ingestion” utility from my CF card is that it does not transfer the JPEGs along with converting the RAW files. So if I have a mix of JPEGs and RAW on the same card, The DNG converter will not grab all all the pictures so I still have to go back to the CF card with some other utility and transfer the JPEGS.

    I would suggest making the converter able to transfer all files to a designated folder while converting the RAWs.

    Any thoughts?

  7. Daniel M Says:

    The utility that I use to transfer files form the CF to the system is able to ‘grab’ EXIF info to use it in the filename.
    If I have files from 2 different days in a card and transfer with DNG converter, the date (dropdown) will be the same for all files. If the date was pulled from the EXIF, then each file would have the correct date.

  8. Mark Turner Says:

    I’ve been following this DNG business with great interest, but haven’t made the jump to converting my files yet. The current articles are leading me in that direction, so maybe I’ll move soon. The prospect of going back and converting thousands of D60 .CRW files and re-archiving them to CDs is daunting.

    I’m concerned about the suggestion to process raw files directly from the compact flash cards through the DNG converter before the original files have been backed up to some other medium. My own workflow involves copying the files from CF to the hard drive on my computer (either laptop in the field or desktop in the office), doing a quick review and rotation of verticals in BreezeBrowser, then burn a backup CD or DVD. Only then do I do a more serious edit to delete less good variations and renumber to fit my filing system. I just don’t trust having my digital originals stored in only one place before any software processing. Perhaps I’m paranoid, but I’d rather be safe than sorry.

    I’m also not convinced I want to add more time to the “ingestion” process while running a laptop on battery power at 10 pm in the middle of nowhere when all I really want to do is go to bed to be rested for the next day’s shoot. It takes about 18 min per gigabyte to copy the files from CF to my laptop as it is, using a PC card adapter (Win XP on IBM ThinkPad R40). Five or more gigs of originals a day, and we’re talking significant copy time, plus burning CDs.

    The DNG benefits of an open format and no sidecar files and are substantial, especially as more software supports it. I’ll stay tuned.

  9. Nick Rains Says:

    Hi Jeff

    Another thought occurred to me – if you point Bridge at the destination folder before starting the ingest proceedure, Bridge will thumb and preview each DNG file as it is written. This eliminates a step and two time consuming tasks can occur concurrently.

    Nick Rains
    Australia

  10. Jeremy Clow Says:

    The flaw here is the converters inability to handle JPEG’s. I exclusively shoot RAW now, but before I learned what I was doing, I shot JPEG. How am I to archive those images? I don’t want to discard them, but now I have an extra step in my post processing. One more step, cuts into my shooting time.

  11. Marc Rochkind Says:

    Great column… I am thinking along the same track. Take a look at my web page (basepath.com/ImageIngester/), where I talk about and have available for download an all-in-one utility that reads from the card, makes two copies, converts to DNG, and gives you a chance to verify visually before offering to delete. I would greatly appreciate feedback from anyone about this so I can work it into a utility that’s both convenient and reliable.

Leave a Reply

You must be logged in to post a comment.