Logging Software for Canon 5D

Is there a good app for logging Canon 5D footage on a MacBookPro including marking best takes, adding multiple markers to individual clips and have that info show up in Log and Transfer prior to ingest?

Surely you just do this in L&T?



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

No.


www.shanerosseditor.com

Listen to THE EDIT BAY Podcast on iTunes
[itunes.apple.com]

No? Like in no? Nada? Nothing?

<<Surely you just do this in L&T.....>>

Of course. But picture this, director goes out on a shoot with client and a crew. The director insists on using a 5D. He shoots an interview or two, some B-Roll and some docu shots. On the way home in the crewbus/airplane/train they go through the footage and log it. I don't want them touching FCP for this, they'll screw it all up (again). Like in Digital Heaven's MovieLogger, the director plays out an xml that I then import, link to the source footage and start bringing in the shots they want via L&T.

They want us to turn around entire virals in a day or two. Only way to do that is to have the footage pre-selected and logged before we start. It would be great to have a little no-brainer standalone app to do just that.

Ok, so I'm dreaming (again).

Only way is to teach them some FCP boot-camp level basics?

Or go with them on the shoot?

All the best,

Tom

"Messing up the clips" shouldn't be a major issue, since you can always lock those files, or make copies. Maybe you could try having them use FCP on the H.264 clips, then after you convert to ProRes, create a copy of the project file to reconnect the H.264-based clips to the ProRes versions for editing. If it's a reconnect, any metadata they add to the clip should be retained. The real question is, since H.264s have no real timecode, whether the reconnect will be successful with markers intact. Try one or two test clips.

Could also do it the old-fashioned way -- use the colour labels in OS to mark desired clips, and just do the logging on a simple TextEdit document. You use FCP to fly to where they want. That's how the story editor and I worked on the last reality show I cut.


www.derekmok.com

Nono. L&T will base timecode on time of day. H.264 will lack timecode. You can transcode the h.264 in compressor, but you cannot manage the media on the finder level. (eg. Rename)



www.strypesinpost.com

> L&T will base timecode on time of day. H.264 will lack timecode.

I was under the impression that they'd be doing this stuff before the clips are transcoded. Unless you want to trust that to the producers and director (shivers).

H.264 doesn't have timecode, but the clips will have the same duration. There may be a way to do stringouts with H.264 clips (which will start at 00:00:00:00), with markers and the whole shebang, and then match that to later ProRes clips just by visual conform or a forced relink. Or, he could use Compressor instead, which will then forego the time-of-day timecode and the "dummy" timecodes on the H.264 files and the ProRes files will, theoretically, match up.

Renaming would be a HUGE no-no. I haven't played with enough DSLR footage at the ingest level, but I'm pretty sure renaming will prevent Log and Transfer from recognizing the contents of the memory cards altogether, as with every other tapeless format I've used.


www.derekmok.com

The timecode issue is not an issue if you get the Canon plug-in for FCP

[www.usa.canon.com]

It adds the ToD TC from the EXIF info on the camera using the date/time stamp.

A better more accurate conversion of the H264 (including ToD) from the 5D is to be had using Rare Vision's 5D to RGB [rarevision.com] and there is a batch script now: [www.noside.fr]


The problem here is the logging and not allowing the Director et al to mess it up.

Unfortunately I think you'd be better going along and/or training them correctly to log within FCP after transfer - after all its not that difficult to do in the list view in the browser!

A few serious recommendations.

They need to make at least 2 copies of the CF cards onto HDDs. Preferably 3. On different HDDs. Storage is seriously cheap and getting cheaper. You can buy an external FW800 1TB HDD for less than a hundred bucks and a 1TB USB2 for less than $60!!! Thats enough for 50+ 16GB CF cards allowing for space on a 1TB HDD.

The DIT simply MUST use 5D to RGB to make colour and detail accurate ProRes files of the originals. (If you don't have a camera assistant/DIT then you NEED one or someone who knows what they are doing).

You just won't get the Quality if you go a Quicktime route (FCP, QTpro, MPEG-Streamclip) as you will encounter Gamma shifts and added noise from the H264 conversion process within Quicktime.

I have done a fair bit with 5D footage and you have to try to squeeze every last drop of quality out of it.

One thing to note is that when copying the CF cards to HDD you should create a disk image .dmg file of the card.

This can then be mounted from the HDD and used in Log & Transfer or 5DtoRGB instead of CF card.

If they did a simple Log & Transfer using the Canon Plug-in onsite for speed using the dmg copies - you could always re-conform using 5DtoRGB later.



Lastly you really should look into getting the Magic Lantern Firmware for the 5D as it adds an amazing amount of filmmaker functionality to the DSLR. [magiclantern.wikia.com]



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Take a look at the Free Demo of HD log. If it does what you need shoot me an email as I have a license i would sell.

That does look promising. $699 huh? Ouch. I'll give the Demo a try though in the next couple of days and let you know.

Thx,
Clay

Quote
Ben
One thing to note is that when copying the CF cards to HDD you should create a disk image .dmg file of the card.

As these cards are PC formated you should make an .iso image.

Andreas

What's the difference and how do you write an iso image on a Mac?



www.strypesinpost.com

A .dmg uses the Mac file system while an .iso as a raw copy keeps the original file system. Easiest would be to create an hybrid file system (HFS+/ISO/UDF). This way any machine/OS will find a mount point. You can do that with Disk Utility. In the Image Format popup just select "hybrid image (HFS+/ISO/UDF)"

To learn a bit more about the underlying frameworks of Disk Utility you can go to Terminal and type "man hdiutil"
A part of it explains the differences:

Filesystem options:
                -hfs    Generate an HFS+ filesystem.  This filesystem can be
                        present on an image simultaneously with an ISO9660 or
                        Joliet or UDF filesystem.  On operating systems that
                        understand HFS+ as well as ISO9660 and UDF, like Mac
                        OS 9 or Mac OS X, it is usually the preferred filesys-
                        tem.
                -iso    Generate an ISO9660 Level 2 filesystem with Rock Ridge
                        extensions.  This filesystem can be present on an
                        image simultaneously with an HFS+ or Joliet or UDF
                        filesystem.  ISO9660 is the standard cross-platform
                        interchange format for CDs and some DVDs, and is
                        understood by virtually all operating systems.  If an
                        ISO9660 or Joliet filesystem is present on a disk
                        image or CD, but not HFS+, Mac OS X will use the
                        ISO9660 (or Joliet) filesystem.
                -joliet Generate Joliet extensions to ISO9660.  This view of
                        the filesystem can be present on an image simultane-
                        ously with HFS+, and requires the presence of an
                        ISO9660 filesystem.  Joliet supports Unicode file-
                        names, but is only supported on some operating sys-
                        tems.  If both an ISO9660 and Joliet filesystem are
                        present on a disk image or CD, but not HFS+, Mac OS X
                        will prefer the Joliet filesystem.
                -udf    Generate a UDF filesystem. This filesystem can be
                        present on an image simultaneously with HFS+, ISO9660,
                        and Joliet. UDF is the standard interchange format for
                        DVDs, although operating system support varies based
                        on OS version and UDF version.

                By default, if no filesystem is specified, the image will be
                created with all four filesystems as a hybrid image.  When
                multiple filesystems are selected, the data area of the image
                is shared between all filesystems, and only directory informa-
                tion and volume meta-data are unique to each filesystem.  This
                means that creating a cross-platform ISO9660/HFS+ hybrid has a
                minimal overhead when compared to a single filesystem image.

The're also examples at the end. You can make a very small AppleScript Droplet which copies your cards either in raw format or hybrid.
In case you are sure the copied cards will be only used with a Mac it finally doesn't matter.

Andreas

Quote
Andreas Kiel
A .dmg uses the Mac file system while an .iso as a raw copy keeps the original file system. Easiest would be to create an hybrid file system (HFS+/ISO/UDF). This way any machine/OS will find a mount point. You can do that with Disk Utility. In the Image Format popup just select "hybrid image (HFS+/ISO/UDF)"

On OS X 10.6.6, my Disk Utility doesn't offer the "hybrid" option, as least in the GUI application. How to do access this option? Thanks.


-Dave

@ Dave
In Disk Utility choose menu item 'File -> New -> Disk Image from Folder...' then select the card or hard disk you want to save. The option will be available. Alternative is DVD/CD master which also should create a platform independent image named .cdr but probably you have to change the extension for some platforms.
You always can convert .dmg files to hybrid with Disk Utility if needed.

@ Clay
Totally forgot to give a comment about your problem. As suggested above give them some basic lessons for FCP and they should be able to do the logging -- but as Derek said renaming the clip is a no go.
After you transfer the clips you will be able to re-connect them easily -- as long file names are kept. You will get errors about reel and timecode. But you can ignore them.

Andreas

Quote
Andreas Kiel
@ Dave
In Disk Utility choose menu item 'File -> New -> Disk Image from Folder...' then select the card or hard disk you want to save. The option will be available. Alternative is DVD/CD master which also should create a platform independent image named .cdr but probably you have to change the extension for some platforms.
You always can convert .dmg files to hybrid with Disk Utility if needed.

Thanks for the info, Andreas. I tried to create a new disk image in Disk Utility and I couldn't find the "hybrid" option. But, when creating one from an existing folder or disk or mounted card, that option does show up. Thanks for the clarification!


-Dave

Great info, Andreas. That should go into the FAQ.



www.strypesinpost.com

Here just a little AppleScript using the standard means which come with your OS.
Copy the code and paste into AppleScript Editor. Then select the EOSxxx folder on the card. The script will search the .THM EXIF data and will extract the original creation date as NDF timecode (pretty the same FCP does in L&T) and the .MOV name. The result will be copied to the clipboard.
You can now type in the TC into the source timecode in FCP, the reel number should always be name of the card.
Feel free to play with it.

set theFolder to (choose folder)
set theFileList to list folder theFolder without invisibles
set exifList to ""

repeat with theFileName in theFileList
	if theFileName contains ".THM" then
		tell application "Image Events"
			set theMetadata to metadata tags of (open (theFolder & theFileName as string))
			repeat with theTag in theMetadata
				set tName to name of theTag
				if tName = "creation" then
					set tagValue to (value of theTag) as string
					tell me
						set theTODlist to words 4 thru -1 of tagValue & "00"
						set text item delimiters to ":"
						set exifTC to (theTODlist as string)
						set text item delimiters to "."
						set movName to ((text item 1 of theFileName) & ".MOV" as string)
						set text item delimiters to ""
						set exifList to exifList & movName & tab & exifTC & return
					end tell
					exit repeat
				end if
			end repeat
		end tell
	end if
end repeat
end
set the clipboard to exifList

Just open TextEdit make a new doc and hit cmd-v. The result will look like below.

MVI_2498.MOV	11:37:14:00
MVI_2499.MOV	11:42:11:00
MVI_2500.MOV	11:44:14:00
MVI_2501.MOV	11:45:50:00
MVI_2502.MOV	11:51:39:00
MVI_2503.MOV	11:53:44:00
MVI_2504.MOV	11:56:58:00
MVI_2505.MOV	11:59:12:00
MVI_2506.MOV	12:02:27:00
MVI_2507.MOV	12:05:06:00
MVI_2508.MOV	12:07:08:00
MVI_2509.MOV	12:18:09:00
MVI_2510.MOV	12:22:57:00
MVI_2511.MOV	12:23:00:00
MVI_2512.MOV	12:52:38:00
MVI_2513.MOV	13:05:40:00
MVI_2514.MOV	14:07:03:00
MVI_2515.MOV	14:14:35:00
MVI_2516.MOV	14:14:49:00
MVI_2517.MOV	14:19:25:00
MVI_2518.MOV	14:24:09:00
MVI_2519.MOV	14:30:26:00
MVI_2520.MOV	14:31:52:00
MVI_2521.MOV	14:38:04:00
MVI_2522.MOV	14:55:41:00
MVI_2523.MOV	15:03:23:00
MVI_2524.MOV	15:04:39:00
MVI_2525.MOV	15:06:10:00
MVI_2526.MOV	15:10:34:00
MVI_2527.MOV	15:14:36:00
MVI_2528.MOV	15:20:14:00
MVI_2529.MOV	15:22:05:00

Maybe I do an 'XML version app' of the script which will allow to log the original H.264 files in FCP and then to use L&T to transfer the wanted/needed files.
Andreas

I've updated a small app of mine to work with the EOS movies.
Usage is simple, just open the xxxEOSyyy folder. The app will process the EXIF data and create a FCP XML on the desktop.
You can import the XML into FCP and start logging with the h.264 files, you even can rename clips.
Once you have the transcoding done on another system (or the same) you can reconnect those h.264 clips to the Prores clips. Since TC and Reel, duration of the files do match there will be no warning or trouble.

This probably only works with FCP 7.
[www.spherico.com]

Andreas

Hi Andreas,

very cool and thanks muchly. Taking a test drive with that later this week...

Cheers,
Clay

Nice work Andreas.

Andreas,

You forgot the frames.
They are written in SubSecTimeDigitized, in 1/250 of a second...
(makes it compatible with QTchange...)

Bouke
www.videotoolshed.com

Bouke,

We were using the THM files for a rough pre-sorting and to get the cam's serial number to figure out the angle. I've never seen a 'SubSecTimeDigitized' in these data.

Andreas

To pick up this EOS logging thing again.

I made some research as I'm faced with the EOS stuff now as well. And I talked to the people who really know.


The EOS files work a bit strange and so does the plugin for FCP.
Normal EXIF data can have unique clip IDs but the EOS cams do not create this entry but spreads this info thru out the THM files in a mystery way.
Using log and transfer with the EOS plugin FCP will create an ID on the fly with this secret/undocumented code which then is baked into the QT metadata of the transferred clip. This will allow to reconnect to the original clips.
At this moment nobody except Canon and Apple do know the secret - but they keep it as a secret and there are no plans to make it public.
So a real field logging tool for the EOS only can be created by either Apple or Canon.

Quote
software engineer
Unfortunately, this time the mechanism ... is private ...

If you use the L&T to transcode the h.264 clips you always can reconnect to the original clips on the cards in the standard way.
5DtoRGB, Compressor or MPEG Streamclip don't know anything about the FCP, plugin and EXIF secrets and therefore don't write the needed QT metadata into the file. With those files you can't reconnect to the matching clips on the cards.
So the non L&T transfers are faster and sometimes better, but will lose the connection to the originals. In any case make some tests -- 5DtoRGB can create much better results but also can create really bad results.

Depending on your workflow this tapeless thing can end up in a senseless thing. Your workflow has to be documented somewhere on your machine for the case of disaster.

What can be done?
In any case make at least one copy of your card as hybrid image.

A)
On set (or anywhere between set and post) use any of the logging tools which do read the EXIF time of day and accept the card name as reel to create a FCP compatible XML.
On later state use L&T to convert the clips (in a different project) and reconnect the the h.264 files from the logging session to these 'online' files. As said, this only works without pain if EXIF TC, duration and reel do match.
That way you always can re-connect your clips to the original source.

B)
Use FCP L&T to do a basic logging. Additional info for a clip can be displayed in the browser by ctrl-click the browser column titles and select what you want to see - you unfortunately can't copy/paste.
Fine tune logging must be done clip by clip in FCP - which is more then boring.
For safety reasons you could make a copy of the transferred clips as well.
To connect to the source works the standard way.

C)
Use a tool to convert to ProRes (or other format)
Use a data base to store the conversion settings for each clip - as you would have done in the old analog days.
For safety reasons you should make a copy of the transferred clips as well.
Then use either FCP or another logging tool.
To go back to the source clips can be painful.

None of this options sounds like fun nor it is a real workflow - it's work without flow. But finally -- it depends of what you want or need to do, maybe you don't need any of that.


Here some personal notes:
In these days tapeless does have the advantage that you already have each take as clip (maybe with some metadata). There also no moving parts in the camcorder or player. Resolution can be higher than ever before. Storage is crazy cheap and fast (just compare the price of digi beta tapes or D5, D1 tapes to the price of the cards used in the cameras - P2 cards are bit different but safer). You also don't need a physical player/recorder or special hardware as in the times of tapes - you only need some software and a card reader(at least as long you aren't a brodcaster).
But there are also disadvantages. Because of the file systems of the cards clip length is limited - other than with tape based systems where clip length was limited by the tape type, tapeless can create clips which span over several 'tapes'. This is great, but as said the physical clip length is limited by the file system on the card. Even if a long clip would fit on the card it will be divided into 'small' pieces. This again might be no problem as there is software which can merge them. But that in most cases includes a transcoding - so we are somehow back to digitizing. Writing back this clip to card in most cases will 'destroy' the card structure. So card structures are new problem.
One thing I'm missing most that there is no tape any more. With an old fashioned tape logging software you could jog/shuttle, go thru the tape (means all clips) with high speed, add markers on the fly and so on.*

Same with all this huge amount of codecs and card strutcures. Some NLEs claim to work native with all/many codecs - but at the end everything needs to be rendered to the required format for output. But the digital world allows to choose that format and you can go to multiple delivery formats from multiple source formats. But these also includes 'interpretation rules' both on ingest and playout. Why does a transcoded h.264 file from FCP look different from one from Compressor, Mpeg Streamclip, 5dtoRGB or from Premiere or AE and so on. And look at all this f****** gamma stuff.

Tapeless world is new and still confusing.

Just some thoughts.
Andreas


*) For that reason I re-invented the tape for P2. My P2 app allows to create a virtualTape spanning all clips for one or more cards. You can use the movie like a tape with really fast response times, add markers, comments etc. without the need to know which clip it is or to edit clip info. The app knows which clip is affected by the user entries.
If there are enough people who would like the same function for the EOS I might create something similar for that format.

Good info there, Andreas. I'm actually transcoding some canon footage through Color right now to prepare for the grade tomorrow. For some reason QuickTime seems to clip whites and blacks on this set of footage. My offline was done with footage transcoded through Compressor. I'm not on snow leopard so no EOS plugin, but I doubt it will make a difference. Anyway, the values are all truncated between 0-100 IRE and it seems the only way to bypass it is to send it through Color, which I think bypasses the color profile setting in camera. Not really sure what's going on in the camera as I don't own a canon DSLR camera myself. I've seen sub blacks from the camera that made it through QuickTime, but not this time.



www.strypesinpost.com

Have you tried to change the 'Gamma Interpretation' in Compressor?
I'm not at a machine with FCS right now, but I think the options are only 'None' and 'Automatic' where 'Automatic' is the default.
You also may apply a 'Black&White Restore' in some cases this can be really helpful.

The cam color profile by default is sRGB, but I don't know how to convert.

I've uploaded a compare picture to [www.spherico.com]

From 1-4 of the columns
1) FCP L&T
2) Compressor automatic gamma
3) 5DtoRGB no gamma
4) 5DtoRGB gamma 1.8

Andreas

I haven't tried that option, as I was in a rush for time earlier.

Between Color and FCP (at least in my case), I won't say there much of a difference in terms of gamma (luminance), but, there is a difference in terms of contrast, little in perception since the values affected reside outside broadcast legal ranges. I have seen shots from the DSLRs that went through the same process but with values in the sub blacks and super whites. I suspect this has to do with the color profile in the camera. Perhaps there is a setting that maps to legal gamut.



www.strypesinpost.com

Quote
Andreas
I've uploaded a compare picture to [www.spherico.com]

From 1-4 of the columns
1) FCP L&T
2) Compressor automatic gamma
3) 5DtoRGB no gamma
4) 5DtoRGB gamma 1.8

Are the portions of the comparison image all take from within one application? For example, is this a composite image saved from within FCP, etc.? Or, is this a composite image where the portions are taken from different applications, or screenshots?

The problem with any of those scenarios is that you are "seeing" the images interpreted via ColorSync (or not), and may vary depending on the internal contents of the original files (metadata may be missing, or incorrect, or different). In other words, it may not be the best comparison, though the differences may be real (with all things considered).

Sorry if I am confusing the issue, but I would think that you would need to view the clips on an external broadcast monitor, assuming that the application sending the clips properly interpreted the contents of the source files...

Hmm... Regardless of how these were assembled, it is still frustrating to see differences show up (without a clear reason).

I guess if we knew how the originals were "captured" and how the composite image was created and saved to a TIFF file, it might help clear things up. I know that opening the same file in FCP, Compressor, and QT Player 7 (even with compatibility set) all yield slightly different looks (on the computer screen, which isn't accurate anyway).


-Dave

Dave,

The comparison picture is taken as a screen shot within one app (in this case FCP - not really with exact the same mask). Within QT, AE it looks slightly different but pretty the same. As a screen shot normally does not include any ColorSync (as long as you do not apply it to the saved file) the result is also the same. That's why I saved the file as TIFF without compression and no ColorSync applied. Again this doesn't mean that each app you work with would interpret the file the same way.
The source file(s) always had been the same, taken from the same native unchanged card of several EOS 7D.
The results were viewed on a pretty good calibrated TFT and they look similar to this screenshot - though with a consumer TFT it's not that visible.
Anyway - the comparison shows the big differences in file interpretation/transcoding and it doesn't matter whether ColorSync is applied to the screenshot or not. Maybe I should have taken the results and layer them with a difference mode (which don't include any ColorSync) - but the result would be pretty the same.

Andreas