Question about drives and transfer rates?

I was just wondering if anyone out there knew what determined transfer rate or mbps speed? For instance you'll see numerous different drives advertised as having the same 7200 Rpm, same interface i.e.; thunderbolt, e sata, firewire 800 etc but with all different transfer rates.

Thanks,

Ian

It's analogous to phonograph records. The rpm does not indicate the high frequency fidelity. Indeed the progression from 78 rpm to 33 rpm phonograph records yielded higher fidelity in spite of the rpms, because the grooving and the cartridges became much more precise. Also not all records, and not all cartridges, provided equal fidelity at the same 33 rpm.

For a given rpm, HDD data read speed depends on the density of data on the path, which can't be too great for the head, and on the number of disks. So different models of 7200 rpm HDDs have quite different read rates because of their differing data densities (times the number of disks) and head designs. And some 5400 rpm HDDs really do have higher read speeds than some 7200 rpm HDDs.

The only speed characteristic of a hard drive that is directly related to the rpm is the "latency". The head that's at the correct radial position on a 7200 rpm spinning disk will take up to 1/7200 of a minute (1/120 seconds) to get to the place where the sought data is. So the "average latency" is 1/240 seconds just because of the rpm. That's a long time if reading many small spans of data, but not when reading a goodly span of video.

There are many more engineering details, but the above gives the essence.

Dennis Couzin
Berlin, Germany

Thanks for the detailed response. So what is a good mbps for editing?

You can't ask that question without specifying the codec, resolution, and frame rate. Also the number of simultaneous streams. And also the processing power of the computer.

Some idea of hard drive requirements can be gotten from page 16 of the Apple ProRes White Paper of July 2009 (not the October 2012 White Paper). Crudely estimate the three-drive RAID 5 read speed as twice the single drive speed. Then for 24p HD ProRes 4444 a single 7200 rpm SATA hard drive, internal to the MacPro or connected by eSATA will let you edit with two streams in FCP7. (The White Paper gives 7200 rpm as if it indicated the drive's read speed, contrary to my earlier post, but understandable for the practical illustration.)

It is famously difficult to peg the read speeds of hard drives. The last batch of hard drives I bought were 2TB HGST Ultrastar 7K4000's because the spec sheet gave the sustained transfer rate as 181 MB/s. Obviously they won't function at anywhere near that rate in my setup, but maybe they're faster than average 7200 rpm drives. Also their uncorrectable error rate is spec'd at 10^-15 versus the more typical 10^-14, which I think valuable for drives that will be many times completely overwritten. I prefer buying hard drives and enclosures separately for the flexibility, and for knowing what I'm buying.

My editor typically edits 50p HD ProRes 422, which is about as demanding as 24p HD ProRes 4444, using various 7200 rpm hard drives in eSATA enclosures, without hangups. When editing 50p HD ProRes HQ we use two internal 7200 rpm hard drives in OSX RAID 0 for double read speed.

Many on this forum swear by big RAID arrays. They might be doing heavier multi-stream editing than me.

('HD' stands for '1920x1080' in this post.)

Dennis Couzin
Berlin, Germany

Thanks for the info!

Mbps is slightly misleading as its not always clear what it means.

"mega bits per second" is thought by most to be 1024 kilobits per second. However it is not.

Transfer speeds, data-rates are all measured typically in decimal so actually 1Mb = 1000 x 1000 bits NOT 1024 x 1024 bits.

This confusion also applies to data-storage sizes!!!

Manufacturers always preferred to use the decimal rendition of ""kilo" and "mega" rather than binary as it mean't they could make drives sound larger than their actual binary formatted bit capacity.

The standard for binary notation was changed to reflect this.

Thus decimal versus binaray notation is such:

K or Kilo or 1000 (decimal) cannot be made using binary so the closest you can get is 1024 and this we write as "Kibi" and kibi should be abbreviated Ki

M or Mega is Mebi - abbr. Mi

G or Giga is Gibi - abbr. Gi

T or Tera is Tebi - abbr. Ti

b is a bit (0 or 1, on or off)

B is a Byte (8 bits)

So 1000bits is a Kb or Kilobit

1024bits is a Kib or Kibibit


The upshot of working out data-transfers as decimal and the data-size as binary basically means that your 1Kib (one Kibibit or 1024 bits) file on your drive will take ever so slightly longer to transfer at 1Kbps than 1 second. 1.024x longer. (1024bits / 1000bits per sec = 1.024 seconds)

This needs to be taken into account when calculating data-rates and data-transfer times.


RPM (Revolutions Per Minute)

The other factor that Dennis touched on is that 7200rpm HDDs are quicker than 5400rpm drives at finding a file (latency) but also (in general) the faster the spin speed - the faster the disk.

You can get 10,000rpm drives which are faster at data-transfer than (similar technology) 7200rpm.

Other factors that affect data-rate are the connection bandwidth (see below), the disk controller, the size of the buffer, the temperature of the disks, the electronics quality, the manufacturer, the length and quality of the cables, the fragmentation of the file (or lack of).

Approx. Connection bandwidths

Firewire 400 (400Mbps or ~50MBps) rarely achieves over 40MBps
Firewire 800 (800Mbps or ~100MBps) rarely achieves over 80-90MBps
Gigabit Ethernet (1000Mbps) rarely achieves over 95MBps
10GigE (10Gbps ~ 10,000Mbps) ~ 1250 MBps
eSATA or Internal SATA:
SATA I ~ 1.5 Gbps or ~ 150 MBps
SATA II ~ 3 Gbps or ~ 300 MBps
SATA III ~ 6 Gbps or ~ 600 MBps
Thunderbolt I ~ 10Gbps or ~ 1250 MBps
Thunderbolt II ~ 20Gbps or ~ 2500 MBps

These are all theoretical maximums but error correction and other overheads take their toll and you never see perfect sustained full speed.

So something to think about is your bottlenecks that come from the connection and type of disk you are using.

For example if you have a RAID-0 of four 130MBps HDDs all working in parallel - this should do at least 400MBps.

...but linking it via SATA I will mean you get no more than 150MBps! This is because the bandwidth is just not sufficient to carry the full speed capable from the RAID.

Likewise if you have a Thunderbolt II connection to a single 130MBps disk you will not be using even a tenth of the possible bandwidth and it will not make the disk go faster.

So to really understand what speeds you can get - all parts of your system must be looked at from the Controller on the Mac/PC, the Connection Bandwidth to the Drives (and housing if external).



I'm one of the guys that makes and runs large RAID arrays and there are several reasons for that.

One is total storage - you, I and everyone else need a lot!


Generally HDDs start to slow down almost in a linear fashion from full.

It means you need to keep as much free space as you can on any disk if you want it to operate anywhere near its peak.

This is because as we mentioned before - the faster the RPM the more data is read or written per second.

As HDDs fill up they write from the outside of the disk to the centre. The centre of a spinning disk is slower (distance covered) than the outer edge. Thus data rate is slower as the disk becomes full!

In an ideal world you wouldn't fill your media HDDs past 50% but try not to fill your HDD media drives past 70%.

Up to 80% if you are really stuck but don't be shocked if you lose some serious performance.

Never fill HDDs past 90% or you are in for some trouble and horribly slow performance and risk of data-loss.


Here is a short video explaining HDD speeds which might help you understand if I didn't make it clear above:







In order to run several large projects of several TiBs (TebiBytes) each AND because I want to have enough space to keep nearly 50% free - I buy as much fast storage as possible.

The other reason is hardware redundancy HDDs are incredibly complex and are prone to wear and breaking like all electronics.

So I run RAID 6 which means I can have 2 HDD hardware failures in the RAID without losing any data or functionality.

I then run a separate dual backup workflow for current projects.


Important Note: RAID is NOT backup even a Mirrored RAID aka RAID 1 is NOT a backup.

A backup is a separate unit or units with copies on.

Backup is for data-loss or corruption. RAID is for hardware failures.


The final and most obvious reason is speed. Given enough bandwidth and enough drives working in parallel (RAID 0, 5 or 6) you can achieve silly speeds of 100s or 1000s of MegaBytes per second.

Bear in mind that you mostly won't necessarily see the benefit unless you use uncompressed HD or many many multiple streams - or perhaps have several editors working off the RAID. The one thing you will notice is speed to load projects and files, relink data and export rendered files.

I can export a HUGE file at between at nearly 1000MBps which means just over a second per GB of stored data - saving myself a LOT of time.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Hey thanks for the detailed information!

I appreciate Ben's discussion of the big storage question. His linked explanation of the slowed read/write of more full drives is important, and perfectly analogous to the phonograph record.

His reminder that RAID is not backup is also important. RAID 0 (which really shouldn't be called RAID, because there's no Redundancy) is the fastest simplest RAID variant. Since I routinely backup, and can endure the delay of complete rebuild in the event of a drive failure, RAID 0 is best for me. Each editor should consider whether they need what RAID 5 or 6 offers.

His final point about the desirability of read/write speed is not about editing per se but about the file transfer work that inevitably accompanies editing. Yes to that.

What's missing from Ben's discussion are down-to-earth requirements for hard drive read speed for FCP7 editing.

Quote
Ben King
Given enough bandwidth and enough drives working in parallel (RAID 0, 5 or 6) you can achieve silly speeds of 100s or 1000s of MegaBytes per second. Bear in mind that you mostly won't necessarily see the benefit unless you use uncompressed HD or many many multiple streams - or perhaps have several editors working off the RAID.

Uncompressed HD is too much for most RAID arrays. 50p 1920x1080 uncompressed 10-bit 4:2:2 flows at 276 million bytes/sec. Figure just two streams and you've exceeded Ben's example of a 4-drive RAID 0 array (or a 5-drive RAID 5 or a 6-drive RAID 6). Whoever's editing uncompressed HD can make yet bigger RAID arrays or consider SSD storage or working in small chunks and using a RAM drive.

On the other hand, ProRes codecs are very efficient for FCP editing. The interested should look at page 16 of the Apple ProRes White Paper of July 2009. Where that page figures a 3-drive RAID 5 array, figure a 2-drive RAID 0 array, or figure just one drive and halve the number of streams. As for SATA transfer speeds, SATA I was way passé in July 2009; the White Paper must have used SATA II. Is eSATA connection working at SATA II speed ever too slow for ProRes editing in FCP7?

I'm committed to eSATA. I now buy enclosures that connect by eSATA only. Ben's approximate figurings seem to show an arithmetical bias against eSATA and toward Thunderbolt:

Quote
Ben King
SATA I ~ 1.5 Gbps or ~ 150 MBps
SATA II ~ 3 Gbps or ~ 300 MBps
SATA III ~ 6 Gbps or ~ 600 MBps
Thunderbolt I ~ 10Gbps or ~ 1250 MBps
Thunderbolt II ~ 20Gbps or ~ 2500 MBps

When talking SATA he converts Gb to MB by multiplying by 100; but when talking Thunderbolt he converts Gb to MB by multiplying by 125. 125 is what we'd expect from straight arithmetic -- a G is 1000 M's and a B is 8 b's. The discrepancy between 125 and 100 is much too large to be explained by the binary vs. decimal terminological matters Ben discussed. It is a fact about SATA that 3 billion bits at the interface yield just 300 million bytes of data. So for us users the 300 MBps figure, not the 3 Gbps figure, is the meaningful one. SATA, along with many other technologies including USB 3.0, uses "8b/10b encoding" whereby 10 bits are used to transfer just 8 bits, with 2 bits sacrificed for error correction. I don't know what kind of error correcting encoding Thunderbolt uses, but they all have substantial overhead, so the 10 Gbps-->1250 MBps conversion is poppycock if the Gbps figure is at the interface and the MBps figure is supposed to be meaningful for users. Ben's conversions are in Wikipedia too, until we revise it.

Dennis Couzin
Berlin, Germany

Quote

The discrepancy between 125 and 100 is much too large to be explained by the binary vs. decimal terminological matters Ben discussed.

Correct - and apologies if it wasn't made clear guys. The established overheads are already applied in the case of SATA as prescribed in the format and are a known quantity for direct SATA connection.

I should have explained that they also call SATA I, II and III by SATA (1.5G), SATA 3G and SATA 6G respectively - this infers that it's capable of 1.5Gbps, 3Gbps and 6 Gbps but ACTUALLY they are not.

Once again it is a marketing ploy! Similar to, but not the same as with the "binary vs decimal" discussion i.e. Hiding the caveats. This time ignoring the encoding and overheads in order to make the format appear to be faster than it is in real world use.

Whereas the other link protocols I listed are also variable according to different overheads and error correction as I mentioned plays a massive part on all connected media or indeed networks.

Another factor I didn't touch on that affects speed of transfer is the particular formatting of a hard disk (or RAID set) is the block size.

I also failed to mention that many of the connections (USB, FireWire, Thunderbolt) simply use a bridge-board to go to the SATA connector on a drive anyway!

So from Firewire to SATA or Thunderbolt to SATA; you will experience the bottleneck from both protocols and the time (albeit tiny) to convert it.

However my "bias" towards Thunderbolt would I thought be obvious...

20Gbps vs 1.5, 3 or 6?

eSATA won't even touch the sides of a 20Gbps link.

A single 20Gbps TB2 connection will easily handle three (and a third) parallel channels of 6Gbps SATA III.

So via TB2 you could get at least 1800MBps connected to just three SATA III channels and given enough connected drives and more channels controlled via a RAID controller possibly a lot more, maybe even all the way up to 2500MBps. (not allowing for overheads).

If you only have eSATA that is for instance at SATA II speed, then that connection will only ever be able to accommodate ~ 300MBps

Quote

but they all have substantial overhead, so the 10 Gbps-->1250 MBps conversion is poppycock

Actually I've been lead to believe the opposite...

I've seen reviews of Thunderbolt 1 enclosures boasting about 1200MBps (50MBps less than theorectial max) so must assume the overhead is actually quite low on the connection. However I will remain sceptical until I get one to play with that can do it!

Also please note this from the Thunderbolt Technology website [thunderbolttechnology.net] :

Quote

The Thunderbolt protocol physical layer is responsible for link maintenance including hot-plug detection, and data encoding to provide highly efficient data transfer. The physical layer has been designed to introduce very minimal overhead and provides full 10Gbps of usable bandwidth to the upper layers.

"very minimal overhead and provides full 10Gbps of usable bandwidth"

It doesn't appear to me that TB has anything like the sort of overheads SATA carries at all.

Please also bear in mind that TB is also bi-directional over dual-channel I/O.

eSATA or External Serial Advanced Technology Attachment is either In or Out. One at a time and NOT true simultaneous I/O.

From what I understand, given the full saturation of the TB bandwidth you could have up to 1250MBps going one way and another 1250MBps coming the other!

This means that if you are rendering out and reading from a Thunderbolt RAID you don't have to share the connection I/O bandwidth as you do with a single eSATA connector.

However one caveat for TB - a single SATA device cannot physically perform read and write operations simultaneously anyway. So the Thunderbolt RAID (most likely using SATA drives) will be task-switching IO operations internally even though the TB protocol won't be.

This then brings up the point about separate Media Drives on separate channels. One for reading (source) and a completely separate one for writing (renders/exports). On Thunderbolt it won't matter as much. On single connected eSATA you will want to keep them separate if you want to maximise your transfer IO speeds.

Now all that being said - I think between us we've made the point that speeds are not alway to be believed and the only real way is to look carefully at your entire setup and work out what is best and what is available within your budget.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Oh by the way in case its not clear.

I'm not advocating one connection over another - simply pointing out the reality of it.

I have many eSATA enclosures that as Dennis work just fine, likewise I have USB3.0, FireWire, Ethernet and the big RAIDs are all SAS.

Hopefully though this has covered most of the questions you may ever have about HDD drive speeds, some of the most common connectors and the caveats with the industry supplying them!



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Quote
Ben King
However my "bias" towards Thunderbolt would I thought be obvious...
20Gbps vs 1.5, 3 or 6?

Ben: I didn't say you had a bias towards Thunderbolt. I said

Quote
dcouzin
Ben's approximate figurings seem to show an arithmetical bias against eSATA and toward Thunderbolt

Didn't I manage, using all those words, to refer to the apparent bias in using the 100 factor for one and 125 factor for the other?

When engaged in a difficult topic it doesn't help to discard half the words in a statement. I wrote:

Quote
dcouzin
but they all have substantial overhead, so the 10 Gbps-->1250 MBps conversion is poppycock if the Gbps figure is at the interface and the MBps figure is supposed to be meaningful for users.

but you omitted all the words after "poppycock". The inconsistency in your table relating Gbps speeds and MBps speeds of the various connections depends on the omitted words. For all these connections

FireWire 400
FireWire 800
10GigE
Thunderbolt I
Thunderbolt II

you relate the speeds with a factor of 125. But for

SATA I
SATA II
SATA III

you relate the speeds with a factor of 100.

We know why the factor is 100 for SATA. It's due to SATA using 8b/10b encoding. But we happen to know that FireWire 800 uses the same 8b/10b encoding. So that's one inconsistency in the tabulation. With SATA (eSATA) the Gbps refers to the augmented data flowing through the cable while the MBps refers to the usable data that's from the cable. With FireWire 800 the Gbps and MBps can't refer to these same two quantities. With Thunderbolt we don't know. This presentation for EE students is informative. It mentions that while PCIe 2.0 uses 8b/10b encoding, PCIe 3.0 uses much lower overhead 128b/130b encoding. Thunderbolt is essentially PCIe.

SATA calls its SATA II 3 Gbps referring to its augmented data rather than the user's real data. This seems to be the common practice according to page 7 of the linked presentation. If we are now constructing tables comparing the different connection speeds, they must be constructed consistently.

Dennis Couzin
Berlin, Germany

Dennis I think you misread a lot of the posts!

Perhaps my use of some English grammatical structure and spelling is not familiar to you?

Regardless, this thread is getting a little argumentative to my mind.

Please do not misquote me:

Quote

Approx. Connection bandwidths

Firewire 400 (400Mbps or ~50MBps) rarely achieves over 40MBps
Firewire 800 (800Mbps or ~100MBps) rarely achieves over 80-90MBps
Gigabit Ethernet (1000Mbps) rarely achieves over 95MBps
10GigE (10Gbps ~ 10,000Mbps) ~ 1250 MBps
eSATA or Internal SATA:
SATA I ~ 1.5 Gbps or ~ 150 MBps
SATA II ~ 3 Gbps or ~ 300 MBps
SATA III ~ 6 Gbps or ~ 600 MBps
Thunderbolt I ~ 10Gbps or ~ 1250 MBps
Thunderbolt II ~ 20Gbps or ~ 2500 MBps

These are all theoretical maximums but error correction and other overheads take their toll and you never see perfect sustained full speed.

Notice the first and last lines?

I'm sorry if you just missed what I had written - however knowing your passion and attention to detail - I have to assume you might not understand exactly what I was saying.

"Approx." or Approximate - that is to say not accurate.

"theoretical maximums" - that is to say what the connection speed could do given no overheads.

Also!!!

The use of the tilde or "~" symbol in the English language is used to to mean "approximate"

So each reference to the "Gbps ~ Mbps" is to be considered vague NOT "Gbps = MBps"


As you know there are many more factors that govern the actual real world situation - as we both alluded to in subsequent posts.

Rather than continue over this desire for perfect detail in my "rough-approximate-inaccurate-for-real-world-calculations table" why don't you write a super-accurate encoding and overhead table instead of worrying about whether I made it clear enough?



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Ben, the problem is precisely that you did write:

These are all theoretical maximums but error correction and other overheads take their toll and you never see perfect sustained full speed.

Yet for SATA you included the hefty 8b/10b error correction factor in your calculation, while for FireWire 800 you did not include the same 8b/10b error correction factor.

For each connection method you give two speeds, one in bits/sec and one in bytes/sec. It is unclear why. Is it just as a convenience for the reader who might be interested in bits or might be interested in bytes? Or is it because in the science of data transmission these two speeds typically refer to different concepts? It looks as though it was the first reason for FireWire 800 but the second reason for SATA. This is the inconsistency I tried to point out.

I think we agree that it is the second speed (with the tilde and with many grains of salt) that matters for users. It's how many bytes of user data can be transferred per second by the connection. So we users should eschew [e]SATA II's 3 Gbps specification and prefer its 300 MBps specification, which we can understand as 2.4 Gbps if we please.

Dennis Couzin
Berlin, Germany

It's not a problem Dennis but you have made it one. The point of a caveat like "approximate" is that until you go and write the full accurate report it does not need to be - that IS the point of using a clause that gives a rough estimate or rounded figures - useful for a quick guide and not to be taken as the whole truth.

If I had stated that these are all 100% accurate I would understand your pedantic replies, however I did not. So lets leave this now as your phonograph analogy and mine are getting stuck on repeat!

I understand you are wanting more accuracy which is fine...

Please go and find the relevant and tested overheads to the other protocols and post them here and we can make up a nice (more) accurate table.

We can then add it to the Wiki page as a reference.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Tasty thread...thanks to both.

Ben's table includes

[Firewire 800 (800Mbps or ~100MBps) rarely achieves over 80-90MBps ]

So, when alerted by SMART status that one-half my internal Deskstar striped 2 TB RAID 0 was going south (after 2 years steady use), I went and bought a 4TB G-RAID box-- Firewire 800 - no SATA available- hoping I could do without such disturbing events. I tested it with Blackmagic's Drive Tester and got only 60 MB's!! It was no better than support for DV as a live edit media drive.

So this box has quickly become the backup strategy for my new internal RAID 0-- the early SMART warning allowed me to copy off hundred of gigs of media to the G-RAID, took about half a day. Picked up a new 1TB bare Deskstar on the MicroCenter refurb shelf, and a second in case, very cheap at $50 each, and re-striped the internal RAID, which tests at 160 MB and allows me to continue editing multistream HD.

I also found Ben's practical fill point for such storage at 70% or less to be very good recommendation.

I had a question about formatting a new drive for block size. Since the G-RAID is strictly a media backup, what's the ideal block size? To start, I've used 64K. What about the internal RAID 0?

- Loren

Today's FCP 7 keytips:
Copy clip Attributes with Command-C
Paste selected Attributes with Option-V
Remove selected Attributes with Command-Option-V !

Your Final Cut Studio KeyGuide™ Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

One thing many people often miss out when looking at drive speeds is the non real time aspects of video editing. Operations such as high speed scrubbing, accelerated playback and rendering are not real time. We do these operations all the time. Eg. Scrub through a video, play through an interview at 2x speed, write the final QT master, etc.. For these operations, you may prefer sufficient headroom in drive throughput. In most of these cases, the faster the drive speed, the less noticeable the lag. Sustained real time playback of a video stream should only be one of the concerns.

If you are on one of the newer machines, USB 3 looks like a fairly affordable fast interface for a RAID array.



www.strypesinpost.com

Disk transfer rates probably don't figure into rendering.
In rendering, the video is

1. read from disk
2. decoded
3. effects-processed
4. encoded
5. written to disk

For red-band (necessary) rendering, the real time engine couldn't keep up with 1-3. I think the operating system is smart enough that processes 1-3 go on simultaneously (a tad out of synch) in real time playback, so whichever process is the slowest of 1-3, probably the effects-processing, is what defeats the real time engine.

When rendering however, while processes 1-4 can go on simultaneously (a tad out of synch), processes 1 and 5 can't go on simultaneously if the two disks are same. When scratch files and render files are on the same disk, there will be read, then write, then read, then write, etc., effectively cutting the disk transfer speed in half. FCP System Settings let you put the render files in their own disk. Then processes 1-5 can go on simultaneously (a tad out of synch) and effects-processing, probably being the slowest of 1-5, will then determine render time.

Dennis Couzin
Berlin, Germany

We assume that processes 2 to 4 is the bottleneck, but that is not often the case. Eg. Rendering timecode. In my case, I have seen exports go around 1.5x real time with a slower WD fw800 RAID 0. And it goes at 0.5x real time with a USB 3 array.



www.strypesinpost.com

OK, if "rendering timecode", creating a few bits of data per frame, is a kind of effects rendering, then it's an example where the disk read and write speeds are more important than the effects processing speed.

In FCP7 lingo, "rendering" means the production of video render files and audio render files. Exporting is not rendering. For example, I always fully render before exporting, but the exporting takes quite a bit longer than the expected reading and writing should take. What does FCP7 do? Does it do a complicated check against the time line data? Does the muxing, and the construction of the QT file, take significant time?

Dennis Couzin
Berlin, Germany

Yes. Exporting and rendering in FCP lingo are 2 different things. And they are two different processes unless "re compress all frames" is checked and it usually shouldn't be.

From experience, the exporting time is another where the speed of the drive can impact exporting speed. Not sure what is going on under the hood, but it is a process that is light on the processors.

These are just some of the areas where I feel is often overlooked when people look at drive speeds, and they affect the fluidity of editing.



www.strypesinpost.com

I'm surprised nobody in this brilliant thread has wisdom about ideal formatting block size! Doesn't it matter?

And a happy 4th to all, belatedly.

- Loren

Today's FCP 7 keytips:
Copy clip Attributes with Command-C
Paste selected Attributes with Option-V
Remove selected Attributes with Command-Option-V !

Your Final Cut Studio KeyGuide™ Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

Depends entirely on the size of the files you want to use. Bigger blocks can mean faster transfer for larger more continuous files, but slower with smaller chunks of data and visa versa.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Any recommendations for HD, Ben?

- Loren

Today's FCP 7 keytips:
Copy clip Attributes with Command-C
Paste selected Attributes with Option-V
Remove selected Attributes with Command-Option-V !

Your Final Cut Studio KeyGuide™ Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

If we're talking about HDD sector sizes here, in particular the choice between 512 bytes (traditional) and 4096 bytes (advanced format), and if we're talking about video files, then the larger sector size is definitely preferred. Even heavily compressed video at 2 Mbps is using over 10000 bytes per frame.

Dennis Couzin
Berlin, Germany

So the question is, what if I need to store with the media a small text file, or a motion tracking data file, or a CC filter list file, say 64K? 128K? Will it disappear?

- Loren

Today's FCP 7 keytips:
Copy clip Attributes with Command-C
Paste selected Attributes with Option-V
Remove selected Attributes with Command-Option-V !

Your Final Cut Studio KeyGuide™ Power Pack.
Now available at KeyGuide Central.
www.neotrondesign.com

Ok now before confusion sets in!

Regarding SECTORS

You don't have a "choice" between 512 bytes (traditional) and 4096 bytes (advanced format)

This is physically set on the HDD platter. You don't get to choose this.

"Beginning in late 2009, accelerating in 2010 and mainstream in 2011, hard drive companies migrated away from the legacy sector size of 512 bytes to a larger, more efficient sector size of 4096 bytes, generally referred to as 4K sectors, and now referred to as the Advanced Format by IDEMA (The International Disk Drive Equipment and Materials Association)."


Regarding BLOCKS

Basically the "Block" size is the smallest division of the data on the disk formatted by the OS

HFS+ Volume Size and the Default Allocation Block Sizes:

If your Volume is less than 256 MB then default Block Size is 512 bytes
If your Volume is 256 MB to 512 MB then default Block Size is 1024 bytes (1K)
If your Volume is 512 MB to 1 GB then default Block Size is 2048 bytes (2K)
If your Volume is more than 1 GB then default Block Size is 4096 bytes (4K)


But to answer your question Loren - If you format a RAID with a block size larger than the files it won't mean you'll lose them.

All that will happen is it will take up the block size "as a minimum" so if you had a 64K block size and has a 32K file on it - it would still take up 64K as that's the smallest chuck of data your formatting accommodates. Essentially wasting space.

But larger Block Sizes mean faster transfer of larger files so in that respect the larger block size is preferred for video files.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

This strand concerns drives and transfer rates. You actually do have a choice about HDD sector sizes, when you buy the hard drive. This year I bought some HGST Ultrastar 7K4000s. I had a choice between the HUS724020ALE640 and the HUS724020ALA640, and chose the former for its 4096 byte sector size. One advantage of buying enclosures and drives separately is that you know what drive you're buying.

As for formatting block size (which Loren asked about and I ignored), there's where you have no choice, and formatting block size is irrelevant to this strand.

Dennis Couzin
Berlin, Germany

Dennis as usual you are getting it slightly wrong yet again. In a RAID array which I mention you CAN get to choose your block size. As for "choosing" sector size you are being annoyingly pedantic - you don't choose them after purchase they are preset - you get one and are stuck with it. Yes you can still choose to buy 512e and n sector disks but once you get one you can't change it. But we WERE talking about block size NOT sector size although it is also relevant. You have been warned on the creative cow several times for your argumentative tone - please refrain from making the same mistake here.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

Let me expand on this as the thread has already moved this way.

Regarding the Block size and the performance of a RAID

You should set the block size appropriately:

• A large block size should be better for large sequential read/write e.g. video files.
• A small block size should be better when using small file, small random read/write & metadata-intensive workflows.

Here's an Apple Support Article on it: [support.apple.com]


When creating a RAID (Usually a Hardware controlled RAID) you should have the option of setting the Stripe size. (I have never used SoftRAID or other Software RAID so I'm not sure if you have this option).

From best performance you should set the block size to match the RAID stripe size or a multiple of the RAID stripe size.

IMPORTANT: If the block size you choose does not match the RAID stripe size (or multiple of) then it can be extremely detrimental to your write performance.



For instant answers to more than one hundred common FCP questions, check out the LAFCPUG FAQ Wiki here : [www.lafcpug.org]

"Block size" has many different meanings in computer technology, and the linked-to Apple Support Article muddies matters the more.

That support article is for Disk Utility's software RAID. It says:

Quote
http://support.apple.com/kb/PH5837?viewlocale=en_US&locale=en_US

The disks in a RAID set store data in chunks know as storage blocks.

Disk Utility 11.3 gives the user five choices for "RAID block size": 16K; 32K; 64K; 128K; 256K. (It doesn't say whether these are bits or bytes.) I thought they might be stripe sizes because those numbers, in bytes, are pretty standard stripe size choices. However Disk Utility offers the same five choices for RAID 0 and for RAID 1, and RAID 1 isn't striped. (Disk Utility offers contatenation as a third kind of "RAID". It offers no "RAID block size" choice for contatenation.) I doubt that Apple's term "RAID block size" will help to clarify this discussion.

Loren's original expression was "formatting block size", and Ben helpfully rephrased this as "allocation block size". HFS+, the file system we here use, can make allocation blocks with size any multiple of 512 bytes. The allocation block size is set when the volume is initialized. I have never seen an option within the OSX GUI to set the allocation block size. (It can be done in Terminal.) Apple Technical Note TN1150 (March 2004) says: "Current implementations of the [HFS+] file system are optimized for 4K allocation blocks". Has HFS+ changed since then? Allocation block size 4K bytes is so much less than the RAID block sizes 16K to 256K, that they must be completely different things.

Introducing RAID stripe sizes into this strand about drives and transfer rates is relevant, and stripe sizes should be chosen large for video editing. But I don't think they should ever be called "block sizes", and I don't think allocation block size is relevant to the strand. Allocation block size is supposed to be small with HFS+, like 4K bytes, and all the reasonable choices of RAID stripe sizes are multiples of this, and much larger.

Quote
Ben King
From best performance you should set the block size to match the RAID stripe size or a multiple of the RAID stripe size.

Has Ben turned the size relation between allocation block size and RAID stripe size backwards, or does "block size" have yet another meaning in that sentence?

This whole strand, from its first frettings about "Kilo" vs "Kibi", has been in terminological hell.

Dennis Couzin
Berlin, Germany