UPDATE 3 - You can now find all posts related to this subject under the new 'Test Mule' tag.
UPDATE 2 - If you'd like to run the same battery of tests at home or on your own machine, take a look at my latest post, here.
UPDATE - Adobe has added code to the latest update of Photoshop CC (14.2) which makes better use of the GPU. This allows for drastic speed increases when running functions that utilise this new functionality. Sharpen times are 4x faster on the nMP over older versions of Photoshop CC. Hopefully this technology will make its way to Lightroom given time. More details over at Macintosh Performance Guide: here & here
When the New Mac Pro (Late 2013) was originally announced back at WWDC in June 2013 I was literally rabid with anticipation. I spend more time with my computer than any other tool (including any single camera) and as a long-time Apple geek, the thought of a new, shiny and wickedly fast workstation class desktop was extremely appealing.
The wait for the machine, from unveiling to release, was long. Extremely long. Apple eventually sneaked some deliveries in under its self-imposed December deadline but most people are still waiting to receive their machines. One benefit of the delay, from my point of view, is that it gave me a chance to talk myself down from an immediate purchase. With a machine as new as the nMP there are a LOT of unknowns and trying to figure out which model to go for, and whether it's worth it, is kind of tricky. Despite wanting a nMP with all of my body and a good proportion of my soul, it's a significant investment and I decided that an informed purchase (or non-purchase) was the way to go.
The machine has been available to order for pretty much a month now, and while review units are out there in the wild and some detailed in-depth reviews have been posted - most notably by AnandTech - the lack of information pertinent to myself as a photographer - and potential buyer - was becoming frustrating. For a wedding photographer January and February are good months to buy a new computer. It's the quiet season right now and I'm not shooting every weekend. That gives me time to shakedown a new computer and getting things running just so. By April/May things are going to be extremely busy and I simply won't have time to introduce a new machine to the workflow. With the weeks marching past I decided that I better go and try and source some of my own benchmarks for the machine. I found out that my local Apple store had a Mac Pro and so I decided to drop in and run as many tests as possible.
My specific interest was how Adobe Lightroom would perform on the Mac Pro. I'd seen limited benchmarks from elsewhere, but I needed far more insight. The Mac Pro is being sold as a 4K video behemoth and while that ability is extremely impressive, I need insight that's more applicable to myself as a professional photographer.
On arriving in the store I found that they had a 6 Core nMP hooked up to a 27" Thunderbolt Display. The specs were as follows:
- Xeon 6c @ 3.5GHz
- 16GB RAM
- 256GB PCI-E Flash
- Dual FirePro D500 3GB
- Running Mac OS X Mavericks 10.9.0
They also had a tricked out CTO iMac with the following spec that I also benchmarked:
- Core i7 4c @ 3.5GHz (Haswell)
- 8GB RAM
- 3TB Fusion Drive
- NVIDIA 780M 4GB
- Running Mac OS X Mavericks 10.9.1
- i7 4c @ 3.4GHz (Sandy Bridge)
- 16GB RAM
- 256 SSD (Used for all tests)
- 2TB HDD
- AMD Radeon HD 6970M 1GB
- Tests run under 10.8.5 and 10.9.1
On arriving in the store, after having a quick gawp/play/fondle, I found an assistant and asked about the possibility of running some tests. Despite this being far from standard practice, all the staff were extremely helpful and a real credit to the shop. We quickly got the Lightroom 5.3 demo installed and I offloaded my 100 image 'Test Mule' folder onto the desktop.
Time at this stage for some notes on the test files:
- The 'Test Mule' folder which was used for all the LR tests consists of a mixed bag of 100 wedding images. These have previously been edited so that I can simulate both importing raw, and exporting final, images.
- The 100 images are made up of 93 .cr2 raw files (from 2x Canon 5DIII), and 7 TIFF files which have been round-tripped to PS.
- The raw files are accompanied by XMP sidecar files.
- The total size of the 'Test Mule' folder is 3.85GB
- Finder - Duplicate 3.85GB 'Test Mule' folder
- Lightroom - Import 100 files (copy) & render standard previews (2048px & high)*
- Lightroom - Export 100 edited images† as resized and optimised JPEGs**
- Lightroom - Export 100 edited images as 'Unadulterated' JPEGs***
- Lightroom - Render 1:1 previews for 100 images
I'm going to move through each test one-by-one and use some charts to better visualise the data. First, off lets take a look at the finder duplication test.
The finder duplication test is really a quick way and dirty way of testing the main internal storage of each machine. Alongside the mix of PCI-E Flash, SSD and Fusion drives in the various computers I also took a moment to test the old-school spinning platter HDD in my current iMac. It has two internal drives, an SSD which I use for work, and a HDD which I use for other things or storage that doesn't need to be speedy. Unsurprisingly the HDD can't keep up.
I'll say now that of all the abilities the nMP possesses, the rapidity of it's internal PCI-E based flash storage is the most impressive. It duplicates a 3.85GB file in well under 10 seconds. That is a downright astonishing figure. I tested this three times as I didn't quite believe the number. The figure in the table is an average of all three, which were 9.0, 9.4, 9.8sec respectively. The next closest machine, the Late 2013 iMac, though extremely speedy, doesn't even come close, taking over three times as long as the nMP. Raw storage throughput in the nMP is insane.
Next up, let's look take a look at the import test. In running this test, first I made a copy of the 100 images (but not the XMP sidecar files) in a separate folder. I then asked Lightroom to copy and import these files and render standard previews. The previews were set to be 2048px and high quality.
This test through-up an interesting anomaly. Despite the much quicker storage in the nMP the Late 2013 iMac consistently beat the nMP during import testing. I don't have a good explanation for this at present and it's a test that I'd like to rerun. Nonetheless both the nMP and the Late 2013 iMac are markedly quicker than my current Mid 2011 iMac. It's interesting to note that this action is significantly faster under Mavericks than Mountain Lion. Because of this fairly noticeable discrepancy, I wonder if the difference between the nMP and the iMac could be down to the fact that the nMP was running 10.9.0 and not the latest 10.9.1. I don't know for sure however, it's just speculation.
Now let's turn to the export testing. For me, export testing is the number one time sink in LR. Exporting a large wedding of 400-500 images or more can take a significant amount of time. Often I'm also exporting these images in a variety of formats, web ready and full size for example, simultaneously. This test was designed to reflect a common export setting that I use. 3600px on the long edge, 90% JPEG quality, low output sharpening for matte paper. Here are the results:
I was quite pleased to see this result, as I feared that the Mac Pro may have ended up level pegging with the Late 2013 iMac. Although it's hard to determine which part of the system infrastructure is responsible, the Mac Pro shows a distinct, though not extreme increase in performance over the CTO iMac. Compared to my own machine, the Mac Pro finishes the test over a minute an a half faster.
Finally, I thought it would be useful to do a similar test in which Lightroom exports full size 'unadulterated' JPEGs. That's to say, 100% quality, original resolution, no output sharpening.
I expected a largely similar result to the previous test and that's exactly what we got. Regardless of the amount of processing being done to the exported JPEGs the Mac Pro retains a solid lead.
The next element I wanted to look at was exactly how much gain the new Mac Pro was bringing to the table over other models. To do this I decided to look at results in terms of percentage gains. To begin with, I thought it'd be interesting to compare storage speed, as measured in the 'Duplicate Folder' test. I've decided to present this test as it's own separate chart because the Mac Pro is so massively superior that it skews how we perceive the other figures when presented alongside tests which resulted in more modest gains.
As you can see, the nMP offers extreme gains over any other past or current Mac when it comes to storage speed.
Next up, I wanted to compare the percentage gain that the Mac Pro offers over the current top end CTO iMac in the Lightroom tests.
The import test, as I mentioned earlier, warrants further testing, but for the other tests the Mac Pro offers reasonable gains of around 10-20% over the Late 2013 CTO iMac. Not life changing, but repeatable and distinct advantages. So how about in comparison to my current machine? What sort of gains are available there?
Here we see gains in excess of 30% in all areas. Again, not huge jumps but certainly significant. It's also worth noting these gains may increase as additional workload is added. My current iMac and the CTO iMac in the store are both quad-core machines. The Mac Pro I tested was a hex-core model. In my personal usage scenario I typically demand the most out of my computer on Thursdays during the middle of the wedding season. It's normally on Thursdays that I complete one edit and prepare it for upload/dispatch to the client. Often I end up doing the following tasks concurrently:
- Lightroom - Export images at high resolution
- Lightroom - Export images at web resolution
- Lightroom - Export images for slideshow
- Aperture - Render 1080p slideshow
Finally, I decided that it'd be interesting to take a look at things from a price/performance standpoint. I'm no mathematician (where's my brother when I need him) and so this is a fairly rudimentary comparison but I decided to work out an average performance increase over our baseline, my current Mid 2011 iMac. I did this by simply totalling the percentage performance increase of all the tests and dividing it by the number of tests. On the one hand, this is skewed by the margin of victory in the folder duplication test, but on the other that's a performance metric that will have an impact on a great many day to day operations. What I mean by that is that fast storage is useful in pretty much all situations and to under value it would be misguided. I then popped over to the UK Apple Online Store and totted up the price of the CTO Late 2013 iMac and the nMP that I'd tested.
The Mac Pro costs roughly 40% more than the CTO iMac and if we look back to earlier charts we can see that in LR we're only getting 10-20% speed increases. By that reckoning, you're paying a lot more for not a huge amount more in useable power (at least from a Lightroom users POV). However, as I mentioned earlier, the Mac Pro should come into it's own when performing a range of heavy-lifting tasks concurrently. If I ever have the opportunity I'd be interested to see whether the additional cores and faster storage can increase the nMP margin of victory over the CTO iMac.
The other area that will play a part in any purchase decision is the possibility, in the future, of better leveraging the FirePro cards. At present, Lightroom (and many other programs) make limited use of the dual GPUs in the new Mac Pro. Given time I would imagine this will change. Technology such as OpenCL will, I hope, become more and more common, allowing for the use of the GPU as another processing engine for general (non-graphics) tasks. The nMP is a machine in which much of its overall computer power resides not in the CPU but in the GPUs. As with so much Apple kit, it's a machine ahead of its time and to a certain extent it is waiting for technology to catch up and meet it. If a future version of Lightroom includes OpenCL support then the Mac Pro is sure to pull further ahead of the competition. Adobe, I hope you're listening!
A final point to consider is the core usage and how this will play a role in which model represents the best bang for the buck.
The above image is from one of the export tests performed on the new Mac Pro and shows the CPU monitor built into OS X's Activity Monitor. Each bar represents one virtual core (via the hyper threading feature on recent Intel chips each core can effectively operate as two logical cores). As you can see Lightroom is making use of only one of the logical cores for each real core, but it is distributing the load effectively.
That said, I'd argue that the 6-Core model is probably the sweet spot for photographers. As core count rises, the base clock speed descends on the chip and for applications that aren't massively parallel, such as Lightroom, higher clock speed is probably of greater value than a greater number of cores. For example, while the 4-core model has a base clock speed of 3.7GHz the 12-core model has a base clock of 2.7GHz.
Also, depending on how many cores are active, the chips maximum clock speed (turbo boost) varies. The 6-core model is particularly interesting because it keeps pace with the 4-core model (which has the highest base clock speed) until it reaches 5 cores or more so although the base clock speed is slightly lower the 6-core should be able to do everything that the 4-core can do and also offers the additional cores for situations where the workload can be effectively split to more than 4 cores. More information on that at the below links:
- Marco Arment - Turbo Boost and the New Mac Pro's
- Mac Performance Guide - 2013 Mac Pro: Choosing the CPU
That's pretty much as far as my analysis can go at the moment. I personally remain undecided on whether or not to purchase. However that is at least in part due to the fact that I am also a flight simmer and I've yet to test the software I use for (X-Plane) on a Mac Pro with the high-end dual D700 GPUs. While the new Mac Pro lost out to the high-end CTO iMac in X-Plane tests, in terms of Lightroom performance, while the difference isn't startling, there's no doubt that the Late 2013 Mac Pro is currently the fastest Mac available for photographers who make serious use Lightroom.
The question everyone has to ask themselves is:
"Is the speed increase worth it for the cost of the upgrade?"
That's a question for each individual to answer, but hopefully these tests will help you arrive at your own decision.