Test Driving SPECviewperf 2020: A Look At The Latest In Workstation GPU Benchmarks

Over the years, the Standard Performance Evaluation Corporation's SPECviewperf benchmark has become the industry standard for workstation GPU benchmarking. Owing to the fact that, unlike video games, there's little concept of a "standard" workload for CAD, content creation, and visual data analysis tools, as well as the sheer complexity of these applications, there is an ongoing need for a standardized multi-application benchmark. This is both to offer a wider, holistic view of GPU performance under workstation applications, and even more fundamental than that, to provide a proper apples-to-apples testing environment.

With updates to the workstation applications used to generate the traces in SPECviewperf 13, the benchmark has been due for a refresh – and that has finally arrived this week with the release of SPECviewperf 2020. An incremental update over SPECviewperf 13, the 2020 version makes some important changes including support for 4K resolutions, as well as updating the workload traces.

Meanwhile, with a few systems already setup for other needs, we decided to take a quick test drive of the new version of the SPECviewperf, giving us a better look at what's new along with an idea of where performance lies with the updated benchmark workloads.

SPECviewperf 2020: What's New

SPECviewperf 13 has been a component in many of our system reviews - particularly on ones with a discrete GPU that target both the gaming and professional / content creation markets. The new version of the benchmark, SPECviewperf 2020, brings along the following updates:

• Removal of the Showcase trace (showcase-02 viewset) from the suite (Autodesk no longer provides support for the application)
• Updated traces using latest versions of the applications (3D Studio Max, Catia, Maya, and Solidworks)
• Updated models in select viewsets (3dsmax-07, catia-06, creo-03, medical-03, and solidworks-05)
• All viewsets capable of being processed at both 1080p and 4K resolutions (SPECviewperf 13's 3ds Max and Showcase viewsets were 1080p-only)
• Advisory notes on minimum amount of necessary VRAM, as well as TDR settings for systems with entry-level GPUs

There are also some updates to the user interface and benchmark results management. Command-line options are supported officially. With SPECviewperf 13, it was unofficially possible by calling the Node-WebKit executable with the appropriate options. The new launcher in SPECviewperf 2020 makes things more straightforward.

Evaluation Setup

The SPECviewperf 2020 benchmark license makes it available for free to everyone other than sellers of computers and related products. We downloaded the latest version and took it for a test drive with some of dGPU-equipped systems processed recently in our testbed.

• Skylake-based Desktop - I (Intel Core i5-6600K / NVIDIA RTX 2070) [ Review Configuration, with the ASUS Dual GeForce RTX 2070 MINI in the first PCIe x16 slot ; Denoted by SKL-DT-2070 in the graphs below ]
• Skylake-based Desktop - II (Intel Core i5-6600K / AMD Radeon RX Vega 56 nano) [ Review Configuration, with the PowerColor RX VEGA 56 8GB HBM2-NANO Edition in the first PCIe x16 slot ; Denoted by SKL-DT-RXV56 in the graphs below ]
• Intel NUC9i9QNX Kit (Ghost Canyon NUC) (Intel Core i9-9980HK / NVIDIA RTX 2070) [ Review Configuration, benchmarked with a PCIe 3.0 x8 link between the GPU and the CPU ; Denoted by GCNUC-2070 in the graphs below ]
• Intel NUC9i9QNX Kit (Ghost Canyon NUC) (Intel Core i9-9980HK / AMD Radeon RX Vega 56 nano) [ Same configuration as above, except for the replacement of the GPU ; Denoted by GCNUC-RXV56 in the graphs below ]
• MinisForum EliteMini H31G (Intel Core i5-9500F / NVIDIA GTX 1050Ti) [ Review Configuration ; Denoted by MF-H31G-1050Ti in the graphs below ]
• ASRock DeskMini Z370 GTX1060 (Intel Core i7-8700 / NVIDIA GTX 1060) [ Review Configuration ; Denoted by ASR-DM-370-1060 in the graphs below ]

The SPECviewperf 2020 workloads are meant to be processed on powerful discrete GPUs, but that doesn't preclude them from being run (sub-optimally) on integrated graphics. In order to get an idea of how the benchmark fares with iGPU solutions, we also added two top-of-the-line ultra-compact form-factor systems to the comparison points.

• Intel's Frost Canyon NUC (Intel Core i7-10710U / Intel UHD Graphics) [ Review Link, benchmarked configuration with updated storage and RAM ; Denoted by INTC-NUC10i7 in the graphs below ]
• ASRock Industrial's 4X4 BOX-4800U (AMD Ryzen 7 4800U / AMD Radeon RX Vega 8 Graphics) [ Launch Coverage, full-length review in progress ; Denoted by ASRI-4X4-4800U in the graphs below ]

The command-line interface was used to test at the two official resolutions. All viewsets were processed together (similar to the official submission candidate requirement). In some cases, one or two of the viewsets were not processed properly in the first pass - in those cases, we triggered the command again with those particular viewsets alone. On specific systems, we found that certain workloads would consistently just force quit, hang, or trigger TDR (more on this in the next section). Those generate 'null' scores or blanks in the SPECviewperf 2020 results report. The composite scores for such viewsets are set to 0 for graphing purposes. Though each model component in the viewset generates a FPS metric, only the composite scores are graphed in the section below. A link to the full results dump from the benchmark is also presented in the conclusions.

Workload Descriptions and Testing Results

SPECviewperf 2020 includes eight different workloads (viewsets) representative of graphics content and behavior of actual applications. They make use of the OpenGL 4.0 and DirectX 12 APIs under Windows. These viewsets can be run only at two desktop resolutions (1920 x 1080, and 3840 x 2160) for official submissions, and need the dsplay scaling to be set to 100% (DPI of 96). The available viewsets are listed below.

• 3ds Max (3dsmax-07)
• CATIA (catia-06)
• Creo (creo-03)
• Energy (energy-03)
• Maya (maya-06)
• Medical (medical-03)
• Siemens NX (snx-04)
• Solidworks (sw-05)

We processed SPECviewperf 2020 at both resolutions on all the system configurations listed in the previous section. The benchmark measures the frame rate at which the GPU renders the scenes in a viewset. Each viewset is composed of different scenes and rendering modes, and the composite score for the viewset is a weighted geometric mean of the FPS measured for the different scenes.

3ds Max (3dsmax-07)

The 3dsmax-06 viewset comprises of 11 different scenes. They have been created from traces of the graphics workload generated by Autodesk 3ds Max 2016 using the default Nitrous DX11 driver. Additional details are available here.

nVidia's GTX 1060 and higher-class GPUs perform better than the Radeon RX Vega 56 nano at both resolutions for the 3ds Max viewset. The 1050 Ti lags behind it, though. The PCIe 3.0 x16 configuration also offers better performance scaling for the RTX 2070 compared to a x8 link (in the SKL-DT configuration, against the GCNUC one). For the RX Vega 56, the performance difference is not as significant. On the iGPU front, AMD's Renoir iGPU offers more than double the performance of the Comet Lake-U iGPU (though consumers should not be running these types of workloads on those machines).

CATIA (catia-06)

The catia-06 viewset comprises of 8 different tests created from traces of the graphics workload generated by the CATIA V5 and 3DEXPERIENCE CATIA applications from Dassault Systemes. Additional details are available here.

The catia-06 viewset had a consistently reproducible issue on AMD GPUs when processing the 1080p CATIA_3DX_loftJet model. TDR would keep triggering, and disabling TDR ended up with a hanging run. Hence, the 1080p scores for the three AMD GPU configurations are 0 in the above graph. Interestingly, the 4K model had no issues in getting processed correctly on all the systems. The driver version and additional configuration details are available in the generated reports linked in the final section.

Another aspect to note in the above composite scores is that the CPU capabilities seem to have more impact compared to the PCIe link-width. The Ghost Canyon NUC with its 8-core Core i9-9980HK capable of bursting higher than the 4C/4T Core i5-6600K in the SKL-DT system actually ends up at the top of the graph with a slight lead over the GCNUC configuration. This impact is also seen in the 4K results for the Radeon RX Vega 56.

Creo (creo-03)

The creo-03 viewset comprises of 13 different tests created from traces of the graphics workload generated by the Creo 4 application from PTC. Additional details are available here.

The relative ordering of the various GPUs is similar to what was seen in the 3ds Max and CATIA viewsets. Like the CATIA results, we see that the impact of the faster CPU overshadows the cutting down of the PCIe bandwidth in half for the RTX 2070. The Radeon RX Vega 56 performs similarly in both configurations.

Energy (energy-03)

The energy-03 viewset comprises of 6 different tests based on techniques used by the OpendTect seismic visualization application. It must be noted that two of the six tests require at least 1GB of frame buffer - failing that, the FPS is set to 0.01 (ensuring that the contribution to the composite score is not meaningful). Additional details are available here.

At 1080p, all four leading configurations using the RTX 2070 and the RX Vega 56 perform similarly. The minor surprise is that the CML-U iGPU performs slightly better than the Renoir's. The latter observation can also be made for the 4K resolution results. However, at that resolution, the RX Vega 56 seems to get impacted by the PCIe link bandwidth, with the performance in the x8 link of the Ghost Canyon NUC falling behind the x16 one of the Skylake desktop. The RTX 2070 seems to be GPU-limited in what it can score, with no discernible performance differences between the GCNUC and SKL-DT configurations.

Maya (maya-06)

The maya-06 viewset comprises of 10 different tests based on traces of the graphics workload generated by Autodesk Maya 2017. Additional details are available here.

Intel's CML-U iGPU is unable to process the maya-06 viewset (force quits consistently), and hence scores 0 for both resolutions. Similar to 3ds Max, the PCIe link bandwidth seems to matter more than the CPU capabilities for Maya. The relative ordering of the performance of the various GPUs is similar to what was observed in the previous viewsets.

Medical (medical-03)

The medical-03 viewset comprises of 10 different tests derived from 5 distinct datasets. The tests use volume rendering techniques commonly adopted for medical visualization - slice rendering and ray casting. Additional details are available here.

The medical-03 viewset is the first one in which the Radeon RX Vega 56 outscores the RTX 2070 by a good margin. The results do not appear to be affected adversely by halving of the PCIe link bandwidth or change in CPU capabilities, pointing to the GPU-limited nature of the workload. Building on the AMD GPUs' suitability for this workload, the iGPU in Renoir delivers 9x the performance of the CML-U version (though, again this is of academic interest only)

Siemens NX (snx-04)

The snx-04 viewset comprises of 10 tests created with traces from the graphics workload generated by the NX 8.0 application from Siemens PLM. Additional details are available here.

The snx-04 viewset is another one in which the AMD GPUs win big - in fact, the Renoir iGPU even outscores the mighty RTX 2070. It is possible that the features required to get the Siemens NX viewsets processed optimally are available only in the professional Quadro / ProViz GPUs from nVidia. In other observations, we see neither the PCIe link bandwidth, nor the CPU capabilities affecting the scores in a major fashion.

Solidworks (sw-05)

The sw-05 viewset comprises of 10 tests created from traces of Dassault Systemes’ SolidWorks 2020 application. A minimum of 4GB of VRAM is required for these traces, failing which the composite score is best discarded / not used to determine the performance of the Solidworks 2020 application on that machine. Additional details are available here.

The iGPUs do not have the required 4GB frame buffers to enable the processing of the 4K version of the Solidworks viewset. Scores could be obtained for the 1080p version, though. In that, we find the CML-U and Renoir scores to be within touching distance of each other. The relative ordering of the GPUs is similar to what was seen in the first five viewsets, with the RTX 2070 scoring big over the RX Vega 56. Halving the PCIe bandwidth appears to slightly affect the scores, but the performance drop is not very significant.

Concluding Remarks

SPECviewperf 2020 has a well-built results browsing interface. In addition to the FPS values contributing to the composite score, the HTML report also has the screen grabs from the viewsets' processing. A configuration page also helps readers to get an idea of the system components and driver versions used to generate the results. The table below links to the detailed results for the graphs above.

Our test drive provided interesting results with which we could analyze the effects of the PCIe link width and CPU capabilities on the performance of different viewsets. It also showed us that the benchmark could be run largely successfully even on entry-level iGPUs, though no consumer should be doing that in a business / production environment. We could have delivered more interesting results with workstation GPUs (given that their feature set and drivers are specifically tuned for these types of applications), but the consumer-focused GPUs used above were good enough for a quick look at the capabilities of the new benchmark.

SPECviewperf 2020 is an incremental update over SPECviewperf 13, but an important one - not only does it update the traces using the latest versions of the softwares from ISVs, it also allows benchmarking at 4K across all viewsets. The infrastructure has become a little more automation-friendly compared to SPECviewperf 13. Overall, the updated version ensures that SPECviewperf remains the go-to benchmark suite across the industry for system buyers wanting to determine the workstation / GPU combination that can deliver the best value for money for their particular workload.