Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Establishing Standard Benchmarks?

Its also important to include system specs beyone the usual. For photoshop the number of history states, Memory USED (not just system memory, but memory assigned to PS), scratch disk (size, location, speed). The test bakedbeans has is very HD and RAM dependant (much more so if you have ~20+ history states) as it writes to the scratch disk. I still maintain its TOO HD dependant because the file size is just what most deal with in PS. So times will be different if you use a scratch disk on your system disk vs a seperate fast (eg Raptor) drive.

Reporting this as well as settings should remain as consistent as possible. Maybe even telling the user to use xx History states and assigning 2GB or RAM. That is about what most people have and yet still allows CS2 to run well...yes newer systems can use more but this allows for the speed of the systems RAM to be the variable, not speed and quantity since most won't be packing it out with 4+GB. Its fairly obvious that more RAM is better, so elimiate RAM volume as much as possible.

It is critical that each test be done with settings as consistent as possible and as few variables as possible. It may require different versions of tests (one that isolates the CPU as well as one that uses the other components....like a PS test that doesn't hit the scratch drive).
 
I think doing some Automator script would work best
two sets of tests..
one with applications that come with a new mac (Finder, Ilife etc)
and the other with pro applications.

I think it needs to heavily focus on multitasking.
and multithreaded apps
 
adamfilip said:
I think doing some Automator script would work best
two sets of tests..
one with applications that come with a new mac (Finder, Ilife etc)
and the other with pro applications.

I think it needs to heavily focus on multitasking.
and multithreaded apps
Click to expand...

You'll get some high-level numbers from benchmarks like those, but they may not really show what's faster. If you have a benchmark that spends 50% of the time reading files, then CPU performance boosts are marginalized. Even if the CPU speed difference between Macs is 500%, you won't see it in this benchmark.

I agree about multithreading. Lots of benchmarks make it look like the MacBook is only modestly faster than a PowerBook, but the benchmark is just running on a single core.
 
hate to repeat efforts made...

but we may have to start this benchmark thing over...

I know the "retouchartists" one seems to have gotten a fair amount of use... but the site's down and the person that posted it seems to have been since banned.

I thought I could host the files myself, but they are specifically prohibited from being redistributed. I find this frustrating as it defeats the whole purpose of this standardized benchmark.

Since we just have the photoshop benchmark, how hard would it to be just to recreate this in more of an open method that could be redistributed more easily? MacRumors can permanently host it as well.

Someone just needs to donate an high rez image that can be used and create the sequence.

I say we create a version number for it, zip it, and allow redistribution with a text file with a link to the most recent version which will be hosted here at MacRumors. MacRumors isn't going away anytime soon.

The long term goal of this when I posted back when was to have some more standardized benches.... besides photoshop, but of course, that's of interest now since the CS3 just came out.

arn
 
Boing Manager

Howdy all. I'm new here and not at all familiar with this new Mac, but I just thought I'd contribute if it's relevant by saying I noticed, looking around at the software that downloads with the SETI project, Boing Manager, there's a little sub program in there that does a little benchmark. Handy? I don't know unless I have something to compare to. Here's what it says about my new MBP 17 C2D: 1878 Floating Point MIPS (Whetstone) per CPU
4975 integer MIPS (Dhrystone) per CPU

Is this test relevant at all?
 
What I'd like to see, at least as far as photoshop routines, is a set of routines that Photoshop has had for many versions, back to possibly 4 or 5 (who had recordable routines first? I know 5 has them) so we can compare a huge array of systems and Photoshop versions dating back 10 years or more.

Think of the comparison we could have... Power Macintosh 8100/100 vs. Mac Pro (when CS3 comes out).

That's what I'd like to see but I'm sure nobody cares what my opinion is :D
 
I'm willing to donate some RAW photo files, write some actions, and let it be freely distributed, but I'm not sure what kind of actions would be best for benchmarking. Also, my camera RAW files are approx. 7.5MB .cr2 files from a Canon 30D. Are those "big" enough? I could make them into .TIFFs of virtually any size, so I guess it wouldn't be a problem.

So if anyone could suggest some actions, I'm up for writing them.
 
Out of curiosity, since this is posted in the Mac Pro section, have you guys thought about stressing out the *real* capabilities of these machines? Apps like Photoshop are 32-bit, and so features like the x64 architecture of the Xeon chips isn't utilized (they're just running in legacy mode), and the user address space is 4gb, meaning 8GB and 16GB RAM machines won't see any benefit in your benchmarks. I also kinda doubt Photoshop will be able to scale to 8-cores effectively.

This is fine if you mainly use Photoshop, and just care about the perf improvement today. But when OSX becomes fully 64-bit with Leopard, and an x64 version of Photoshop is released, I think the true power of the mac pro's will finally be revealed.
 
Hi folks,

I have a question about Xbench; Testing a 2.66Quad with 6GB RAM gives a score of about 165 . Now I tested a 3GHz Quad with 7GB RAM and it gives 153.

Both machines have the X1900 Ati Card and both tests were done after rebooting.

Any insight?

Thanks
 
Establishing Standard Benchmarks & Xbench

It seems that most of you are against using Xbench. I am not involved with Xbench other than as a user. Let me say that as long as you are comparing your computer against others like it on Xbench what difference does it make. The same algorithms are used for your machine, and yours and oh, also yours. I dam not a geekhead, but just let me post the latest "read Me" from the Xbench web site below here:

Version History

1.3 [2006-08-13]
  • Corrected a mistake that caused the altivec test to be turned off on PowerPC machines
  • Turned off coalesced graphics updates for all platforms on Mac OS 10.4.4 and higher
  • Switched compiler to GCC 4.0 on PowerPC. This provides some boost to floating point and AltiVec scores, and these have been recalibrated accordingly. This also raises Xbench's system requirements to 10.3.9 or higher.
  • Revised machine database to include the MacBook, Intel iMac and several other models
  • Added code to dynamically load machine database on launch from the Xbench website
  • Added support for temporarily turning off beam sync on Tiger while running graphics tests
  • Fixed an issue that causes Xbench to fail to launch on Leopard
  • Built with Xcode 2.4
What's wrong with that. Please enlighten me.

thanx,
gholmes
 
finiteyoda said:
Out of curiosity, since this is posted in the Mac Pro section, have you guys thought about stressing out the *real* capabilities of these machines? Apps like Photoshop are 32-bit, and so features like the x64 architecture of the Xeon chips isn't utilized (they're just running in legacy mode), and the user address space is 4gb, meaning 8GB and 16GB RAM machines won't see any benefit in your benchmarks. I also kinda doubt Photoshop will be able to scale to 8-cores effectively.

This is fine if you mainly use Photoshop, and just care about the perf improvement today. But when OSX becomes fully 64-bit with Leopard, and an x64 version of Photoshop is released, I think the true power of the mac pro's will finally be revealed.
Click to expand...

Make that an _SSE4_ 64-bit version :)
 
I just cannot believe there isn't a decent standard benchmark program for Macs. I mean come on!
 
Blender is a free 3D modeling program
Which stresses the system CPU completely not so much the RAM or the hard drive speeds.

for more information go to: www.blender.org

Here are is a "unofficial" Benchmark that we can all use
or can be Macrumors "Official" Benchmark, lol.

http://www.eofw.org/bench/


My :apple:MBP Early 08
2.5Ghz dual core.
4GB RAM
8600M 512 VRAM
250Gig HD

with ONE thread in use: 2:16.48
with TWO threads in use: 1:15.39

There you are a Great "Realworld" benchmark that does not cost money like Photoshop and is a small download
~16MB all up including the "Test" file.


What Software I used
-Blender 2.48 which is compatible with both windows and all UNIX systems
-test.blend (the test file :D)


Here is what the screen looks like :D

~Regards
Name101
 

Attachments

  • screen-capture.jpg
    screen-capture.jpg
    362.4 KB · Views: 235
Using 8 threads I can get 00:27:24s. With 4 threads I get 00:29:62s. This is in OSX 10.5.5. I'm going to check and see if there's any difference between platforms. OSX vs Vista.

Edit: I just ran the test in Vista x64. With 4 threads I get 00:27:35s. With 8 threads I get 00:25:21s. Slightly faster. Perhaps the 64bit blender exe is taking more advantage of the 64bit capabilities of the CPU in Windows than in OSX. I don't really know. Also, it could be due to the fact that the Vista tests were run on a freshly booted up machine. If there's a difference after booting back into OSX I'll make a note of it.

It's worth noting that after glancing over the results browser of this little test my scores in both Windows and OSX are right where they should be. All the 8 core systems are getting around 12-13 seconds. My system has 4 cores and as such should the render should take twice as long as it does on the 8 core systems; which it does. 3D renders make good tests in at least the respect that performance scales very linearly with the number of cores.
 
Ok
That's Interesting.
I have only played with two cores. but going from 4 to 8 is a lot less than I thought.. Maybe some of that time goes to "prepping the Scene"

I don't know. Maybe I should try and get my hands on a more complex "*.blend" file.

I do have a file.. But I am unsure if I'm allowed to redistribute. I did download the file off a forum and the user was offering the file for free.
I have Messaged Him about the use of the file.

But with 2 Cores running it did take my computer 23:56.13 To render.
I think this will allow for the 4 extra cores on your desktop to really show an advantage.

~Regards
Name101



EDIT: He just replied to the PM. He said We (macrumors) can do what we want with it. :D

Here is the File and a small screen shot on what the image is.
 

Attachments

  • screen-capture.png
    screen-capture.png
    41.1 KB · Views: 929
  • Benchmark.blend.zip
    45.2 KB · Views: 149
Blender Test. Not much of a difference between the platforms. I used a 64bit kernel with OS X, so the results seems pretty close to Vista 64bit.

OS X 16.78s
Vista 64 16.34

8-threads, xparts =8, yparts=8.
 
Exman said:
Blender Test. Not much of a difference between the platforms. I used a 64bit kernel with OS X, so the results seems pretty close to Vista 64bit.

OS X 16.78s
Vista 64 16.34

8-threads, xparts =8, yparts=8.
Click to expand...

Good grief man. Your 4 core Hackintosh is performing nearly the same as the high end 8 core Mac Pros in this test and in Geekbench. Overclocking FTW!
 
Exman said:
Not bad for a $1,000 setup eh ;)
https://forums.macrumors.com/posts/6370013/

The next upgrade for me is Nehalem. When I saw an early overclocking screenshot of one @ 4Ghz, 20% faster at the same clock speed. I just want one :)

Cheers,
Click to expand...

Now that it's fall and the temperatures are coming down I'm going to clean my case out, organize my airflow a bit better, and push for 4GHz. I'd like to break the 9000 Geekbench barrier with my $1000 4 core setup too. :D

I just can't help but smirk when I walk into an Apple store and see the price tags on those fancy looking 8 core Mac Pros in the back. It's especially satisfying running benchmarks on those machines and seeing that the score of my Hack get pretty dang close to a machine with 4 additional cores. If only they made dual socket 775 motherboards. One can dream.
 
I think you can hit 9,000 Geekbench score if you can get that Q6600 to 4GHz and maybe tighten the timings to 4,4,4,8 3, 30 or there about :)

Penryns have 12MB L2 cache and SSE 4 so Geekbench likes it better at the same speed. There is one distinct advantage desktops have over Mac Pros - that is the RAM performance (latency and bandwidth); hence it isn't surprising to see 4-core Hackintosh's outpacing 8-core Mac Pros in the Photoshop tests ;)
 
Exman said:
I think you can hit 9,000 Geekbench score if you can get that Q6600 to 4GHz and maybe tighten the timings to 4,4,4,8 3, 30 or there about :)

Penryns have 12MB L2 cache and SSE 4 so Geekbench likes it better at the same speed. There is one distinct advantage desktops have over Mac Pros - that is the RAM performance (latency and bandwidth); hence it isn't surprising to see 4-core Hackintosh's outpacing 8-core Mac Pros in the Photoshop tests ;)
Click to expand...

I've having a tough time getting my system to boot into OSX at anything higher than 3.71GHz. It's not temp related because my temps don't get any higher than the mid 40s and my cooling setup is pretty good. My guess is my ran is holding me back. It could also be my motherboard as it's a 1st gen GA-P35-DS3L and I've never even bothered to update the BIOS.

I've got another Hack running with a 3.1GHz Core 2 Duo in a newer GA-EP35-DS3L. I might switch the motherboards around and see if I have better luck. The EP35 also supports higher frequency RAM and that might help out. Unless, of course, my RAM is the limiting factor.

Sometimes I'm tempted to sell my quad core system and use the money to start piecing together an 8 core Hack Pro with a Skulltrail motherboard. The idea of being able to overclock a pair of quad core Xeons sounds very sexy. With that sort of setup it seems like one could absolutely destroy the 8 core Mac Pro performance in OSX.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back