Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Establishing Standard Benchmarks?

arn

arn

macrumors god
Original poster
Staff member
Apr 9, 2001
16,409
5,865
Every time a new computer comes out, there's a lot of talk about benchmarks... but there just isn't a great standard set.

Do people want to put together a relatively standard set with some standard software to post so when new computers come out this year, we'll be able to to point to these files and say... try these and post your results.

XBench seems pretty useless.

Unfortunately, not everyone has Photoshop... but some standard photoshop scripted filters, some music and video encoding with iMovie and iTunes, perhaps some handbrake encoding. Again... video files will need to be posted to download, so you can't use copyrighted materials.

anyone want to post what they would want to have tested? I think it's fine to have some test with Pro software, but also needs to have some software that comes with every Mac (iLife would be good to use).

I think over this year, this will be very important to have especially when the new intel macs come out.

arn
 
There are plenty programmers on these forums too - if you wanted to put together some (non-application) hardware benchmarks; which could be used as the standard benchmark at least on Macrumors.
 
whooleytoo said:
There are plenty programmers on these forums too - if you wanted to put together some (non-application) hardware benchmarks; which could be used as the standard benchmark at least on Macrumors.
Click to expand...

well, I think there are a lot of challenges in developing a good benchmarking program. hence the issues with Xbench. I think sticking to practical applications would be far more useful. ie. please download this video, and do these steps and post the times here.

yellow said:
Is there some argument against using XBench?
Click to expand...

yeah, Xbench seems very inconsistent, and even when you get numbers, you don't really know what to do with them.

arn
 
yellow said:
Is there some argument against using XBench?
Click to expand...



Yes..

Sorry about cross-posting this arn.But I feel like people arn't getting the message concerning XBench

I am under the impression that since the Intel Mac's have come out and we DTK folks are getting new iMacs I can tell you this much..

XBench's last revision of 1.2 came out in July 2005.

The last beta version of OS X for Intel 10.4.3 came out November 2005

Now..Before OS X for intel came out in november there was no support for Altivec..
And..XCode Tools were at version 2.0 beta I believe..This also did not have Altivec in it.
It had to be done using emulation ( Rosetta )

XBench 1.2 HAD to have been compiled using older compilers without Altivec support..

OS X 10.4.4 for Intel now has native support for Altivec..

This makes XBench report bad numbers..
 
Peace said:
XBench 1.2 HAD to have been compiled using older compilers without Altivec support..

OS X 10.4.4 for Intel now has native support for Altivec..

This makes XBench report bad numbers..
Click to expand...

beyond all that, even with doing practical benchmarks, there are going to be some version-ing issues... but it means a lot more to someone saying. In Photoshop Rosetta, it was 10% slower on the intel mac than the intel G5.

or, in Native Intel Photoshop, it was 50% faster than on the g5 Photoshop etc...

vs. the "CPU Test" in Xbench was 10% slower, while the threaded test was 50% faster. ?? what does that mean.

arn
 
This seems like an great idea. It would be good if we could all have easy, organised access to the data as well or am I asking too much.

Many new mac owners (and there will be lots of them this year ;)) want to know about the iLife apps.

I would like to see:

Handbrake encode a large file to iPod format (need to specify the settings and have a standard none copyrighted file available)

iMovie encode - could use the same file as above.

iTunes rip a CD... can't have a standard unfortunately... unless everyone loves Milli Vanilli ;)

Photoshop actions - standard file would again be good. There are enough photoshop users to test this properly.

I think it would be important that users have specific hardware details that they must state like CPU, RAM, GPU etc... sometimes people forget.

And rules... like reboot before test etc..
 
yellow said:
Is there some argument against using XBench?
Click to expand...

I haven't used XBench, so I can't comment; but developing "our" own set of benchmarking software would give us more control over the tests.

Features I'd include:

- Benchmark test browsable and downloadable through the benchmarking tool itself (with an iTMS-like UI).

- All dependent raw data files (QuickTime movies etc.) similarly would be downloadable via the tool.

- The tool would enable the user to submit the test results, and (using the same UI as above) give a direct comparison between their Mac and other submitted results, highlighting other similarly specified machines which achieve better results etc. (to highlight potential problems with the user's Mac).

I don't know if these are features of XBench today.

Given that so much software these days is dependent on Apple provided frameworks/APIs, benchmarking software is probably more valid than it would have been previously. I.e. most Mac browsers now use WebKit, so benchmarking software that tested WebKit load & render times would be similar to an application test, etc.
 
arn

i can 'donate' large tiffs , with or without layers - and any actions needed to do a test.

and if there is anything else i can help with just scream - i think making hte macrumors.com test/benchmark or MR real world bench tet - whatever available to download from MR would be awesome

all in one little package
 
Thanks for not slapping me arn :)

Using laymans terms.Steve Jobs was not exagerating when he said the new iMac is 2-3 times faster..It is..

Any software that is built as a universal app or native ( Intel only ) will run faster.Much faster than the G5 iMac..

Some software.Stuff that doesnt depend on Altivec engine and is not a universal binary will run up to par under Rosetta..

For example Microsoft Office 2004

I cant even tell the difference between my DTK and my dual 2.3 G5 for Excel,Word and Entourage..

Apps that depend on the Altivec engine like encoding video and some audio will run sluggishly under Rosetta..
Apps that depend on Altivec and are universal binaries or native ( built for Intel only ) will run just as fast if not faster than the G5 iMac and in some cases darn near close to my dual G5.

As soon as I get my iMac I will be doing extensive testing..
I'll report back then..
 
Since Consumers are gamers there should be 1 or 2 Gaming benches. There isnt anything that stresses a system like a game of Doom3 or what have you. Gaming benches are very important, stuff like opening apps is on the verge of silliness with todays hardware. A suite of 4 or 5 benches that can be put on every new Mac. Apple iam convinced doesnt want comparisons made between models otherwise they would provide a suite of benches. It would be very good for Mac Rumors to have a set of benches so that the User can see what machine may be best for running certain software.
 
Dont Hurt Me said:
It would be very good for Mac Rumors to have a set of benches so that the User can see what machine may be best for running certain software.
Click to expand...

Yes, and to have a table with each machine with each amount of ram and different video cards for easy reference
 
organizing the data shouldn't be a problem.

it's just a matter of coming up with the benchmark material and the actual sequence of events.

arn
 
I'll help in anyway you guys want me to. I'd love to be involved. I have the 17" iMac from BEFORE they put in the iSight. I have iMovie, iTunes, Final Cut Express, iDVD. THe usual stuff. If I can be of any help encoding video or audio, please let me know. i'd love to be envolved.
 
arn said:
organizing the data shouldn't be a problem.

it's just a matter of coming up with the benchmark material and the actual sequence of events.

arn
Click to expand...

Will it be a script to run all the stuff in the file - or will it be a series of tests or files for the user to run (with a .txt file for rules / instructions)

we might as well start a list of things to test

1) Photohop
2)
 
1) Photohop
2) Quicktime Export to some format (h.264?)
3) iTunes AIFF -> AAC
4) iMovie ?
 
BakedBeans said:
Will it be a script to run all the stuff in the file - or will it be a series of tests or files for the user to run (with a .txt file for rules / instructions)

we might as well start a list of things to test

1) Photohop
2)
Click to expand...
2 has to be encoding H.264 (or some sort of video file)
 
- Game Benchmark. Needs to be freely downloadable demo with benchmarking (FPS) capability
- Xcode compile ?
 
Maybe we should include some sort of compression benchmark, like stuffing/unstuffing some standard set of files.
 
AtHomeBoy_2000 said:
Well, someone step up and take control here. Soemone needs to take charge.
Click to expand...


Well, I don't mind - I can source all the stuff needed and do any admin type stuff (like writing the rules etc). We are going to have to test things that take a long time - not opening of apps etc - thats a bit pointless.

I will Draw something up and report back.

EDIT - at the moment we want to stay away from any 'un-optimized' apps like handrake - And i dont know why the new imacs seem to be doing so bad in itunes either - thats strange.
 
Ok

1) Photoshop Test

Largeish Photoshop (.psd) file with an action script - testing multiple and single processor aware actions (Can do that separately or as one test). Would be good to include a link to download the 30 day trial if the user doesn't have CS2 (or the latest version)

2) 720p + 1080p Video Frame Rate Test

Using apples trailers to check frame rate and processor usage.


3) iTunes and Quicktime Encoding

not sure about this

4) iMovie Render - Soft Focus

Again not to sure


I got my ideas from online tests etc and places like barefeats.

i think after a restart would be fairest
 
BakedBeans said:
3) iTunes and Quicktime Encoding

not sure about this
Click to expand...


Maybe one of our digital audio forum members could do a stereo high-quality location recording of an environmental sound i.e. city noise, the beach, factory, railway station etc. (no copyright issues) of maybe 15-45mins of length and convert/save it as an 16bit AIFF or Apple Lossless file for downloading.
 
What a brilliant idea. PM me if you want some help in some way (I'm not a programmer though).

Is it going to be implemented through the MacRumors Guides?
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back