Apple Patent for Resolution Independent User Interface

[shawnce]

So after reviewing the patent application it looks like the primary focus of the patent is about how to generate and record "recipes" for things like buttons (with reflective lighting effects, complex color gradients, etc.) so that a rendering system can generate a bitmap appropriate for display given the current UI scaling factor.

Example image from patent application

SVG like but more tuned for the specific purpose of UI widgets instead of a generic vector based system.

[0006] In one embodiment, the invention provides a method to represent a graphical user interface object's material map in a procedural and, therefore, resolution independent manner. The method includes receiving values for each of a plurality of attributes associated with a material map object, associating a value for each of the plurality of attributes, and storing the plurality of attributes and their associated values in a file. The file may be a "flat" file or a hierarchically-ordered file. The collection of attribute-value pairs comprise a complete description of the graphical user interface object's material map and may be used by a rendering module to create a visual representation of the material map at any number of resolutions. In addition, because material maps in accordance with the invention are represented procedurally, they may be encrypted to prevent unauthorized inspection or use.

I hope Apple rolls this tool out to developers... I know I would love to use it for building recipes for some custom UI widgets that I need to implement with support for resolution independence (not trivial to generate some of the nicer looking lighting and shading effects without falling back on bitmap textures, etc.).

 

[shawnce]

They can't patent this!

Vectorized interfaces have been around a while, and Windows 95 had the same exact implementation of UI scaling that they're talking about now (without the bitmapping). WTF Apple?

Yes, it even had it with the slider that you'd move to scale the DPI of the GUI up. IIRC, I remembered that and suggested it as a simple way to make aqua scalable (yay!)

That isn't what the patent application is about...

 

[macintel4me]

I get the resolution independence thing for most things. I don't know how icons would work in this world, however, with "procedural" descriptions of images. Maybe instead of one icon, five or six must be included with the app and the OS up-samples or down-samples depending on the resolution and icon size. But then again, that doesn't sound very "procedural" to me.

 

[shawnce]

I get the resolution independence thing for most things. I don't know how icons would work in this world, however, with "procedural" descriptions of images. Maybe instead of one icon, five or six must be included with the app and the OS up-samples or down-samples depending on the resolution and icon size. But then again, that doesn't sound very "procedural" to me.

Use of PDF for simple icons and multi-resolution icon resource files for complex ones should cover this aspect of things well... of course SVG is another option (in theory).

 

[Peel]

I think monitors already report this. At least I remember reading about something about the DDC channel (used for plug-n-play) reporting the screen's physical dimensions.

Great. I like it when you find out that your older hardware is compatible with new software. Thankfully whatever committee came up with the Plug-N-Play protocol, added some data on this so that it can be leveraged in the new UI.

 

[bretm]

LOL, funny to see how few people know what resolution independence does. LCD-Display all have a certain number of dpi (dots per inch). Usually it's 72, 75 or 100 for laptop displays. When you look closely at your screen, you can see those little squares, each of those squares has 3 rectangles (red, green and blue, mixed together you get a square in every color you want). As it is now, cranking up the resolution makes everything smaller, to a point where it's hard to read, click buttons etc. And changing the resolution of a LCD-Display to something different than its native resolution (the one that corresponds to the number of pixels the display actually has), looks terrible.

Resolution independence makes it possible to scale things without changing the resolution. If you have to squint sometimes, make it bigger, if you wish to have more space for windows, make it all smaller. Resolution independence lets you zoom in without getting blurry and loosing details. Try that zoom function (CTRL+scroll up) and you'll see how bad it looks. With resolution independence, it would still be super sharp (until a certain degree I guess). You can't convert everything into vectors or have pictures with infinite resolution, but it should be enough to double or triple the resolution on LCD-Displays. You can't really buy any of those displays yet because you wouldn't be able to read anything without a magnifying glass right now, but who knows, Apple might present new high res displays after Leopard is out giving you super sharp pictures.

Your point is moot because 99% of stuff worth looking at is based on pixels on the internet, NOT vectors. And those 15 years of web pages aren't going to change to vectors any time soon. Gifs, photos, quicktimes, windows media, real players, etc. are all a static pixel size. We're talking logos, product artwork, etc. It all looks good displayed pixel for pixel. Double the res and scale up the interface and the interface will look good, but those pictures will have to be 1/2 the size or be scaled up, forcing them to be interpolated, forcing the to look soft at the same physical size they used to.

Think regular TV on an HDTV television. You've got more than twice the res, but guess what? Regular TV looks worse on an HDTV than it does on a regular TV.

 

[MikeTheC]

I haven't dabbled with it as my x86 PC died a while ago and so I haven't had a box to dedicate to running Linux, but from what I understand, I think they're rigging it so that, for instance, icons can be authored in SVG format. One can only speculate whether other UI elements and objects would be handled similarly. If so, then you take an object and, as others here have said, do an on-the-fly dynamic rasterization of it, just like what a PostScript printer does, and voila! you have an object of a consistent size and clarity (meaning it doesn't shrink nor grow as the resolution changes, nor does it become pixelated and fuzzy as the resolution drops.

As far as web and other UI-design is concerned, the only thing I can think of which could be a concern to designers are static, fixed-size images (like, for instance, photos). Everything else on the site should work without issue.

My big concern is that you're going to have a situation where such a theoretical web site is going to look totally awesome on a Mac, but like complete ***t in Windows because the web site's default appearance (and how a Win32 environment web browser app displays content) just aren't as high a quality as what an Aqua (or whatever)-environment-based app would be.

Only time will tell... (I sure hope that Apple either adopts an open standard for this or promulgates one so that we don't have lots of incompatible, identically different rendering technologies out there making a complete butcher's job out of surfing the web and so forth.)

 

[povman]

Only time will tell... (I sure hope that Apple either adopts an open standard for this or promulgates one so that we don't have lots of incompatible, identically different rendering technologies out there making a complete butcher's job out of surfing the web and so forth.)

Actually, a website done properly using CSS should be fully scalable. By 'properly' I mean, the CSS says what the developer actually wanted, eg 50% of page, 1cm etc. So there's a standard already for websites. As for images, people would have to either start moving to SVG or they better specify real sizes for their images in the source. (unless image formats already have support for real sizes?)

 

[Highland]

Web scaling and UI scaling are quite different beasts (with some similarities). I'm pretty sure this patent would be regarding UI, not web.

Here's a blog post I made specifically about resolution independence and icon design: http://islayer.com/blog/?p=87

There's quite a few links at the top to other related articles.

As for websites and res ind... please have a read of what's coming for WebKit and Dave Hyatt's opinion on where it's all heading.
http://webkit.org/blog/?p=55

The simple answer seems to be that there's no one solution that covers every situation well. Vectors are brilliant for some things, but fail miserably at others. It's pretty easy to include several sizes of a bitmap in one file... OS X's file/application icons already do this (as do OS 9, Windows and most other modern OSs).

povman: using CSS "px" isn't bad... that will just be scaled up. Hyatt suggests that one CSS px may not always equal one screen px. So they can be used and scale perfectly using new techniques (ie. not the current font-sizing seen in most browsers today).

 

[benbow]

scalable icons

I have not done any graphics programming for over twenty years so most of this thread's scaling and rendering discussion is beyond me.

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps? More likely they use lines and polygons, and fill patterns such as a drawing program might use. The OS then pixilizes or rasterizes the image to the scale requested. Vectors must still exist at some level in the scaling software (deep down) to describe the orientation of the primative line elements, polygons, and their fill patterns / colors.

I also remember writing programs to display scientific data plots using a 1970s monochrome display that could display only dots or vectors, and if one worked at it axes and numbers. Royal PITA.

 

[Highland]

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps? More likely they use lines and polygons, and fill patterns such as a drawing program might use. The OS then pixilizes or rasterizes the image to the scale requested. Vectors must still exist at some level in the scaling software (deep down) to describe the orientation of the primative line elements, polygons, and their fill patterns / colors.

The OS X icons are bitmaps. Each icon (.icns) file can have several sizes, ranging from 16x16 pixels up to 128x128 pixels. Leopard introduces 512x512 pixel icons. These are essential, especially at the smaller (16x16 and 32x32) sizes. Icons need A LOT of tweaking to look good when that tiny.

Here's a screen shot of Icon Factory's Icon Builder with an icon at various sizes: http://iconfactory.com/graphics/software/iconbuilder/screen1.jpg

You can see the resources that are being built on the left.

 

[JFreak]

Was actually just thinking of this today, because my 15 inch screen is soo small. What would happen when I upgrade to the next generation of 50" screens?

You would go crazy, because you have now used to use less than a megapixel

No, seriously, I have a 100" screen (HDTV projector) and I surely can appreciate the size because my eyes have been lasered. For most tasks, the 1280x720 screen size is good enough, but I would surely appreciate higher pixel counts when I do a Protools mix. Currently I get few dozen visible tracks, but it wouldn't hurt to see more. I like the direction things are going...

Everything new just costs a lot of money

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence?

Yes, a rough example. That's how it goes if you scale a bitmap that is "reasonably big" to begin with. But if the UI elements are vectorized, the end result is even smoother. You know, a vector is kind of an instruction set "what happens when you go from A to B" so assuming all things are vectorized, things look smooth from screen to photo print and a really big advertisement...

The problem is, not all things will be vectorized at once. I bet when Apple releases Leopard, the most common things should be vectorized, but there would be a lot of bitmaps still being used with 3rd party applications and less common items. It's a good start, but don't expect everything to be perfect to begin with

My big concern is that you're going to have a situation where such a theoretical web site is going to look totally awesome on a Mac, but like complete ***t in Windows because the web site's default appearance (and how a Win32 environment web browser app displays content) just aren't as high a quality as what an Aqua (or whatever)-environment-based app would be.

That's always going to be the situation, unless the CSS or other standard begins to specify the display settings (gamma and such). To begin with, Mac and PC gamma is so different that web sites designed with Mac look different when viewed at PC and vice versa. Granted, it's not "complete #%*t", but it can be a disaster.

I want to see web sites including display calibration information!

Think regular TV on an HDTV television. You've got more than twice the res, but guess what? Regular TV looks worse on an HDTV than it does on a regular TV.

It's all a question about "upsampling". There are *HUGE* differences in upsampling algorithms, so some SDTV->HDTV conversions look like crap while others look smooth. The same goes to content scaling in the web pages; it needs to be done, and to be done with good quality.

The analogy is good, but it lacks in the respect of cost management; if a brand "A" television costs a hundred and a brand "B" television costs a thousand, wouldn't you thing the brand "B" would have allocated some time to think about the upsampling issue? I bet they would have, thus justifying the price difference. Brand "A" could have just assumed things are soon going to be in HDTV anyway... But the same thing does not apply when talking about the internet. Content is being made in some resolution and likely viewed in another, so there has to be a unified way to scale things. This is missing, and it has to be done soon. Somebody (Apple) has to lead and after some time (few years) general public can enjoy the results.

Hopefully

Any designer worth his/her salt will have to learn to work at the resolution of the device. It's not terribly difficult: print designers have been doing this for decades. Unfortunately, web browsers are only beginning to allow the necessary controls.

There have been print designers for +100 years, but the graphical web itself is only a teenager. There must be a difference! Sure, prints have become better in a hundred years, but so will web pages! We just have to give web designers a time to mature.

Today's web site is like a newspaper in 1900, if the analogy stands.

 

[R303blue]

Ok, great higher res sites, will bandwidth be ready?

As a web designer myself at a big company where I create tons of graphics daily and need to be fully optimized, don't you think this is going to cause nightmares with file size and optimization? Will there still be a *standard* res to work in that will be a good start across the board? I used to do a lot of print work, and my biggest limitation IMO was the size of files and having a computer powerful enough to work in those sizes. With the web I can practically be on a G3 these days and still have room to spare, but with higher res everything, not only will layered photoshop files reach massive sizes, computers will be crunching a lot more rendering new equipment obsolete quicker, and storage space needed will go up exponentially in time. I think this causes designers to just end up spending a lot more money (as if we don't have to already with soring costs of the products we use) in the long run. I better see a hefty raise from all this ;-) Maybe I'm just not getting it, and usually I embrace great new technology. I guess we'll see what happens.

 

[Highland]

Yes, a rough example. That's how it goes if you scale a bitmap that is "reasonably big" to begin with. But if the UI elements are vectorized, the end result is even smoother. You know, a vector is kind of an instruction set "what happens when you go from A to B" so assuming all things are vectorized, things look smooth from screen to photo print and a really big advertisement...

The problem is, not all things will be vectorized at once. I bet when Apple releases Leopard, the most common things should be vectorized, but there would be a lot of bitmaps still being used with 3rd party applications and less common items. It's a good start, but don't expect everything to be perfect to begin with

I haven't found that to be true. If you have a large bitmap, and scale it down to whatever size is required, it looks "about as good" as a vector that's been created at size.

Not everything can be done with vectors as complex textures etc are very difficult. Complex vectors (like what would be required to do OS X style UI elements and icons) are usually much slower to render (you could cache as bitmaps, but what's the point?).

Some things DO work as vectors though, but certainly not everything. Don't expect to see Apple change everything over.

As a web designer myself at a big company where I create tons of graphics daily and need to be fully optimized, don't you think this is going to cause nightmares with file size and optimization? Will there still be a *standard* res to work in that will be a good start across the board?

If what's publicly available is anything to go by, then there will still always be a 1:1 size, but the user can scale the interface up or down.

Like I said, web zooming isn't really the same as UI res independence. Similar, but not the same. Also, we're not talking about print ready sizes... only about 3x what we use right now. Ie. 200ish DPI, not 600 DPI!

 

[shamino]

Your point is moot because 99% of stuff worth looking at is based on pixels on the internet, NOT vectors.

You don't need to vectorize everything to take advantage of resolution independence (although that also works.)

Bitmap-based interfaces can also be resolution independent if the bitmaps contain images rendered at a high resolution, and contain DPI information so the system will know how much to scale it for different displays.

And those 15 years of web pages aren't going to change to vectors any time soon. Gifs, photos, quicktimes, windows media, real players, etc. are all a static pixel size.

And all of these have DPI parameters as a part of their info. If the content creators set the DPI values correctly, very little will need to be changed. Cameras, scanners, rendering tools, and Photoshop all support this. Images designed for today's web browsers should already be set for 72 dpi (although there is a lot of badly produced stuff already in circulation, of course.)

We're talking logos, product artwork, etc. It all looks good displayed pixel for pixel. Double the res and scale up the interface and the interface will look good, but those pictures will have to be 1/2 the size or be scaled up, forcing them to be interpolated, forcing the to look soft at the same physical size they used to.

Assuming that the images are displayed on a 1:1 scale factor. The whole point of putting DPI in your image files is that this isn't necessarily going to be the case. If I have a 1" square image, and I make it 72dpi (72x72), of course, it will get blocky when rendered at 150dpi. But I can also make my image at 300dpi (300x300), and it will look good on any display.

There's no need to replace images with vectorized content to get good results on a resolution independent UI.

 

[shamino]

As for images, people would have to either start moving to SVG or they better specify real sizes for their images in the source. (unless image formats already have support for real sizes?)

They do. Load your favorite image into Preview and do a "get info" on it. You will find an "Image DPI" setting. Most good editors allow you to view/modify this. (I have personally used this on Graphic Converter and Photoshop Elements.)

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps?

OS X's icons are a collection of bitmaps. Theoretically, the file may contain hundreds at all kinds of resolutions. In actual practice, most apps use one one - (typically 128x128 for OS X apps, or the legacy 32x32 size for Classic apps) and the Dock scales it up or down as necessary.

The OS X icons are bitmaps. Each icon (.icns) file can have several sizes, ranging from 16x16 pixels up to 128x128 pixels. Leopard introduces 512x512 pixel icons. These are essential, especially at the smaller (16x16 and 32x32) sizes. Icons need A LOT of tweaking to look good when that tiny.

The icon file format supports multiple sizes, but the Dock doesn't use them. I ran tests on this by creating an icon with some distinctive differences in all the different sizes. When applied to something in the Dock, they all get ignored. The Dock uses the largest size image in the icon and scales it to produce all the other sizes, ignoring the smaller images in the file.

Is this a bug? Maybe, but it looks fine anyway.

It's all a question about "upsampling". There are *HUGE* differences in upsampling algorithms, so some SDTV->HDTV conversions look like crap while others look smooth. ...
The analogy is good, but it lacks in the respect of cost management; if a brand "A" television costs a hundred and a brand "B" television costs a thousand, wouldn't you thing the brand "B" would have allocated some time to think about the upsampling issue? I bet they would have, thus justifying the price difference.

You'd like to think so, but experience has shown that inexpensive devices often look just as good as (or sometimes better than) more expensive models from other manufacturers.

Not always, but often enough that you should never assume high quality simply because of a high price tag. But this is really off-topic here...

 

[dusanv]

They can't patent this!

Vectorized interfaces have been around a while, and Windows 95 had the same exact implementation of UI scaling that they're talking about now (without the bitmapping). WTF Apple?

Yes, it even had it with the slider that you'd move to scale the DPI of the GUI up. IIRC, I remembered that and suggested it as a simple way to make aqua scalable (yay!)

Still, they can't patent that! Prior art omg.

I agree, it's ridiculous. Gee, who would have thought? Vectorized graphics!

This is one of the things that totally turns me off when it comes to Apple. They frequently patent obvious and prior art stuff and what's worse, pay for totally frivolous patents (Amazon's 1-Click, the Creative menu patent, ...).

 

[Highland]

The OS X icons are bitmaps. Each icon (.icns) file can have several sizes, ranging from 16x16 pixels up to 128x128 pixels. Leopard introduces 512x512 pixel icons. These are essential, especially at the smaller (16x16 and 32x32) sizes. Icons need A LOT of tweaking to look good when that tiny.

The icon file format supports multiple sizes, but the Dock doesn't use them. I ran tests on this by creating an icon with some distinctive differences in all the different sizes. When applied to something in the Dock, they all get ignored. The Dock uses the largest size image in the icon and scales it to produce all the other sizes, ignoring the smaller images in the file.

You're right. The dock is a special case though... it needs to be able to scale smoothly from large to small. I assume that's why Apple chose to only use the 128x128 resource.

In actual practice, most apps use one one - (typically 128x128 for OS X apps, or the legacy 32x32 size for Classic apps) and the Dock scales it up or down as necessary.

While I completely agree with most of what you've said, I feel I have to correct this point A lot of icons have 16x16 and 32x32 resources, and those are used everywhere in the OS. The Finder and open/save dialogue boxes are common places for them.

I agree, it's ridiculous. Gee, who would have thought? Vectorized graphics!

This is one of the things that totally turns me off when it comes to Apple. They frequently patent obvious and prior art stuff and what's worse, pay for totally frivolous patents (Amazon's 1-Click, the Creative menu patent, ...).

I think that's just the industry in general. If Apple didn't patent this, someone else would and Apple would have to license it. Amazon's 1 click and Creative's menu aren't Apple's fault!

It sucks and is completely out of control.

A patent on browsing music organised by Artist/Album/Song? F$#K OFF!

 

[Manuel Moreno]

seems that apple is going to introduce if not a new, a more polish interface. the screenshots associated to the patent reveal a graphical interface maker with a lot of parameters for each control, including the photoshop layers overlay modes for any material...

 

[dusanv]

I think that's just the industry in general. If Apple didn't patent this, someone else would and Apple would have to license it. Amazon's 1 click and Creative's menu aren't Apple's fault!

It sucks and is completely out of control.

A patent on browsing music organised by Artist/Album/Song? F$#K OFF!

You're right, the whole thing is out of hand. Everyone is patenting obvious stuff. Hopefully it's just a defensive patent on Apple's part.

 

[gnasher729]

But whatever the dimensions, I hope there will always be a way to override the settings. I happen to like high resolutions (I use a 20" monitor at 1600x1200 at work - 100dpi. Most people I know prefer lower DPIs on their screens.)

To be more precise, I prefer text and other things displayed a bit larger, and at the moment I can only get that through lower DPIs.

Then of course there are people displaying on a TV, and there you definitely want things displayed much much larger because you watch it from three or four meters distance! I just checked, just for fun, my TV displays about ten lines of text in exactly the same height as my MacBook screen.

So what the software has to take into account is several factors: Screen size in hardware pixels, screen size in centimeters, text size preference of the user, viewing distance.

 

[gnasher729]

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps? More likely they use lines and polygons, and fill patterns such as a drawing program might use.

You lost.

Every icon is turned into a texture, and then just drawn into a square of arbitrary size. Since this is done by your 3D graphics card, the OS could just as easily rotate the icons by any angle, transform them, and so on, at practically no extra CPU time and GPU time at all.

 

[OdduWon]

You lost.

Every icon is turned into a texture, and then just drawn into a square of arbitrary size. Since this is done by your 3D graphics card, the OS could just as easily rotate the icons by any angle, transform them, and so on, at practically no extra CPU time and GPU time at all.

Yeah, a Icon in OsX is just a square you can paste an image into. or it think there is a way to make paths in PS and use the icon application in the Developer kit to create transparentness in the remaining square. But i haven't got it to work yet

 

[ajprice]

I understand that resolution independence can make the interface graphics bigger, so it would emulate, say, 1024x768 on a 1280x960 monitor and have smooth graphics and fonts. Would it work the other way around, making the interface graphics smaller and emulating a bigger resolution than the monitor (ie get the interface the scale of 1280x960, but on a 1024x768 monitor??

 

[AppliedVisual]

I understand that resolution independence can make the interface graphics bigger, so it would emulate, say, 1024x768 on a 1280x960 monitor and have smooth graphics and fonts. Would it work the other way around, making the interface graphics smaller and emulating a bigger resolution than the monitor (ie get the interface the scale of 1280x960, but on a 1024x768 monitor??

It can work that way, but not well. The problem with trying scaling down is that you only have so much physical resolution to work with and you begin to compromise visual quality.

Where resolution independence will really shine is with upcoming displays that will have much higher pixel densities. True resolution independence will allow for individual applications and display elements to be scaled independently. And we'll be able to view web pages, documents, video windows, etc.. based on factors of intended resolution and specified size rather than the current less defined approach.

In addition to my animation and video work, I also install display systems (kiosks, AV installs in places like sports bars, public venues, government buildings, etc..). I often run into situations where certain displays need to be configured for the elderly or persons with impaired vision. True resolution independence will make it easier to deliver larger displays such as the 30" or even HDTV displays such as the newer 46" 1080p panels and have them show lower resolutions with better scaling. I have found the Dell and Apple 30" displays to already be invaluable in this regard. A 30" display locked at 1280x800 gives perfect 4x enlargement and works great for elderly computer users. Now if we could take that same principal to the next level and deliver 1280x800 on a 30" display, but actually make use of all the pixels instead of just turning 4 pixels into one, with proper smooth scaling of text and other items. That would be great and make it even easier for the audience to read what's on the screen.