12:20 pm, 2 Feb 08
a better use for sha1 URLs
[Here's an old thought I never posted; dumping it here as I clean up my home directory. I never posted it before because I thought I ought to look at existing work in this space before I start feeling too creative -- this seems obvious enough to exist already. Whatever.]
If I use an image, I (in some circumstances, at least) want the display to break if the image changes. If I source an image off of someone else's site and use ("steal") it for my MySpace background, I'd rather see a broken image link than goatse when they notice where their bandwidth is going.
Images don't use the fragment part of their URLs. Let's say from now on I reference images with URLs containing like this:
And then my browser could regard image data that don't have that SHA-1 as a broken image.
Additionally, my browser could implement a cross-site cache of image data. If there are two references to different URLs with the same SHA-1, it only needs to be loaded once. A browser with this functionality could even work with images like
which means "pull from cache if you have it, otherwise I give up". In some sense, then, the fragment becomes the real identifier of the image, while the rest of the URL is a hint as to one source to get the image data from.
If I use an image, I (in some circumstances, at least) want the display to break if the image changes. If I source an image off of someone else's site and use ("steal") it for my MySpace background, I'd rather see a broken image link than goatse when they notice where their bandwidth is going.
Images don't use the fragment part of their URLs. Let's say from now on I reference images with URLs containing like this:
<img src='http://example.com/foo.png#sha1=012312103120321'>And then my browser could regard image data that don't have that SHA-1 as a broken image.
Additionally, my browser could implement a cross-site cache of image data. If there are two references to different URLs with the same SHA-1, it only needs to be loaded once. A browser with this functionality could even work with images like
<img src='#sha1=012312103120321'>which means "pull from cache if you have it, otherwise I give up". In some sense, then, the fragment becomes the real identifier of the image, while the rest of the URL is a hint as to one source to get the image data from.
It seems to me that your fragment idea is not essential to the concept. Since the assertion and check is client side, there's no reason to make it part of the URL. (I think?) So you could also do it like this:
It's an interesting idea, especially for javascript. That would avoid the attack that occured on perl.com just recently where one of their ad providers lost their domain, and pretty soon the included javascript was redirecting users to a porn domain.
Only problem is, 90% of the time, we want the included file to silently update without requiring a similar update in the including file. For instance, when one fixes a logo, or CSS, or even javascript. But perhaps something like this could be made into a best-practice for cross-site executable code.
Reason #235 i hate working on the web
A quick counter-anecdote: It's become very common practice to include foo.css?12283, where the number is a serial number. This is done to explicitly foil transparent (and broken) web caches between your site and the end user.So, to an extent, we're already stuck doing the annoying/stupid "change all referrers" thing.
Re: Reason #235 i hate working on the web
I do that too, but in the last two projects I've worked on, I've created a system that shields developers from having to know about it. The developer writes and tests with foo.css, but when it goes out to production, it gets magically translated to foo_abcdef12345.css. (View source on any page on upcoming.yahoo.com and you'll see.)But are there still ISPs that use proxies that re-encode images at higher compression to save bandwidth? Or was that a relic of the days of dial-up? Tricks like that would break this system.
Also, I believe there's already a standard URN scheme for identifying a resource by its digest -- something like "urn:sha1:". This is, implicitly, the kind of addressing used by most distributed hash tables.
I could be wrong, but on the ordinary web, a content-oriented URN seems like a minority use case. But some people drafted these extensions to HTTP for that purpose.
The fragment part of the URL is interesting in that no-one really "owns" it as far as being able to define specs against it.
I recall a few months back some discussion on the "WhatWG" mailing list about adding support for embedding part of a video stream by putting the start and end timestamps in the fragment; the general consensus in the end was that the HTML spec didn't have the right to impose rules on how fragments are interpreted on various video formats. The HTML spec defines how fragments relate to HTML documents, and there is a matching spec for XML, but it's a bit strange to have talk of URL fragment values in a specification for a multimedia container format like AVI or Quicktime.