do not step out
14/Jun/2005 | 01:51

Folders Never Die, They Just Go to /dev/null and Get Indexed

I usually write my essays in a program called MacJournal before I post them to the MovableType-based waste processing plant known as /damage. Today, I'm writing this one in BBEdit, a straight-up text editor. Why? Because I authored the site in it, I'm just writing words, and GOLLY GEE GOSH ALL I REALLY NEED IS A TEXT FILE.


Enter SPOTLIGHT. But MacJournal lets me organize my writing, notes, and site entries like a composition book. METADATA. I know what each entry is about already, and I can point to which one I want to reference by just looking at the title. LIVE SEARCHES. How often do I really need to see my article on other stupid names for Bonjour along with other stuff?

See, both of the voices in my head are wrong. I need to do two things: organize and locate. Search systems like Spotlight all rely on various pieces of metadata, but how much metadata can be written by a computer that can't think like I do?

Let me cut to the chase and tell you where I'm going with this.

People are talking about metadata searches replacing folders. Some think it's really going to happen; some absolutely love their deeply nested folder structures. Me? I need organization, and file paths accomplish that. But I also need flexibility, something file paths can't pull off. There's got to be a solution here.

What Can Be Said About the Human Brain? (Or: The Big Dilemma)

I'm lazy. We're all lazy, to some extent, and metadata is WORK. That's right. Metadata means figuring out how you want to classify or tag things, and that requires thinking, and thinking is HARD WORK. If I stick to plain text files inside well-organized nests of folders, most of the metadata work is done for me by slick kernel extensions, but OH SHIT, that means I've gone through the task of classifying everything anyway, and WE'RE RIGHT BACK TO WHERE WE WERE BEFORE.

So folders are here to stay. My resume is in ~/Documents/Employment/Resume and those invoices for last month are in ~/Desktop/work/Accounting/Invoices/Incomplete . . . We don't need no stinking metadata here. I've got a deep and complex folder hierarchy, and the world is a hierarchical place. Leaves on trees; atmospheric layers; shoulder, arm, hand, fingers. But those invoices also pertain to that internal finances audit we're doing, so maybe they should go in ~/Documents/Company/Audit as well. So much for hierarchy when the human brain can't classify anything to save its life.

Luckily for us, we've been working with metadata the whole time: folders and directory hierarchies are just pieces of metadata that describe where a file exists on the disk. Perhaps we can exploit this. If the location of a file is a piece of metadata, and a file can have an arbitrary amount of metadata associated with it, why can't it then exist in more than one place? Our invoice file could exist in ~/Desktop/work/Accounting/Invoices/Incomplete as well as ~/Documents/Company/Audit at the same time. Is this too difficult to process, though, as human beings? Leaves on trees.

So our problem isn't that we as humans have to classify things, and it's not that data shouldn't be so rigid as to be classfied in the first place. Our problem is just how much classifcation we can do before the extra work expended negates any of the benefit of live searches and flexible storage and retrieval schemes. I know I don't want to have to get info on a file and add a bunch of stuff for it to be locatable or otherwise useful. Work is hard, remember? I take computer out of box, I plug in computer, I turn computer on, it does work for me. Not the other way around.

Where are we left? Folders are too rigid, though we could theoretically apply extra "locations" to files, and adding metadata is out of the question because, again, I'M LAZY.

We're left with no real solutions. Metadata will not replace folders, and the folder needs to evolve past its current incarnation. Anyone who thinks one will usurp the other doesn't understand the benefits and drawbacks of both methods. I'm reminded of a quote by Albert Einstein at times like this:

The significant problems we face can not be solved at the same level of thinking we were at when we created them.

Back to the drawing board.


Unless we just change what a 'folder' means.

Make all folders be 'smart folders'. Make them smart folders that can also ADD metadata to any file dropped into them.

Drop a file into Incomplete, and it gets tagged with:

- Work
- Accounting
- Invoice
- Incomplete

The Incomplete folder itself is a folder that only matches if ALL these are present.

(This is not really a change from the current concept of a folder. The difference is how you store, index and retrieve the metadata.)

To avoid long file listings in, say, the Work folder--which matches any file tagged with "Work"--if a subfolder exists, anything matching that subfolder is NOT listed in the current folder listing.

Folders inherit matching requirements from their parents.

Folders that match multiple tags using OR instead of AND would be read-only.

Disclamer: The above is a quick, late-night (or early morning) hack that might have serious problems my brain isn't aware of at the moment.

Also, I expect something like this is what WinFS is all about, and I've wasted a bunch of effort 'inventing' this...

...good thing this isn't Slashdot.

posted by Alex on June 14, 2005 at 05:35


I should know better than to use dashes on teh interweb.

posted by Alex on June 14, 2005 at 05:40

You folks ever heard of aliases/symlinks (same concept, different implementation)?

posted by Michael Grant on June 14, 2005 at 11:20

Remember all the hubub about piles?
That's one solution.

I like having a heirarchical folder structure, but I do want to be able to organize based on metadata, or just whatever strikes my fancy.

I think piles would allow me to do this. At the moment I just use folders of aliases, but piles seemed much more elegant. Often spotlight/smart folders can put things together, but sometimes the metadata just isn't there.

posted by Jon on June 14, 2005 at 12:05

Aliases/shortcuts/symlinks aren't the same. They're just extra files that are pointers to other files. We're talking about having a file physically exist in more than one place.

But UI tricks like aliases, piles, whatever, they don't solve the fundamental problem:

It's extra work, and terribly abstract work, at that. Metadata matches are inaccurate, as well, so just tagging something (WORK SUCKS, MAY AS WELL MAKE ALIASES) for use in a smart folder is going to occasionally (or perhaps frequently, depending on how well-tagged your items are) return things you don't want. No, metadata isn't absolute enough, and folders are TOO absolute.

WinFS isn't a great solution, either, I'm not convinced. For all intents and purposes, it's a giant database sitting underneath the OS. Databases are great for some things, but they're overkill for day-to-day tasks. We need something more computer-language-precise AND semantic at the same time.

Alex: I cleaned up your post for you. I don't allow HTML in the comments, so it didn't break anything . . . Just produced some weirdness. Corrected.

posted by mikey-san on June 14, 2005 at 13:15

Right, I agree that adding metadata to files is a pain in the ass.

The point of the folders I described was to set the metadata up once, and then have it auto-apply to anything you drop in that folder. (Folders are really just metadata anyway, but we add it with a drag and drop that feels to us physical orginization, which for some reason is less tedious than logical organization)

You could probably do exactly what I was describing by using Folder Actions and Automator combined with Spotlight... it'd just be easier to have it built into the get info window. "Apply these spotlight keywords to any files in this folder"

Folders as file 'classes' complete with inheritance, was what I was going for.

You'd still need to set up the metadata once, but you already had to name the folder anyway.. if it was finder-integrated, it would be pretty painless. and you're much more likely to add the metadata when you're thinking about HOW to organize your data than when you're WORKING on your data.

...I really need to get my own blog for this sort of stuff.

Thanks for the cleanup, BTW.

posted by alex on June 17, 2005 at 05:33