If we expanded Pholio to support other types of assets (sounds, videos, 3d models?) would that be a good fit for the second use case? I'm not sure how sophisticated we can make some of the interactions (inline commenting on 3d models might get tricky) but we could probably get 80% of the way there pretty easily.

Or are you more interested in more like a "Library" of assets -- sort of a fancier version of Files, with better support for categorizing/tagging/organizing things?

I think we planned for internal comments on Phame at some point, we might just need to finish gluing them together.

Both are excellent suggestions, and it can be hard to choose between them. But so far, I'm favoring on "Pholio" to be expanded first, seeing that we use it frequently.

Annotation interactions around sound and video assets can be done by highlighting/marking a timestamp in similar fashion as SoundCloud and YouTube, and these JavaScript libraries would come in handy for the purpose:
http://schillmania.com/projects/soundmanager2/
http://popcornjs.org

When it comes to 3D models, WebGL supports the Wavefront (*.obj) format, and similarly there's a JavaScript library that nicely utilize this API to make annotations an easy feat: http://threejs.org/

If you eventually decide to expand the "Files" section as a "Library", it would be nice to see if the comment system were synchronized/integrated with "Pholio", if that's feasible.

I think it's reasonable to do this in two steps. First being just being able to display/render more file types (video, audio, 3d). Annotation could come sometime after that. I don't have any real ETA though, as Pholio is further down on our current roadmap, but adding even just video and audio files seems reasonable easy.

Reasonable indeed. It more or less matches how I pictured it. :)