Page MenuHomePhabricator

Make Diviner useful for third-parties
Open, LowPublic

Assigned To
Authored By
epriestley
Mar 5 2014, 8:48 PM
Referenced Files
None
Tokens
"Mountain of Wealth" token, awarded by PXke."Like" token, awarded by 1stone."Mountain of Wealth" token, awarded by pouyana."Like" token, awarded by kristo.mario."Like" token, awarded by JamesRamm."Like" token, awarded by GOPalmer."Mountain of Wealth" token, awarded by joshuaspence."Mountain of Wealth" token, awarded by svemir."Mountain of Wealth" token, awarded by noisy."Mountain of Wealth" token, awarded by dctrwatson.

Description

Currently, Diviner is only really useful for generating Phabricator documentation. Eventually, it would be nice to make it useful for projects in general. Work on T988 has made major strides toward this goal, but significant work remains, especially to support languages other than PHP and Javelin-flavored Javascript.

This is not a priority for us, but if you're interested we'd love to hear about your use case and which languages you'd like to generate documentation for.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Does Diviner work today if I just have a repository containing Diviner documents (like src/docs in rP)? Would I need to setup some sort of post-receive hook in Phabricator to rebuild the docs?

You'll need a .book file, but it should work OK, at least in theory.

And, yeah, no way to auto-rebuild via Harbormaster yet.

How do the docs on this install get built?

When I push this host, the update script has some bin/diviner ... calls after we cycle the daemons and such.

Oh, so you actually need to sync the host to update documentation, you can't just push to a diffusion repository?

I imagine to make this scalable the generated documentation (which would likely be built via Harbourmaster) should probably live in the database?

The generated documentation lives in the database. It could be published from any other machine, but the publishing machine currently needs a working copy on disk, a copy of Phabricator to run bin/diviner, and direct write access to the database to publish the documentation. In Phabricator's case, the production host conveniently has all of these things after it gets deployed, and it gets deployed often enough that the difference between "on deploy" and "on push to remote" is pretty moot.

We could do the same thing with a cron job or a post-receive hook, although those won't scale well across multiple machines. I suspect the general-purpose way forward involves moving some of the publishing operations to Conduit and maybe moving some of the Diviner code into arc so it can be run more easily on Harbormaster/Drydock hosts.

Ah right, I misunderstood.

+1 on moving some of the code to Arcanist.

I might be able to move this forward a bit.

I am also in favour of Doxygen generation, with support for advanced features like mscgen and dot code. Would probably just result in a redirect to doxygen on the machine running phabricator?

I'd be interested in documentation for Lua and Ldoc

Some things that are maybe worthwhile (in addition to the blocked tasks):

  • Allowing Diviner books to have projects declared explicitly (by implementing PhabricatorProjectInterface and specifying projects in the *.book file)
  • Linking Diviner atoms/symbols to repositories, in addition to books.
  • Add an "Edit Book" interface for modifying projects and policies for Diviner books.
  • Add policy support to the Diviner application (see D13091).
  • Adding links to Diviner documentation from Project pages somewhere.
  • Fix the "static" publisher (I don't believe this works currently).
  • Maybe formalize the relationship between Diviner and Symbols.
  • Clarify DivinerAtom and DivinerAtomRef... I find the distinction here a little unclear.
  • Formalize Remarkup rules for linking to Diviner articles. I'm not sure how the @{article:This is an article} stuff works at the moment... is this linking strong enough?

Some things that we could likely build, but I'm unclear on how much value they provide:

  • Allow subscriptions to Diviner books.
  • Allow Herald rules to act on changes to Diviner documentation.
  • Give books a monogram.
  • Allow mentions from Diviner documentation (for example, mentioning a Maniphest task in documentation).
  • Allow Diviner books to be mentioned.
  • Implementing transactions. I think that this would be messy given that (currently) no user would really ever interact with Diviner, it is always the omnipotent user.

(There's also a uri.source in Books which should probably get an implicit value if the repository is set but uri.source is not.)

Implementing transactions. I think that this would be messy given that (currently) no user would really ever interact with Diviner, it is always the omnipotent user.

I think we should pursue this at some point, and anticipate a future time when Diviner updates happen over Conduit. This seems like it's probably a prerequisite if Harbormaster is triggering documentation rebuilds -- we can't give build hosts direct access to the database.

We'd potentially get some stuff like comments-on-documentation more cheaply after this, although I'm not really sure if that's a good idea. The comments on PHP.net feel pretty bad to me about 90% of the time.

Probably a prerequisite for implementing anything like arc diviner, or whatever, too. That said, it's probably also quite a lot of work.

Hi, I'm new to this but mostly we've been enjoying phabricator's features a lot. It seems like there's limited support for integrating with doxygen, as others have mentioned. This would really be nice, as I'd like to have documentation generated and available with access permissions correctly set through phabricator's account management. It would be great to have!

Is the vision for Diviner that it will become as full featured as something like sphinx (the place to write Serious Docs for all my software projects) or just moderately useful for things that are not Phabricator?

I'd have to defer to the Phacility guys on that, but I would like to see Diviner become a full-fledged tool which can be used separately to Phabricator itself. I don't know enough about sphinx to compare the two products.

The vision is for it to be full-featured (e.g., support a variety of languages and input and output formats -- but also support integrations, policies, cheap setup via Harbormaster, easy document preview via arc, etc), but there isn't much demand so progress is likely to be slow. I doubt we'll ever have a superset of Sphinx's features (e.g., I don't know what "ePub" format is and would be slightly surprised to peer into the future and learn that we support it), but the architecture is (mostly) generic.

There are four ideas I'm aware of that we don't have good/obvious architectural room for growth on right now: atom types, patched/runtime-modifiable symbols, internationalization, and versions.

On atom types, we hard-code the existence of types like "article", "class", "function", and "method". If you want to document a programming language where these concepts are totally foreign, it might be difficult. Since most languages use these concepts, I think we're on firm ground here, but if you invent a cool moon-language that uses "feelings" and "koans" instead, Diviner won't understand them and we probably won't accept them into the upstream.

On patched/modifiable types, we make an effort to handle duplicate definitions (function X is defined in Y.c:23 and Z.c:93) but don't have any builtin support for understanding that there's an extension relationship there (e.g., that Z.py is monkey-patching the class definition in Y.py). I'm not sure how many issues this would create in practice, or how important it is, or how difficult it would be to support, but we're at least broadly assuming the structure of an atom is exclusively a function of the definition of that atom in the file where we find it defined plus some well-defined set of "extends" or "inherits" relationships, not a combination of that stuff plus several monkey patches in other arbitrary files.

On internationalization, I think this would likely need to be a layer on top anyway (for source documentation, it doesn't make sense to write comments in several languages), but we currently have no provisions for it. Since I think it would happen at the very end of the pipeline I imagine we could add it later without much disruption. I expect demand to be very low. PHP is the only programming language I'm aware of that has documentation available in multiple human languages. However, because Diviner also blurs the line between pure source documentation and "articles", I expect we'll see some reasonable use cases eventually.

We don't have any core support for different versions of the documentation right now. This is probably the most important of these items. The v0 hack would be to just build different books for different versions, but at some point we probably need to put more structure on this so you can, e.g., jump between version 1.3 and 1.2 of the docs and see what changed in f(). I don't think this is difficult, but we don't have code there or a specific plan for it yet.

some well-defined set of "extends" or "inherits" relationships

I guess monkey-patching is probably "well-defined", too, but I mean that we assume that we can look at "X.c" and that file either contains or links to everything we need in order to understand the structure of class "X".

The case we don't currently handle is where you have to look at the entire codebase to find any "Y.py" which might say "after examining X.py and learning about the structure of class X, they are lies, apply these additional changes to it".

We could probably accommodate this by saying "monkey patching is like extends" and creating a new "X, Patched by Y.py" atom. So you'd look at "X" and see the entire "natural" definition of the atom, but also see that something in the codebase creates a patched version of X. This is potentially confusing because all X in the codebase might really be Patched-X. Philsophically, my stance here is generally: too bad, monkey patching is evil and this is the price you pay.

To add to the comment of @cburroughs above, it would be useful to cater to existing workflows.

In Plone the solved that at some point by adding FunnelWeb that allows any html that conforms to a templated format to be imported into Plone. It means that if you have a workflow that includes sphinx there is no need to change it, but instead set up a pipeline to import the sphinx html output to Plone whenever there is a change.

It would be great to have a the possibility to set up a pipeline from Sphinx builds, or the html of other documentation systems to Diviner, and a quick route to make it useful and easier to bring into the fold of existing workflows.