Currently, Diviner is only really useful for generating Phabricator documentation. Eventually, it would be nice to make it useful for projects in general. Work on T988 has made major strides toward this goal, but significant work remains, especially to support languages other than PHP and Javelin-flavored Javascript.
This is not a priority for us, but if you're interested we'd love to hear about your use case and which languages you'd like to generate documentation for.
| rP Phabricator | |||
| D13256 Allow books to be archived | |||
| D13064 Namespace Diviner book URLs | |||
| D13303 | rP9921cbc41ac3 Allow atoms to be queried by book | ||
| D13297 | rPa3010148e24d Mark diviner atoms as closed in search index | ||
| D13304 | rP751162d363e6 Set icons for Diviner PHID types | ||
| D13091 | rP6b7d7401ca6d Modernize Diviner | ||
| D13254 | rPaddcef962b93 Publish books with the most-open policy | ||
| D13114 | rP006a877996aa Allow ghost atoms to be rendered | ||
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Open | joshuaspence | T4558 Make Diviner useful for third-parties | ||
| Open | joshuaspence | T5790 Move Diviner code to arcanist | ||
| Resolved | None | T8352 Link Diviner books to repositories | ||
| Resolved | joshuaspence | T7458 Diviner should integrate with main search | ||
| Duplicate | None | T8364 Remarkup isn't rendered | ||
| Open | epriestley | T823 Add image support to Diviner | ||
| Resolved | epriestley | T988 Make Diviner a first-class application | ||
| Restricted Maniphest Task | ||||
| Duplicate | joshuaspence | T6310 Parameters description is not Remarkuped | ||
| Resolved | epriestley | T5792 Allow Diviner documentation to be generated with Harbormaster + Drydock | ||
| Open | None | T7101 Allow creating links to books | ||
| Open | None | T8311 Allow Diviner to import documentation | ||
| Wontfix | epriestley | T8536 Merge symbols and Diviner | ||
| Open | joshuaspence | T8555 Fix static publishing for Diviner | ||
| Open | joshuaspence | T8711 Allow Diviner paths to be specify as includes rather than excludes |
Does Diviner work today if I just have a repository containing Diviner documents (like src/docs in rP)? Would I need to setup some sort of post-receive hook in Phabricator to rebuild the docs?
When I push this host, the update script has some bin/diviner ... calls after we cycle the daemons and such.
Oh, so you actually need to sync the host to update documentation, you can't just push to a diffusion repository?
I imagine to make this scalable the generated documentation (which would likely be built via Harbourmaster) should probably live in the database?
The generated documentation lives in the database. It could be published from any other machine, but the publishing machine currently needs a working copy on disk, a copy of Phabricator to run bin/diviner, and direct write access to the database to publish the documentation. In Phabricator's case, the production host conveniently has all of these things after it gets deployed, and it gets deployed often enough that the difference between "on deploy" and "on push to remote" is pretty moot.
We could do the same thing with a cron job or a post-receive hook, although those won't scale well across multiple machines. I suspect the general-purpose way forward involves moving some of the publishing operations to Conduit and maybe moving some of the Diviner code into arc so it can be run more easily on Harbormaster/Drydock hosts.
I am also in favour of Doxygen generation, with support for advanced features like mscgen and dot code. Would probably just result in a redirect to doxygen on the machine running phabricator?
Some things that are maybe worthwhile (in addition to the blocked tasks):
Some things that we could likely build, but I'm unclear on how much value they provide:
Implementing transactions. I think that this would be messy given that (currently) no user would really ever interact with Diviner, it is always the omnipotent user.
I think we should pursue this at some point, and anticipate a future time when Diviner updates happen over Conduit. This seems like it's probably a prerequisite if Harbormaster is triggering documentation rebuilds -- we can't give build hosts direct access to the database.
We'd potentially get some stuff like comments-on-documentation more cheaply after this, although I'm not really sure if that's a good idea. The comments on PHP.net feel pretty bad to me about 90% of the time.
Probably a prerequisite for implementing anything like arc diviner, or whatever, too. That said, it's probably also quite a lot of work.
Hi, I'm new to this but mostly we've been enjoying phabricator's features a lot. It seems like there's limited support for integrating with doxygen, as others have mentioned. This would really be nice, as I'd like to have documentation generated and available with access permissions correctly set through phabricator's account management. It would be great to have!
Is the vision for Diviner that it will become as full featured as something like sphinx (the place to write Serious Docs for all my software projects) or just moderately useful for things that are not Phabricator?
The vision is for it to be full-featured (e.g., support a variety of languages and input and output formats -- but also support integrations, policies, cheap setup via Harbormaster, easy document preview via arc, etc), but there isn't much demand so progress is likely to be slow. I doubt we'll ever have a superset of Sphinx's features (e.g., I don't know what "ePub" format is and would be slightly surprised to peer into the future and learn that we support it), but the architecture is (mostly) generic.
There are four ideas I'm aware of that we don't have good/obvious architectural room for growth on right now: atom types, patched/runtime-modifiable symbols, internationalization, and versions.
On atom types, we hard-code the existence of types like "article", "class", "function", and "method". If you want to document a programming language where these concepts are totally foreign, it might be difficult. Since most languages use these concepts, I think we're on firm ground here, but if you invent a cool moon-language that uses "feelings" and "koans" instead, Diviner won't understand them and we probably won't accept them into the upstream.
On patched/modifiable types, we make an effort to handle duplicate definitions (function X is defined in Y.c:23 and Z.c:93) but don't have any builtin support for understanding that there's an extension relationship there (e.g., that Z.py is monkey-patching the class definition in Y.py). I'm not sure how many issues this would create in practice, or how important it is, or how difficult it would be to support, but we're at least broadly assuming the structure of an atom is exclusively a function of the definition of that atom in the file where we find it defined plus some well-defined set of "extends" or "inherits" relationships, not a combination of that stuff plus several monkey patches in other arbitrary files.
On internationalization, I think this would likely need to be a layer on top anyway (for source documentation, it doesn't make sense to write comments in several languages), but we currently have no provisions for it. Since I think it would happen at the very end of the pipeline I imagine we could add it later without much disruption. I expect demand to be very low. PHP is the only programming language I'm aware of that has documentation available in multiple human languages. However, because Diviner also blurs the line between pure source documentation and "articles", I expect we'll see some reasonable use cases eventually.
We don't have any core support for different versions of the documentation right now. This is probably the most important of these items. The v0 hack would be to just build different books for different versions, but at some point we probably need to put more structure on this so you can, e.g., jump between version 1.3 and 1.2 of the docs and see what changed in f(). I don't think this is difficult, but we don't have code there or a specific plan for it yet.
some well-defined set of "extends" or "inherits" relationships
I guess monkey-patching is probably "well-defined", too, but I mean that we assume that we can look at "X.c" and that file either contains or links to everything we need in order to understand the structure of class "X".
The case we don't currently handle is where you have to look at the entire codebase to find any "Y.py" which might say "after examining X.py and learning about the structure of class X, they are lies, apply these additional changes to it".
We could probably accommodate this by saying "monkey patching is like extends" and creating a new "X, Patched by Y.py" atom. So you'd look at "X" and see the entire "natural" definition of the atom, but also see that something in the codebase creates a patched version of X. This is potentially confusing because all X in the codebase might really be Patched-X. Philsophically, my stance here is generally: too bad, monkey patching is evil and this is the price you pay.
To add to the comment of @cburroughs above, it would be useful to cater to existing workflows.
In Plone the solved that at some point by adding FunnelWeb that allows any html that conforms to a templated format to be imported into Plone. It means that if you have a workflow that includes sphinx there is no need to change it, but instead set up a pipeline to import the sphinx html output to Plone whenever there is a change.
It would be great to have a the possibility to set up a pipeline from Sphinx builds, or the html of other documentation systems to Diviner, and a quick route to make it useful and easier to bring into the fold of existing workflows.