Develop a Nuance-based Phabricator reporting/support flow
Closed, ResolvedPublic

Description

We've had clear bug reporting guidelines in Contributing Bug Reports for a long time now, but still receive many reports which are not successful in following these guidelines. For example, in the last couple of weeks:

  • T12130: Not up to date, not upstream code, no reproduction steps.
  • T12122: Not up to date, resolved by update.
  • T12114: Not up to date, no real reproduction steps.
  • T12111: Not up to date, no real reproduction steps.
  • T12095: Forked, no working reproduction steps.
  • T12108: Not upstream code?
  • T12102: No version information, no real reproduction steps.
  • T12096: Not a bug report.
  • T12083: No working reproduction steps.

We also receive a fair number of reports which sort of make some kind of effort to follow these guidelines, but have complicated reproduction cases which are not actually reproducible, like these:

  • T12129: Vague, complicated reproduction steps, not reproducible.
  • T12112: Complicated reproduction steps, fails to reproduce.
  • T12113: Hugely complicated reproduction steps, failed to reproduce, ultimately resolved (user had already registered another account with the same email address).
  • T12081: "Reproduction steps" omitted critical information where Phabricator had been manually modified, resolved in ~6 minutes with a reproduction environment.

We also receive many reports which successfully follow the guidelines. In almost all cases, the outcome of these reports is good:

  • T12118: Contextualized in about 3 hours (submitted at 4AM).
  • T12117: Fixed in about 3 hours (submitted at 3AM).
  • T12097: Diagnosed in 15 minutes, fixed in 30 minutes.
  • T12092: Fixed in 9 minutes.
  • T12090: Workaround in 4 minutes, fix in 7 minutes.
  • T12087: Workaround in 40 minutes, documentation fix within 24 hours.
  • T12086: Fixed in 6 minutes.
  • T12082: Fixed within 24 hours.
  • T12080: Fixed in about 12 hours (submitted at 9:30 PM).

Finally, across all reports, things occasionally turn into long discussions (like T11597 and T9640) where everyone ends up pretty grumpy because we don't value the same things that users do and particularly because we don't assign the same value they do to our time, their time, or the value of contributions.


Of these, the "build a machine" reproduction cases are most concerning to me because they're very time consuming. The "discuss values" and "reporter didn't follow the rules" cases are somewhat concerning to me because they're emotionally negative for us and for users. In particular, it's personally frustrating to wake up to, say, 3 reports where no attempt seems to have been made to follow the rules and then maybe one "go build a machine from scratch to try to reproduce this crazy issue which almost certainly will not reproduce", since my next hour or two are then shot having a series of generally negative interactions that waste my time.


One issue with this approach is that it appears to reward users who do not follow the rules. For example, there is an obvious incentive to ignore the rules or not bother to read them when submitting a report, since that lets you submit the report faster. I think your outcome is actually much worse on average (reports that follow the rules are almost always resolved quickly; following the rules frequently resolves issues without requiring a report; reports that do not follow the rules rarely have positive outcomes), but you get something submitted faster and it isn't obvious that you're hurting yourself.


Tentatively, I'd like to experiment with a system like this:

Two Queues: Two Nuance support queues, basically "free" and "paid". Anyone can submit to the free queue, and support requests from free instances go to that queue too. You can submit to the paid queue by having a Phacility SAAS instance or buying a support plan.

Free queue is public, with no guarantee of a response.

Paid queue is private, with guaranteed response times (not super rigorous ones).

Community Review: Items in the free queue are initially private, and must undergo community review. In this phase, a community member confirms that the report follows the rules (for example: up to date, not forked, has valid reproduction instructions which work for them). This is a human intelligence task, but not one that requires in-depth knowledge of Phabricator. If the report follows the rules (and isn't printer fax spam), they pass it through. We then deal with it normally, in most cases by promoting it to become a real task and then moving forward normally.

While a report is in the review phase, these rules are explained to the submitter and they're offered ways to review other reports or fast-track their request by buying a support plan. (And maybe let them jump to the top of the queue by performing review?)

This is promising to me because I think it makes the cost of not following the rules more obvious, reframes reproduction instructions as stuff the community has to be able to follow, gives the user a reality check like "oh, man, I don't want to go repro any of this garbage, so probably no one will want to repro mine, maybe I should put in a bit more work" and gives users who want to contribute a way to contribute that is unambiguously valuable (vs submitting patches). (Ideally we'd just pay users who do community review, but not sure how practical that is given all the tax/legal stuff -- we could cross that bridge when we come to it.)

Critically, it gives us a free pass to ignore (or never even look at) reports which don't follow the rules.

We could also experiment with cheap promotion ("first queue promotion for $5"); I'm not sure if this would be positive or negative overall. I suspect some of the worst offenders, like T12112, place so little value on our time that they would not pay $5 even for immediate review. However, this could also create users who feel entitled to a comprehensive response because they're paying for it. We could also promote users who submit quality reports so they can skip community review, even in the free queue (or give them credits in the paid queue or something so that we don't have to have an uncomfortable conversation about not abusing your skip-review powers).

I'm not exactly sure what we'd do with general support requests. Maybe we just write up some simple rules for them too and submit them through the same process, then turn them into documentation or blog posts or Ponder questions if they're good enough. Maybe you just can't ask them in the free queue. I'd like to basically nuke 95% of Ponder and turn off open access to it.


Broadly, users don't seem to care too much about getting no response, or a very slow response, or even an inaccurate response. Many, many questions in Ponder have no answer or an inaccurate answer, but we don't see users complaining about that that I can recall.

Users do seem care -- sometimes, a lot -- when a human says "no" immediately.

So some of what this approach would do is make those "no" responses much slower (e.g., automatic expiry after 30 days without community review or something for the "build a box from scratch" stuff that no one has time for) and less human (still technically a human, but a human following a checklist).


Another thing we can do is publish average response/resolution times for queues. I think our resolution times are exemplary, but you currently only know that after we resolve something. This also makes a soft response guarantee more palatable: "guaranteed next-business-day response + median response time 4 minutes" feels a lot better than not knowing the median response time.

Related Objects

Mentioned In
D18815: Allow additional options in .arcunit files
T13011: Let community to fund features and bug fixes for increased priority and earlier delivery
T13007: Integrating `arc lint` and auto-fixers
2017 Week 29 (Late July)
T12251: Add author information when creating a build in Buildkite
T12778: Repository mirroring is much more frequent than necessary
T12738: Nuance: As a Helpdesk, for Phabricator/Phacility support
T12705: Set visibility through arc diff
T12681: Upcoming Support Product / End of Paid Prioritization
T12525: amckinley's Onboarding
T12511: Add Support For User Definable Macro Or Template Markup to Phriction Similar To WikiMedia Templates
T12335: Ability to lock a task to prevent further edits
T12275: Allow to rename applications
Z1336: General Chat
T12176: Importing large SVN repository causes a full disk
T9212: Community Feedback: How should we handle free support?
Mentioned Here
T12922: Update Bug Report docs
T12157: Make it visibly clearer that a Phabricator user account has been disabled
T12162: Releasing Phabricator OVA for Easy Installation
T12172: Better color scheme for diffs
T12177: Provide time-to-review statistics and reporting in Facts
T12179: Feedback due to D17076
T12183: Stop Subscribe on Comment
T12188: Provide a global configuration option for repository "Autoclose On" branch setting
T12192: Reorder & Hide Login Modules on Main Login Page
T12193: Allow mentions to work in Phriction edit notes
T12195: Implement "new" reCAPTCHA UI
T12212: New Feature Task Test
T12220: New custom type field: Project Tags
T12222: Add custom fields to the "Task Action..." menu
T12223: Allow to duplicate Forms
T12229: Breadcrumbs for Project structure
T12230: When I edit a task I cannot see some fields I had there when I created the task
T12233: Allow dashboard panels to be "hidden" when considered "empty"
T12234: Add a "Duplicates" tab to the "Related Objects" section on task page
T12239: Search for deleted or renamed files in Diffusion
T12240: Best of Both Worlds: one email to everyone with privacy controls and good threading
T12241: Make it easier to find objects while in remarkup box
T12242: arc commandeer
T12247: mail notification for those tasks which overdue date
T12250: Add repository short name as a variable in build steps
T12251: Add author information when creating a build in Buildkite
T12259: Report on delta in task list between two points in time
T12260: Report on historical assignment of points
T12263: Javascript licence
T12274: Find/filter inline comments that are not "Done"
T12275: Allow to rename applications
T12276: Allow to configure arc commit message
T12277: Show the points of the sub and parent tasks in the Task Graph
T12278: Choose specific form on Create Subtask
T12283: Rocket.Chat
T12285: uml support
T12288: Phabricator OAuth 2.0 initiating SSO login from a Third Party website
T12290: `arc diff` should copy the differential URL to clipboard
T12291: Diffusion doesn't display file moves/renames in an easily consumable/reviewable way
T9640: Make Phabricator compatible with PHP7
T11597: Adding a user to multiple groups would save a lot of time
T12080: Clicking "Manage Password" next to repo URI does not change page
T12081: "Unable to establish a connection to any database host" via ssh git push
T12082: Differential emails are rendering literal HTML tags
T12083: Can't set Related Task/Commit
T12086: Differential keyboard shortcuts for navigating diffs disappeared in the last update
T12087: Clusterizing a repository onto a service with multiple machines results in ambiguous leader
T12090: `bin/hunks migrate` fails on differential_hunks row with NULL changes field
T12092: conduit.querydiffs generates error when optional `id` argument is empty
T12095: Differential reads and writes draft status to different tables, leading to various UI inconsistencies
T12096: How to remove rule to make me reviewer
T12097: Object Herald Rules for Commits on Projects no longer working as expected
T12102: Cannot create wiki
T12108: Exception while running arc diff/land script from git extension.
T12111: Daemons keep failing but then I can't restart
T12112: Upload file stalles network connections
T12113: Email in LDAP registration form is empty
T12114: Upside-down segment in task graph
T12117: Conduit API method audit.query fails with 'Undefined class constant CONCERN_ACCEPTED'
T12118: Differential email does not contain reviewers/subscribers information anymore
T12122: Cannot login to Phabricator
T12129: Mercurial repository updates cause high CPU usage
T12130: DateTime parsing error in phd log
OCram awarded a token.Feb 8 2017, 6:13 AM
OCram added a subscriber: OCram.
hskiba added a subscriber: hskiba.Feb 10 2017, 9:47 AM

Here's an attempt to manually classify recent feature requests. I'm not sure how much value we're really getting out of this channel except from users who are already part of Community:

In the last chunk of them, these were from Community or similar:

These were not valuable (usually, they fail to describe a root problem):

These were duplicates (most of which also didn't follow the rules):

These are maybe somewhere in the realm of reasonable eventually, but probably not great uses of our time (none were filed with a root problem adequately described, but there might be something if we dig deep enough):

Finally, these were relatively reasonable-ish:

20after4 added a subscriber: 20after4.EditedFeb 22 2017, 6:36 AM

FWIW I think this is pretty genius. Especially the repro-or-it-didn't-happen aspect.

I definitely have experienced a "fix in master within 5 minutes" response more than once when it's reproducible. I have also been involved in at least one thing that was a pain for you to try reproducing only to find out that it was caused by downstream changes and wasted your time. It feels really shitty to put someone though that but it's not much consolation for epriestley knowing I felt bad about it.

Hopefully this will solve most of the issues outlined above.

For whatever it's worth, I appreciate what you do and I always feel you go above and beyond what is reasonable to expect from the $0 I'm paying you
Honestly, I would not feel too ripped off if I had to pay $50 per bug report. I'm sure most people wouldn't appreciate that though, especially if without prior experience to tell them it's worth it.

I really like the idea of letting people earn credits for the paid queue by reviewing reports in the free queue. Now if only you could turn it into a marketplace where people offer phucks (phabricator bucks) as a bounty for reproducing their bug report so it can get through the queue. They could buy phucks for $1 (minimum purchase: $5) and set the bounty to any value they want. You really want your bug report looked at and can't be bothered to document how it's reproducible? Well for a bounty of $500 phucks, someone will probably be willing to try. ;)

You could pay reviewers with Mana ;)

urzds added a subscriber: urzds.Jul 12 2017, 11:14 AM
epriestley closed this task as Resolved.Jul 16 2017, 11:23 AM
epriestley claimed this task.

The new paid support application is now in closed beta. This will become publicly available in the relatively near future, then become then channel for SAAS support after that. It may later become the channel for some types of free/community support (most likely bug reports, per above) but this probably won't happen for a while.

We've launched a separate copy of Discourse (https://discourse.phabricator-community.org/) to replace all the current community channels on this install, and closed registration here, so this install is now invite only. We may add a pathway for paying customers to register here via OAuth from admin in the future, but it's not yet clear how much use/interest this would really see.

See also T12922.