Page MenuHomePhabricator

Community Feedback: How should we handle free support?
Closed, DuplicatePublic

Subscribers
Tokens
"The World Burns" token, awarded by nateguchi2."The World Burns" token, awarded by thoughtpolice."Heartbreak" token, awarded by staticshock."Evil Spooky Haunted Tree" token, awarded by chad.
Assigned To
None
Authored By
epriestley, Aug 18 2015

Description

By "support", I mean anything where you're asking the upstream to spend time working with you on something. That includes bug reports, feature requests, revisions / pull requests, help with setup, IRC, etc.

We've grown to a scale where our current approach to free support isn't working very well anymore. We're going to make some changes, but don't have a great pathway forward. Since users on this install are among those most directly impacted, I figured I'd see if anyone has any useful feedback or ideas.

Currently, our support approach is mostly based around IRC and Maniphest. We try to get bug reports and feature requests into Maniphest, and funnel users to IRC for other issues. I think some of the problems we're seeing are:

  • IRC is presented as a primary support channel, but is a bad UX.
    • Many user problems don't help the upstream improve Phabricator anymore (e.g., one-offs, not real bugs).
    • Many Europe/Asia users show up at times when no one is active.
    • Many other questions are too complicated for the community to answer out of the goodness of their hearts (e.g., hour-long fishing expedition).
    • Major issue: we (and docs) say go here to get support; you usually don't get any.
  • Maniphest is often a bad UX.
    • Users with the most free time have the loudest voices.
    • Some users feel snubbed by an adversarial upstream if we don't spend enough time on an issue.
    • Some of the discussion is very low-value ("what's the status on this?").
    • Basically no luck documenting our way out of any of this.
      • No immediate incentive for users to read the documentation or craft good reports.

I think there are three sort of core problems here:

  • Bad IRC expectations: We tell people to go to IRC, where they don't get support. We should set expectations better.
  • Time Consuming for Upstream: We spend a ton of time on support, but it has much less impact on making Phabricator better or helping users than spending that time on development. This is really frustrating for us.
  • Emotionally Negative for Users: Some users feel rejected if we don't spend enough time helping them or can't prioritize their issues. This can be really frustrating for users.

These problems all have straightforward solutions individually, but they interact with each other in complicated ways.

For IRC, I think we're pretty clearly going to drop it as a primary support channel. There's no real way to give Europe/Asian users a good experience with an unlogged medium with a relatively high barrier to entry. However, it would be nice to replace it with something else instead of just saying "no support".

Reducing the time the upstream spends on issues is ideally an issue of improving issue quality: the top, say, 20% of issues are hugely valuable for us, it's just the other 80% that get less and less valuable down to the point where they're starting to take up way more of our time than they can ever repay. This seems hard. One problem is maintaining fairness, or an illusion of fairness (see below). We can also just shut down channels, but then we'll be throwing good reports away too. Literally spending less time on issues (i.e., rejecting them more abruptly) makes the experience more negative for users.

User emotions are probably the least straightforward. Ideally, we'd spend all the time required to resolve each problem, but this is obviously not possible (even if we had the time for it, many requests are contradictory or impossible), and we want to reduce the amount of time we spend. Generally, telling users "no" is negative, and telling them "no" quickly is even more negative: users feel like we haven't taken them seriously.

Some alternatives are essentially to ignore users or pretend to listen to users. I'm stating these in kind of an extreme way, but the effect of any change here is that users will have substantially less access to the upstream. However, this is what essentially every other company with a free product does, and it's probably actually a better experience for users (e.g., it's a less negative experience to be ignored than rejected).

These mechanisms are, on average, worse for users than a direct line to the upstream, but both give users a credible illusion that their voices have been heard, so the experience is more positive even though the outcome is worse.

We can ignore users with a community forum (we've inched forward in this vein by modernizing Ponder); we can pretend to listen to them with a private report queue which we basically just dump in the trash after sending a "Thanks for your valuable feedback, we'll consider it carefully!" canned response, or a public "vote on features" queue, although I like this less. Broadly, these mechanisms allow us to skim the wheat from the chaff without breaking the illusion of a caring upstream.

The most straightforward approach is probably:

  • Bugs and feature requests move from Maniphest to Nuance, a private queue, and we mostly start ignoring that channel with a polite form response. We remove public access to create tasks. (Users who we recognize as net positive contributors retain full access.)
  • General support requests move to Ponder, which we mostly ignore.

This doesn't feel great, but it's pretty much what every other company does and does seem to be the best balance of concerns.


Some other alternatives:

Some kind of paid support. I don't really see much of a way forward here. Ideally, we'd, say, make users pay $3 to file a ticket and this would just magically align incentives and solve all of the problems, but if the amount is trivial and just trying to align incentives, this makes users feel like they've invested a lot: a user who has paid $3 to get support expects vastly more from it than a user who has paid $0, since it's now "paid support". If the amount is actually in line with costs, there's already prioritization / consulting / Phacility. We also haven't had much luck trying to get users to make trivial payments to align incentives (see various experiments with Fund). The value of each ticket to us is also hugely different, from good bug reports (which we'd gladly pay to receive) to fishing expeditions ending in environmental causes (which we'd charge thousands of dollars for under consulting).

Gamify support. We could try blending community + queues into something like the League of Legends "Tribunal" process. Something like this:

  • You file your issue.
  • You're told "your issue awaits judgement; judge other issues to promote it to the top of the queue".
  • You vote on report quality, value, etc., of other open issues for a bit. This gives you and your issues points.
  • Your issues are also boosted by others judging them worthy, and hampered or rejected by others judging them unfit.
  • This process has cool sounds and visual effects.

This feels like it would be amazing if Phabricator was actually a game, and could probably work for some users as-is, but I worry it's a little too out there for many first-time users. Could maybe work if this side of things was substantially downplayed until you got into it, since I'd expect only a few users to do this anyway.

Event Timeline

epriestley raised the priority of this task from to Normal.
epriestley updated the task description. (Show Details)
epriestley added subscribers: epriestley, chad.
chad awarded a token.Aug 18 2015, 5:05 PM
nevogd added a subscriber: nevogd.Aug 18 2015, 5:22 PM

Just to spitball...

Could you replace IRC with something like a Conpherence thread:

  • With T7580, this would get you at least as good notifications as in IRC
  • This would get you the logging for Europe/Asia hours
  • There's less friction to get a more complicated issue into Maniphest/Ponder/Nuance if it seems like a chat isn't productive (in particular, would force people to create accounts)
  • Could possibly have some way of selecting a bunch of messages and exporting them to a maniphest task or similar that which is then searchable (it seems many questions in IRC are dupes or very similar because no history/searching). That would create an easy trail for people to pursue first for common setup issues.
  • You could do something like embed the thread on a panel on the default dashboard for new users (cool but possibly irrelevant?)

For increasing the quality of the reports you could:

  • Add a big link to default dash/home page which pops up a pre-templated maniphest task

You could do something similar with Herald to set expectations more appropriately by auto-commenting or prioritizing things that sound like setup problems (mentions of hosted repos, ssl, permissions, apache, etc.)


Overall, as a community member, things seem pretty ok. I've never felt emotionally negative about the sometimes terse responses.

I think Nuance/private queue isn't as ideal because then you lose the public searchability.

chad added a comment.EditedAug 18 2015, 6:46 PM

Maniphest alone is 2,334 requests for "support" for 2015, or roughly 10 tickets per day. This doesn't include other support channels or if a ticket is a 10 minute response or a 10 hour response.

The IRC support was a significant positive factor when we chose Phabricator. I dug up the old evaluation doc and right at the top had "Fanatical upstream support". The fact that it is IRC on freenode was nice since that signals a thing people use (instead of some astroturf fake free support ala google apps). That isn't to say that it will continue to be a worthwhile tradeoff to do all of that support, but I am very glad you did when we were new!

One of the oddities (compared to open source projects) is that there is no asynchronous channel for setup/confused-new-user/this-is-far-out-there-but-has-anyone-seen-this-before threads. Opening a ticket for 'how do I install X' for the 'project X bug tracker' is usually frowned upon. But I think in the case of Phabricator people are pointed at Maniphest? (or at least end up there without any other options) It sounds like Nuance/Ponder are hoped to be (1) part of the answer (2) better for upstream to manage than a mailing list. I think Ponder is also has a chance of being able to do 'here are a bunch of similar things' reasonable well and even if community members can't go on hours long fishing expeditions out of the goodness of their heart at least things can maybe get tagged/categorized/sorted. Maybe that would help with the volume of requests clogging up Maniphest?

Also getting ignored forever or closed by a computer seems normal for Stack Overflow but (inevitable?) comes across as annoying during bug tracker spring cleaning (and paradoxical cleaning sooner can come across worse). Ironically part of the problem is that upstream actually uses Maniphest to plan your work (instead of throwing cold over the wall) and thus need to keep it clean.

I know I'm also guilty of going epriestley, ^^ X for something specific to my install being busted up so an answer Soon would be Really Nice but I didn't need to interrupt what you are doing Right Now. Maybe that makes sense as a Nuance queue? Even if everyone currently there never abused directly asking epriestley something it's not the right expectation to get set your first time there. See also T6797 re community support.

You have a much better grasp of how much time is sunk into low return support, but the frustration is clear. I've read several variants of the essay in T7073#132756 -- and don't get me wrong it's a Damn Good essay series that has changed my view of how a project with a single opinionated upstream can function -- but I think the number of times it has been written is a sign that something other than the projects bug tracker is the right place to funnel people.

It sounds like (from 10000 feet and with your own tools) the metaphorical phabricator mailing list needs to split into -user and -dev. In most cases (at least from the point of maintaining reasonable boundaries) that seems to be a well worn and workable pattern. My egalitarian instincts hope the boundaries can remain soft & social instead of ACL bits -- it takes a lot of chutzpah to escalate straight to the LKML -- but I know that Phabricator is not a community driven project.

chad added a comment.Aug 18 2015, 7:50 PM

I'll be bringing Ponder officially online shortly. If anyone is interesting in being a "Ponder Moderator", please let us know, it does have some basic moderation features and battle testing it here should improve its usefulness everywhere.

avivey added a subscriber: avivey.Aug 18 2015, 8:48 PM

I think improving Conpherence is one of the major blockers to dropping IRC. The big things that are missing right now are T7580 and T6868. Right now the general chat in Conpherence feels dead because it doesn't indicate anyone is online to help with something, and this compounds the problem because then no-one adds messages to it and it feels even more dead.

The other issue is that it feels like upstream have already dropped IRC; I don't go there anymore because it also feels practically dead. The result is that both forms of real time discussion with upstream now feel dead or unused, which isn't great in terms of feeling like you have an impact on the software.

An issue I find with users feeling like they have no impact on things is that they're far less likely to report bugs, and much more likely to give up. I know I do this when using software that a) doesn't have a built-in bug reporter and b) the hurdles to tell someone something is broken are too high (like signing up for a mailing list when all I want to do is report one bug).

chad added a comment.Aug 19 2015, 2:48 AM

Most support shouldn't happen in IRC nor Conpherence, it's just handy sometimes to have one on one conversations. Right now these real time channels take a number of bug reports or how do I reports that should be funneled into other applications. I like having a "hangout" place, overall though.

I know sometimes I've asked in IRC channels "is this a bug or intended?" before filing a task. It feels like a real time channel for asking whether or not something is intended is the best option, rather than filing a task?

chad added a comment.Aug 19 2015, 3:23 AM

My concern (for me) is that real time issues lead to real time distraction, and thus less code produced. If we send everything into Nuance, I could just look at it several times a day when I have free time, and not worry about immediate disruptions. We'll have Herald on Nuance, so if something comes in "urgent" well get pinged to go look at it.

chad added a comment.Aug 19 2015, 3:28 AM

Maybe "real time" support should just be ended overall?

Yeah, I think we should end real-time support for the general public.

There's too much stuff like this, where I spend an hour walking someone thorough an issue and the solution is a bad hostfile in their environment causing an issue on their external-to-Phabricator build server:

https://secure.phabricator.com/chatlog/channel/6/?at=210420

There's no way to filter this stuff reliably and no way to bail out gracefully once it's clear that I'm on a fishing expedition.

Basically, I want to stop giving users an expectation that they're going to get help with this kind of stuff.

There are a bunch of regulars (@hach-que, @cburroughs, @avivey, etc) who are all net positives and who I'd like to still have real-time or near-real-time access, but I think anything realtime + upstream going forward should be labeled "developer hangout - no support" or something (or read-only to the public, or invite-only if necessary).

In T9212#132956, @chad wrote:

Maybe "real time" support should just be ended overall?

Yep - it should.

Even in typical company IT tech stuff for internal tickets is not realtime. We even especially told people that we (IT guys) will not respond unless problem is thoroughly analysed and described by person filling task. Any internal real-time support is reserved only for situations where suff breaks totally and a lot of customers are affected.

So... Best case scenario here would be to push forward with Nuance so it's somewhat usable for You and as @chad said " just look at it several times a day when I [@chad] have free time" and instead of allowing Maniphest tasks be filled with "howto X" or "help, i didn't follow docs and now phabricator doesn't work" 😉

Ponder seems like good alternative for "howto X" while Nuance seems like it would be great for "can't read lol, phab not works lol, help". That would free up maniphest for tasks about real bugs and features needed to advance Phabricator.

ps. if there's a badge involved I'd be happy to volunteer for Ponder Mod (at least one of many)

ftdysa added a subscriber: ftdysa.Aug 19 2015, 11:13 AM

As much as I have enjoyed the "real time" support, I agree that you guys should drop it. As a user of phabricator and as someone who has to support a similar suite of software for a company internally, I've always been surprised how responsive you guys were to support issues while still churning out work.

I think Ponder could help out a lot with the general questions that pop up. I've been following phabricator's development for over a year at this point and it seems like there are a lot of community "regulars" that could help filling out questions with useful answers. At the beginning it will probably take a bit more work, but after a while you could just point users at Q123 and be done with it.

Bugs and feature requests move from Maniphest to Nuance, a private queue, and we mostly start ignoring that channel with a polite form response. We remove public access to create tasks. (Users who we recognize as net positive contributors retain full access.)

I hope it doesn't have to come to this but I understand if it has to. Maybe create a Respond to Open Tasks Day and then just churn from the week's open tasks in one fell swoop. I've always enjoyed the insight into how phabricator is progressing.

I can't think of any other org that offers free, real-time support that isn't completely community-driven (i.e., advanced users, not core-team), and those are only where the community is very large and Growth (new installs divided-by knowledgeable users) is low.
So probably yeah, free real-time support is very nice, but we just can't afford it.

In Maniphest, maybe "Status: Need Info" and a few canned responses would help?

I never used IRC support to reach out to Phab devs. I never used Conpherence, either. In fact I don't think I would, because I know you are already very busy with building the platform and dealing with tasks.

Generally stackexchange is very helpful in finding answers to common questions. Maybe Ponder could be turned into this, or this kind of thing could be externalised to stackexchange.

I also have a mostly good experience with "rude" rejections. Most of the time it is very understandable (even if it takes a bit of back and forth) and the rest is probably just caused by the extreme frustration recently.

I like the idea of splitting -dev and -users. Some of my wild ideas are probably correctly addressed to Maniphest and the devs directly, but for many it would probably be of a better experience if they were filtered on a -users mailinglist first, before being escalated to -dev.

Badges and generally "reputation" might help, but is difficult to predict. I like the idea, though. Maybe a -dev/-user split could be based on this.

I was recently also thinking about approaching you for paid support. I was thinking about a pricing scheme similar to hosted Phab ($20 per user and month). Then I read about your payment expectations for Consulting... ;)
Joke aside, you probably can only reasonably give full support for stock installations (if at all), which we don't have due to some necessary local customisations (translations, space switcher, etc. - you've probably seen it...). So I guess this does not match our need.

Another case is T9164, where I explain our general intent and how we would like to use Phabricator, together with a bit of feedback on the rough edges and the request for "best practices", which I filed in Maniphest in lack of a better place (I didn't know about Ponder, yet). The primary intent is to provide you with some (hopefully valuable) feedback on ways to use Phabricator, so that in the future it can hopefully be shaped to better accommodate this. I guess that here the junk-queue-tool would have helped you, so you could reply "thanks for your valuable feedback" and thrown it in the trash. Or hopefully remembered it later, when you though about adjacent changes... ;)

staticshock added a subscriber: staticshock.

Firstly, I agree that stopping real time support is a good idea, It's never a great situation to be distracted whilst in the middle of things. - Many of the first time install / issues could be solved with fool proof installation process or something like an official Docker or Vagrant image.

I think classifying things better would at least help the current situation in Maniphest - Something like:

  • Feature suggestions - (I think these are useful for people to add as the conversation from the community on these can indicate which features are worthwhile to persue upstream).
  • Serious bugs / vulnerabilities - (Vulnerabilities is easier to classify, maybe errors in phabricator could be changed to give a severity level, The case I see this coming up is that a change is made and I go to some deep/hidden page and it starts throwing errors).
  • Installation Issues - (I think these are probably the bugs that take the most time and give the least mean value to phabricator as a whole; As was said above, these could probably be ignored / come under consultation.)
  • Other - (There's probably other things that people will talk about, I don't know if these should be put into Maniphest and perhaps should move to Ponder).

A not insignificant amount of tasks created on this installation are "Tests", people playing around with it as a demo - or trying to find vulnerabilities, these could be reduced with a separate "Demo" installation.

So far, we've been happy with the whole support experience, ours being limited to just posting Maniphest tasks, even if sometimes the tasks are rejected.

thoughtpolice added a subscriber: thoughtpolice.EditedAug 23 2015, 5:49 PM

One more vote from Haskell.org: I appreciate all the help in the past, but even then I am also of the opinion you should end the real time support. I think moving to something like a daily queue with Nuance is probably the best bet. I know for a fact we've pinged @epriestley more than once with breaking issues since we kind of live on the edge but for the most part all of our components are now very stable and I doubt they will fundamentally change soon, so we can live with the slight breakages sometimes (with solid backups upon every update of Phabricator).

One question I haven't seen addressed here is: what happens to Differential? To be quite honest I've been able to do most of my work out of tree with extensions, and in general the extensible only has gotten better over time. The fact Phabricator is free software is important to us, and has been very useful. But occasionally I really do write something I want to give back to everyone! Like D10387. But if you're going to lock Maniphest, I'm wondering what you plan on doing here, and what we'll see after all this.

Perhaps Nuance can also have an 'incoming non-Phacility patches queue', which you can isolate and kick most out with a 'thanks' or whatever every day when you have a minute or at least categorize them, or do nothing. I don't think workboards support diffs yet, but I was thinking it would be interesting if they could be assigned to a specific board in a project (e.g. my patch above would hit the Auth project and be put in Backlog Patches automatically.

robla added a subscriber: robla.EditedSep 14 2015, 7:01 PM

I tried answering @epriestley Q134, but got an error (which I failed to copy) and then another error "#1062: Duplicate entry '0-PHID-USER-4188de7cae592cf1bce4' for key 'key_oneanswerperquestion'". So...I'll put my answer here

Thanks @epriestley! There's a lot of reading there that I don't have time to read+absorb right now, but my skim gives me the gist. I think that you're right that a community needs to form. Being too insistent on "quality" questions is a community killer, which is why people like IRC. I think giving people a good ramp from the realtime "why is X a problem?" all of the way to the feeling of satisfaction that they now know X is going to give people the most satisfaction (and thus, be the best community builder). I may elaborate more on T9212 if/when I get the time to read enough to participate, and if I still feel the same way after reading everything.

chad added a comment.Sep 14 2015, 7:03 PM

oh @robla, yeah I'd be interested in a repo of that error, if you can. I've seen it in my sandbox (once) but I also nuke my questions and answers all the time so assume it was development related.

chad updated the task description. (Show Details)Sep 14 2015, 7:07 PM
SAPIBELTV renamed this task from Community Feedback: How should we handle free support? to Community Feedback: How should we handle free support? ipibel.Sep 21 2015, 12:58 PM
SAPIBELTV updated the task description. (Show Details)
epriestley renamed this task from Community Feedback: How should we handle free support? ipibel to Community Feedback: How should we handle free support?.Sep 21 2015, 12:58 PM
chad updated the task description. (Show Details)Nov 4 2015, 7:04 PM
stevex added a subscriber: stevex.Oct 17 2016, 5:31 PM

I'm going to merge this into T12134, which begins defining a more concrete plan roughly following some of the outline above. The major change from an upstream perspective since this task was filed is that we have a more specific set of harder technical requirements around support for Phacility SAAS instances (particularly after the launch of free instances), so it makes more sense to let those requirements drive product design and then accommodate general free support under that umbrella. Broadly, I expect to move support (bug reports, feature requests) into Nuance and continue reducing user access to the upstream.

Since this was filed, we effectively shut down IRC and stopped encouraging users to go there. Ponder and Conpherence replaced it somewhat, and have been somewhat successful (with other changes) in keeping general questions out of Maniphest. However, they mostly serve as graveyards where requests go to die, which isn't necessarily great, even though users don't seem to mind very much that no one ever answers anything in Ponder.

Support for organizational users who we have direct relationships with has generally also moved into private rooms in Conpherence, and this has worked well, but these users have almost always submitted high-quality requests anyway and never really contributed to any of the core issues here.

Several suggestions in this thread focus on stronger request types ("Feature Request" vs "Bug Report"). We developed documentation -- Support Resources, Contributing Bug Reports, and Contributing Feature Requests -- and changed the Maniphest workflows after EditEngine came online. However, many users still do not follow the rules, and report quality remains low (see T12134 for specifics), and arguing with users about these rules remains contentious. I think we got a little headroom out of this, but it didn't really change things very much in a sort of qualitative sense.

urzds added a subscriber: urzds.Jul 12 2017, 11:12 AM
epriestley updated the task description. (Show Details)Nov 10 2018, 12:30 PM