Page MenuHomePhabricator
Feed Advanced Search

Feb 25 2017

epriestley added a comment to T12316: Phacility Deploy Log 2017 Week 8.

The re-launched hosts seem OK, I'm waiting for DNS to update and then redeploying them.

Feb 25 2017, 12:10 PM · Phacility, Ops
epriestley added a comment to T12316: Phacility Deploy Log 2017 Week 8.

The hosts launched into an "Auto-Assign Public IPv4" subnet but didn't get IPv4 addresses. Restarting them didn't help, and there's no apparent way to assign public addresses after the fact. I'm going to throw them away and try again.

Feb 25 2017, 12:06 PM · Phacility, Ops
epriestley added a comment to T12316: Phacility Deploy Log 2017 Week 8.

I'm going to put more hosts into production immediately before pursuing other issues.

Feb 25 2017, 11:59 AM · Phacility, Ops
epriestley added a comment to T12316: Phacility Deploy Log 2017 Week 8.
  • Some reserved instances ended on Feb 19th; I should make sure reserved instances match production instances.
Feb 25 2017, 11:29 AM · Phacility, Ops
epriestley edited the content of Phacility Cluster Devices.
Feb 25 2017, 11:21 AM
epriestley added a comment to T12316: Phacility Deploy Log 2017 Week 8.
  • The tier as a whole is 88% full.
Feb 25 2017, 11:12 AM · Phacility, Ops
epriestley added a parent task for T12316: Phacility Deploy Log 2017 Week 8: T12217: Reduce the hardware cost of Phacility free instances.
Feb 25 2017, 11:11 AM · Phacility, Ops
epriestley added a subtask for T12217: Reduce the hardware cost of Phacility free instances: T12316: Phacility Deploy Log 2017 Week 8.
Feb 25 2017, 11:11 AM · Ops, Phacility
epriestley created T12316: Phacility Deploy Log 2017 Week 8.
Feb 25 2017, 11:10 AM · Phacility, Ops
epriestley edited the content of Changelog.
Feb 25 2017, 2:18 AM
epriestley created 2017 Week 8 (Very Late February).
Feb 25 2017, 2:17 AM
epriestley updated subscribers of T12314: Support formal task subtypes (like "bug" vs "feature").

So here's a tentative plan for subtyping:

Feb 25 2017, 1:09 AM · EditEngine, Maniphest, Custom Fields, Feature Request
epriestley added a comment to T12314: Support formal task subtypes (like "bug" vs "feature").

Custom Forms are the general tool we want to use here, but they currently have some limitations for this use case. As a starting point, here's what they can do today, some of the strengths, and some of the weaknesses.

Feb 25 2017, 12:49 AM · EditEngine, Maniphest, Custom Fields, Feature Request
epriestley added a comment to T12315: Arcanist setup documentation is circular.

Maybe the "Install Arcanist" header on Quick Start could be like:

Feb 25 2017, 12:29 AM · Bug Report
epriestley added a comment to T12315: Arcanist setup documentation is circular.

That is, this passage:

Feb 25 2017, 12:15 AM · Bug Report
epriestley added a comment to T12315: Arcanist setup documentation is circular.

There's a short version and a long version. Each version links to the other version, saying "here's a different version of these instructions, if you'd prefer that one". How could we make this more clear?

Feb 25 2017, 12:13 AM · Bug Report

Feb 24 2017

epriestley added a comment to T12314: Support formal task subtypes (like "bug" vs "feature").

This is a followup to a private discussion elsewhere, I'll flesh it out a bit when I have a chance.

Feb 24 2017, 10:20 PM · EditEngine, Maniphest, Custom Fields, Feature Request
epriestley added a comment to T12298: Allow daemon pools to autoscale down to 0 processes.

Hibernating daemons currently show as "Waiting" in the Daemon console, but I'm not going to worry about that for now.

Feb 24 2017, 6:49 PM · Daemons, Ops, Phacility
epriestley added a comment to T12296: Improve Phacility repository import performance.

Ah, okay. This is the relevant part:

Feb 24 2017, 6:46 PM · Customer Impact, Diffusion, Ops
epriestley added a comment to E1394: Dentist.

good luck w/ teeth 😁

Feb 24 2017, 6:07 PM
epriestley added a revision to T12296: Improve Phacility repository import performance: Unknown Object (Differential Revision).
Feb 24 2017, 6:05 PM · Customer Impact, Diffusion, Ops
epriestley added a comment to T12296: Improve Phacility repository import performance.

Starting with repository update:

Feb 24 2017, 5:56 PM · Customer Impact, Diffusion, Ops
epriestley added a revision to T12298: Allow daemon pools to autoscale down to 0 processes: Unknown Object (Differential Revision).
Feb 24 2017, 5:14 PM · Daemons, Ops, Phacility
epriestley added a revision to T12298: Allow daemon pools to autoscale down to 0 processes: Unknown Object (Differential Revision).
Feb 24 2017, 5:06 PM · Daemons, Ops, Phacility
epriestley added a comment to T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").

Broadly, there is nothing Phabricator could have done differently to anticipate or prevent this issue.

Feb 24 2017, 2:31 PM · Guides, Security
epriestley added a comment to T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").

We could also provide some kind of bin/auth burn-everything-to-the-ground to automate these steps, but I worry that it would almost never be tested (I think it's very difficult to test comprehensively in an automated way) or run, and there's a good chance it might be forgotten about when authentication changes occur.

Feb 24 2017, 2:24 PM · Guides, Security
epriestley updated the task description for T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").
Feb 24 2017, 2:12 PM · Guides, Security
epriestley added a comment to T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").

Ah, you're right. It looks like request data may also have leaked. I'll edit my earlier posts to reflect that. Briefly, the change is: passwords and VCS passwords are potentially at risk.

Feb 24 2017, 2:12 PM · Guides, Security
epriestley added a comment to T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").
NOTE: If Aphlict is served from the same domain as Phabricator (e.g., just on a different port) with Cloudflare as an intermediary, none of the rest of this comment applies: browsers could have sent credentials with the request and Cloudflare could have cached them.
Feb 24 2017, 12:50 PM · Guides, Security
epriestley added a comment to T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").

To cycle passwords, you can do this:

Feb 24 2017, 12:43 PM · Guides, Security
epriestley added a comment to T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").

Although this problem is severe, the absolute size of the leak appears to be small (my task title is hyperbolic). From the Cloudflare disclosure:

Feb 24 2017, 12:21 AM · Guides, Security
epriestley updated the task description for T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").
Feb 24 2017, 12:15 AM · Guides, Security
epriestley updated the task description for T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").
Feb 24 2017, 12:15 AM · Guides, Security
epriestley created T12313: Cloudflare leaked all HTTPS traffic on the internet into public caches ("Cloudbleed").
Feb 24 2017, 12:11 AM · Guides, Security

Feb 23 2017

epriestley awarded Bug Meister to recipient: nicolast.
Feb 23 2017, 11:38 PM
epriestley awarded Bug Meister to recipient: willplatts.
Feb 23 2017, 11:37 PM
epriestley added a comment to T12304: Herald fails to add auditors if the author is an auditor.

Thanks for the report! This should be fixed in HEAD of master, and promote to stable tomorrow, a little more than 24 hours from now. Let us know if you run into any issues.

Feb 23 2017, 11:37 PM · Herald, Bug Report
epriestley added a comment to T12302: Herald adding different auditors but only the last one is kept.

Thanks for the report! This should be fixed in HEAD of master, and promote to stable tomorrow, a little more than 24 hours from now. Let us know if you run into any issues.

Feb 23 2017, 11:36 PM · Herald, Bug Report
epriestley closed T12304: Herald fails to add auditors if the author is an auditor as Resolved by committing Restricted Diffusion Commit.
Feb 23 2017, 11:19 PM · Herald, Bug Report
epriestley closed T12302: Herald adding different auditors but only the last one is kept as Resolved by committing Restricted Diffusion Commit.
Feb 23 2017, 11:15 PM · Herald, Bug Report
epriestley added a revision to T12304: Herald fails to add auditors if the author is an auditor: Unknown Object (Differential Revision).
Feb 23 2017, 10:47 PM · Herald, Bug Report
epriestley added a revision to T12302: Herald adding different auditors but only the last one is kept: Unknown Object (Differential Revision).
Feb 23 2017, 10:44 PM · Herald, Bug Report
epriestley added a comment to T7954: Make it easier to upload XHProf profiles.

I didn't "ref" it here, but D17401 sort of technically did this. I'd like to clean it up a bit more before calling this resolved, though -- particularly, this is basically a secret that only I know about.

Feb 23 2017, 7:34 PM · Infrastructure, XHProf
epriestley added a comment to T12172: Better color scheme for diffs.

Also, above, I used the term "deuteranomaly" when I meant "deuteranopia". I mistakenly believed they were describing the same type of colorblindness, but they are two different types.

Feb 23 2017, 7:13 PM · Accessibility, Feature Request
epriestley added a comment to T12311: Allow users to choose both "Larger Fonts" and "High Contrast" at the same time.

Possibly this should really be three dimensions (Size, Contrast, Color Correction).

Feb 23 2017, 7:11 PM · Celerity, Accessibility
epriestley added a comment to T12172: Better color scheme for diffs.

See T12060#213081 some end notes and followups.

Feb 23 2017, 7:11 PM · Accessibility, Feature Request
epriestley added a comment to T12060: Colorblind accessibility for red/green highlighting in e-mails.

See some additional discussion in T12172. Briefly:

Feb 23 2017, 7:10 PM · Accessibility, Design, Feature Request, Restricted Project
epriestley added a project to T12311: Allow users to choose both "Larger Fonts" and "High Contrast" at the same time: Celerity.
Feb 23 2017, 7:03 PM · Celerity, Accessibility
epriestley created T12311: Allow users to choose both "Larger Fonts" and "High Contrast" at the same time.
Feb 23 2017, 7:03 PM · Celerity, Accessibility
epriestley closed T12060: Colorblind accessibility for red/green highlighting in e-mails as Resolved by committing Restricted Diffusion Commit.
Feb 23 2017, 6:57 PM · Accessibility, Design, Feature Request, Restricted Project
epriestley closed T12172: Better color scheme for diffs as Resolved by committing Restricted Diffusion Commit.
Feb 23 2017, 6:57 PM · Accessibility, Feature Request
epriestley added a comment to T12157: Make it visibly clearer that a Phabricator user account has been disabled.

@chad did another pass on user hovercards and they should be even more clear now (this screenshot doesn't demonstrate it clearly, but we also greyscale the profile photo);

Feb 23 2017, 6:46 PM · Profile, Feature Request
epriestley added a comment to T12288: Phabricator OAuth 2.0 initiating SSO login from a Third Party website.

Why do you have multiple authentication providers?

Feb 23 2017, 6:43 PM · Auth, Feature Request
epriestley added a revision to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile: Unknown Object (Differential Revision).
Feb 23 2017, 6:41 PM · XHProf, Ops, Phacility
epriestley added a comment to T12196: PHP7, particularly before 7.1.2, segfaults a whole lot while running CLI scripts.

Yeah, I'm basically just keeping this open as a dumping ground for the moment and purely selfishly because my segfaults aren't resolved yet -- brew didn't have PHP 7.1.2 last I checked, although it seems like it now does. I've upgraded and I'll see if that improves things.

Feb 23 2017, 6:04 PM · Bug Report
epriestley added a revision to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile: Unknown Object (Differential Revision).
Feb 23 2017, 5:59 PM · XHProf, Ops, Phacility
epriestley added a revision to T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile: Unknown Object (Differential Revision).
Feb 23 2017, 5:31 PM · XHProf, Ops, Phacility
epriestley edited projects for T12310: Phd out of memory error, added: Bug Report (Needs Information); removed Bug Report.

We need reproduction instructions which allow us to reproduce this issue in a clean environment. See Contributing Bug Reports and Providing Reproduction Steps.

Feb 23 2017, 3:56 PM · Bug Report (Needs Information)
epriestley closed T12307: Users unable to add new credit cards to Autopay? as Invalid.

This has been "resolved" by switching to bartering various sundry goods and foodstuffs instead of billing via credit card.

Feb 23 2017, 12:02 AM · Phortune, Phacility

Feb 22 2017

epriestley closed T12301: `Create Form` always leads to page creating forms for configuring forms as Resolved by committing Restricted Diffusion Commit.
Feb 22 2017, 11:00 PM · EditEngine, Bug Report
epriestley added a revision to T12301: `Create Form` always leads to page creating forms for configuring forms: Unknown Object (Differential Revision).
Feb 22 2017, 10:34 PM · EditEngine, Bug Report
epriestley closed T12306: Failing svn connection brings daemon down as Resolved by committing Restricted Diffusion Commit.
Feb 22 2017, 10:11 PM · Daemons, Bug Report
epriestley added a revision to T12306: Failing svn connection brings daemon down: Unknown Object (Differential Revision).
Feb 22 2017, 10:09 PM · Daemons, Bug Report
epriestley added a revision to T2794: Implement Phage (like Hypershell): Unknown Object (Differential Revision).
Feb 22 2017, 9:49 PM · Phage, Infrastructure
epriestley added a revision to T12218: Reduce the operational cost of a larger Phacility cluster: Unknown Object (Differential Revision).
Feb 22 2017, 9:49 PM · Ops, Phacility
epriestley added a comment to T12298: Allow daemon pools to autoscale down to 0 processes.

No time like the present.

Feb 22 2017, 9:14 PM · Daemons, Ops, Phacility
epriestley created T12307: Users unable to add new credit cards to Autopay?.
Feb 22 2017, 12:41 PM · Phortune, Phacility
epriestley added a subtask for T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile: T7954: Make it easier to upload XHProf profiles.
Feb 22 2017, 12:16 PM · XHProf, Ops, Phacility
epriestley added a parent task for T7954: Make it easier to upload XHProf profiles: T12297: Make Conduit API calls on `admin.phacility.com` reasonably easy to profile.
Feb 22 2017, 12:16 PM · Infrastructure, XHProf
epriestley moved T12306: Failing svn connection brings daemon down from Backlog to vNext on the Daemons board.
Feb 22 2017, 12:15 PM · Daemons, Bug Report
epriestley added a project to T12306: Failing svn connection brings daemon down: Daemons.

It is expected that taskmasters will exit and restart after an unexpected failure, but this should not trigger setup warnings. I'll see if I can reproduce this.

Feb 22 2017, 12:14 PM · Daemons, Bug Report
epriestley claimed T12304: Herald fails to add auditors if the author is an auditor.

This might be the same as T12302, but I'll see what's going on.

Feb 22 2017, 12:05 PM · Herald, Bug Report
epriestley added projects to T12305: Persistent chat overlaps various "next" buttons: Design, Conpherence.

Persistent chat was originally a separate column (so it didn't overlap with anything on the page) but was brought into the main page recently.

Feb 22 2017, 12:04 PM · Conpherence, Design, Bug Report
20after4 awarded T10756: Make daemons work correctly no matter where they are or how many copies are running a Orange Medal token.
Feb 22 2017, 4:54 AM · Restricted Project, Daemons

Feb 21 2017

epriestley claimed T12302: Herald adding different auditors but only the last one is kept.
Feb 21 2017, 11:02 PM · Herald, Bug Report
epriestley claimed T12301: `Create Form` always leads to page creating forms for configuring forms.

Yeah I'll take it, pretty sure it's my fault.

Feb 21 2017, 11:01 PM · EditEngine, Bug Report
epriestley added a comment to T12301: `Create Form` always leads to page creating forms for configuring forms.

I think this is probbbbbbbably an actual bug unless I'm totally out of it -- possibly my fault from D17372.

Feb 21 2017, 10:54 PM · EditEngine, Bug Report
ofbeaton awarded T10279: Allow bot users to see email addresses of users a Like token.
Feb 21 2017, 9:26 PM · Feature Request
thoughtpolice awarded T2794: Implement Phage (like Hypershell) a Love token.
Feb 21 2017, 8:57 PM · Phage, Infrastructure
epriestley added a revision to T12298: Allow daemon pools to autoscale down to 0 processes: Unknown Object (Differential Revision).
Feb 21 2017, 4:50 PM · Daemons, Ops, Phacility
epriestley added a revision to T12298: Allow daemon pools to autoscale down to 0 processes: Unknown Object (Differential Revision).
Feb 21 2017, 4:48 PM · Daemons, Ops, Phacility
epriestley added a comment to T12251: Add author information when creating a build in Buildkite.

No.

Feb 21 2017, 4:36 PM · Harbormaster, Feature Request
epriestley moved T6615: Worker task table has some remaining awkward keys from Availability to vNext on the Daemons board.
Feb 21 2017, 12:37 AM · Infrastructure, Daemons
epriestley closed T10756: Make daemons work correctly no matter where they are or how many copies are running, a subtask of T10751: Make Phabricator Highly Available, as Resolved.
Feb 21 2017, 12:35 AM · Restricted Project, Infrastructure, Prioritized
epriestley closed T10756: Make daemons work correctly no matter where they are or how many copies are running as Resolved.

I believe this is effectively resolved now -- we're running all the daemons on multiple hosts on secure. There's remaining followup work (T6768, T6615, T10753, T10754) but I think the core is in reasonable shape.

Feb 21 2017, 12:35 AM · Restricted Project, Daemons
epriestley added a comment to T12115: ApplicationTransactionPublishWorker can retry indefinitely, sending email as a side effect, if a transaction takes more than 2 hours to publish.

T11708 has almost nothing to do with this, but the fix for this will rewrite the code that's running into issues and probably moot them.

Feb 21 2017, 12:31 AM · Restricted Project, Daemons, Differential, Transactions
epriestley merged task T11708: Daemons erroneously saying "Diff does not exist" into T12115: ApplicationTransactionPublishWorker can retry indefinitely, sending email as a side effect, if a transaction takes more than 2 hours to publish.
Feb 21 2017, 12:30 AM · Daemons, Differential, Bug Report
epriestley merged T11708: Daemons erroneously saying "Diff does not exist" into T12115: ApplicationTransactionPublishWorker can retry indefinitely, sending email as a side effect, if a transaction takes more than 2 hours to publish.
Feb 21 2017, 12:30 AM · Restricted Project, Daemons, Differential, Transactions
epriestley added a comment to T11708: Daemons erroneously saying "Diff does not exist".

I'm just going to merge this into T12115, which isn't really related, but will rewrite this code and probably "fix" this, since a reproduction case seems elusive.

Feb 21 2017, 12:30 AM · Daemons, Differential, Bug Report
epriestley moved T5401: Record total time in queue for tasks from Backlog to vNext on the Daemons board.
Feb 21 2017, 12:27 AM · Daemons
epriestley closed T8809: Support more complex human-usable weekly clocks in trigger infrastructure as Wontfix.

This could probably be built with RRULEs now, but we don't currently have use cases / plans around a general-purpose cron-like tool.

Feb 21 2017, 12:27 AM · Daemons
epriestley closed T3554: Implement a `bin/worker kill <criteria>` for batch-flushing tasks from the queue as Resolved.

Use cases seem to be fairly well covered now.

Feb 21 2017, 12:25 AM · Restricted Project, Daemons
epriestley closed T5334: View a graph showing worker load over time as Resolved.

This is 2+ years old and autoscaling probably covers it now. T5401 is probably a more tangible attack on this.

Feb 21 2017, 12:23 AM · Restricted Project, Daemons, Facts
epriestley closed T7484: Lock is not locked by this process as Wontfix.

This got a little more work when clustering was written but it's essentially a log of two-year-old ghost sightings at this point and not actionable.

Feb 21 2017, 12:21 AM · Restricted Project, Daemons
epriestley added a comment to T8871: Indexing a task with 2,000 comments required a lot of RAM in mid-2015.

I guess this one can live for now since that's reproducible/actionable.

Feb 21 2017, 12:19 AM · Search, Daemons
epriestley renamed T8871: Indexing a task with 2,000 comments required a lot of RAM in mid-2015 from Phabricator daemons autoscale themselves to OOM to Indexing a task with 2,000 comments required a lot of RAM in mid-2015.
Feb 21 2017, 12:19 AM · Search, Daemons
epriestley closed T9603: PHD daemons regularly stopping/starting as Resolved.

Probably a dupe of T11708? I'm just going to kill this one since it's old as dirt.

Feb 21 2017, 12:16 AM · Daemons
epriestley closed T10148: diffusion.rawdiffquery can not encode/transport all diffs (e.g. to non-utf8 files) as Resolved.

I believe this was likely resolved by D16460. This (binary transport over Conduit) is still a problem in general, and T5955 is the general remedy, but diffusion.rawdiffquery specifically now returns data via Files.

Feb 21 2017, 12:15 AM · Conduit, Daemons
epriestley closed T9855: DifferentialChangesSinceLastUpdateField may fail policy checks and prevent mail from sending as Resolved.

I think it is likely that this was resolved by D17123. If it wasn't, we don't have a reproduction case anyway so we can't move forward. I'm going to call this one dead until more information turns up.

Feb 21 2017, 12:13 AM · Daemons, Differential
epriestley closed T10811: `phd-daemon` crash after automatic restart following config change as Resolved.

We were never able to reproduce this convincingly and as far as we know the reporting install no longer uses Phabricator, so I'm going to close this out.

Feb 21 2017, 12:12 AM · Daemons, Bug Report