epriestley (Evan Priestley)Administrator
Overengineer

Projects (77)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Feb 8 2011, 1:28 AM (349 w, 1 d)
Roles
Administrator
Availability
Available

Psyduck is the greatest pokemon of all time.

Recent Activity

Tue, Oct 17

epriestley committed rPd36f98a15a6a: Clarify acceptable values for `--threshold` in `search ngrams` (authored by epriestley).
Clarify acceptable values for `--threshold` in `search ngrams`
Tue, Oct 17, 9:32 PM
epriestley closed D18712: Clarify acceptable values for `--threshold` in `search ngrams`.
Tue, Oct 17, 9:32 PM
epriestley added a comment to D18712: Clarify acceptable values for `--threshold` in `search ngrams`.

This is how I pad out my experience points for Hall of Heroes.

Tue, Oct 17, 9:32 PM
epriestley committed rARC617c2e46d6a5: Make "line" and "char" strictly optional in ArcanistLintMessage (authored by epriestley).
Make "line" and "char" strictly optional in ArcanistLintMessage
Tue, Oct 17, 9:30 PM
epriestley closed D18711: Make "line" and "char" strictly optional in ArcanistLintMessage.
Tue, Oct 17, 9:30 PM
epriestley created D18712: Clarify acceptable values for `--threshold` in `search ngrams`.
Tue, Oct 17, 9:29 PM
epriestley created D18711: Make "line" and "char" strictly optional in ArcanistLintMessage.
Tue, Oct 17, 9:23 PM
epriestley committed rP63d1230ade0f: Parameterize the common ngrams threshold (authored by epriestley).
Parameterize the common ngrams threshold
Tue, Oct 17, 9:14 PM
epriestley closed D18710: Parameterize the common ngrams threshold.
Tue, Oct 17, 9:13 PM
epriestley added a commit to T13000: Sustained MySQL I/O overwhelmed db009 / huge Ferret engine ngrams table: rP63d1230ade0f: Parameterize the common ngrams threshold.
Tue, Oct 17, 9:13 PM · Ops, Phacility
epriestley added a revision to T13000: Sustained MySQL I/O overwhelmed db009 / huge Ferret engine ngrams table: D18710: Parameterize the common ngrams threshold.
Tue, Oct 17, 9:00 PM · Ops, Phacility
epriestley created D18710: Parameterize the common ngrams threshold.
Tue, Oct 17, 9:00 PM
epriestley removed a member for GitHub: worldkingma.
Tue, Oct 17, 1:19 PM

Mon, Oct 16

epriestley lowered the priority of T13008: Process slot exhaustion in Phacility web tier from Unbreak Now! to Normal.

I'm going to call this tentatively done and leave it open for a week or two to see if there's any fallout.

Mon, Oct 16, 3:55 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

Something is making a lot of requests

Mon, Oct 16, 3:54 PM · Ops, Phacility
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: Unknown Object (Diffusion Commit).
Mon, Oct 16, 3:39 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

The behavior of mod_reqtimeout is weird in production. Some of this is the LB, but some of it I don't have a good explanation for.

Mon, Oct 16, 3:38 PM · Ops, Phacility
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: Unknown Object (Diffusion Commit).
Mon, Oct 16, 3:10 PM · Ops, Phacility
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: Unknown Object (Diffusion Commit).
Mon, Oct 16, 3:04 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

I was able to get web001 to wedge a lot of requests in "R" ("Reading Request") -- note "SS" is "seconds since beginning of most recent request".

Mon, Oct 16, 2:55 PM · Ops, Phacility
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: Unknown Object (Diffusion Commit).
Mon, Oct 16, 2:37 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

I'm also unable to wedge secure. mod_status is working there, but since I can't wedge it it isn't very useful.

Mon, Oct 16, 2:34 PM · Ops, Phacility
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: Unknown Object (Diffusion Commit).
Mon, Oct 16, 2:23 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

I'm going to take a more detailed look at local behavior for this specific request pattern.

Mon, Oct 16, 2:19 PM · Ops, Phacility
epriestley committed rP819b833607a6: Tweak rate limiting point counts for omnipotent users (authored by epriestley).
Tweak rate limiting point counts for omnipotent users
Mon, Oct 16, 1:44 PM
epriestley closed D18708: Tweak rate limiting point counts for omnipotent users.
Mon, Oct 16, 1:43 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rP819b833607a6: Tweak rate limiting point counts for omnipotent users.
Mon, Oct 16, 1:43 PM · Ops, Phacility

Sat, Oct 14

epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

This is working somewhat better than before, but I'm still able to wedge apache with enough requests using a pre-patch version of libphutil. I'm going to take a more detailed look at local behavior for this specific request pattern.

Sat, Oct 14, 4:06 PM · Ops, Phacility
epriestley created D18708: Tweak rate limiting point counts for omnipotent users.
Sat, Oct 14, 3:00 PM
epriestley added a revision to T13008: Process slot exhaustion in Phacility web tier: D18708: Tweak rate limiting point counts for omnipotent users.
Sat, Oct 14, 3:00 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

It's currently possible for cluster hosts to hit rate limits against admin while doing normal deployment stuff, since they may make a lot of requests to admin very quickly.

Sat, Oct 14, 2:45 PM · Ops, Phacility
epriestley added a comment to T13009: MySQL may take several seconds after restart to begin listening on domain socket.

I expect I can just reduce the data size to something manageable with the current workflow fairly easily

Sat, Oct 14, 2:29 PM · Ops, Phacility
epriestley committed rPe21d603356fc: (stable) Disconnect rate limits in the PhabricatorStartup shutdown handler (authored by epriestley).
(stable) Disconnect rate limits in the PhabricatorStartup shutdown handler
Sat, Oct 14, 2:15 PM
epriestley committed rP0e645b8f11d2: Disconnect rate limits in the PhabricatorStartup shutdown handler (authored by epriestley).
Disconnect rate limits in the PhabricatorStartup shutdown handler
Sat, Oct 14, 2:15 PM
epriestley created T13009: MySQL may take several seconds after restart to begin listening on domain socket.
Sat, Oct 14, 12:28 PM · Ops, Phacility

Fri, Oct 13

epriestley edited the content of Changelog.
Fri, Oct 13, 9:50 PM
epriestley committed rPHU7a90825f14b2: (stable) Promote 2017 Week 41 (authored by epriestley).
(stable) Promote 2017 Week 41
Fri, Oct 13, 9:50 PM
epriestley committed rP9365c6c99928: (stable) Promote 2017 Week 41 (authored by epriestley).
(stable) Promote 2017 Week 41
Fri, Oct 13, 9:50 PM
epriestley created 2017 Week 41 (Mid October).
Fri, Oct 13, 9:49 PM
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

(I also verified that real multipart/form-data works fine from a browser, by changing a project's profile picture here.)

Fri, Oct 13, 9:40 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

On the server die, the connection test looks better: secure001 saw 8 HTTP 500's (this might be a bug in the multipart/form-data parser for whatever cURL is doing) and then 135 HTTP 429s, which is more or less exactly what the code is supposed to do. So I think most of the inconclusiveness on what was happening from the client side was curl / nohup stuff.

Fri, Oct 13, 9:39 PM · Ops, Phacility
epriestley committed rPc5e8de945020: Make `bin/storage dump` insert CREATE DATABASE and USE statements (authored by epriestley).
Make `bin/storage dump` insert CREATE DATABASE and USE statements
Fri, Oct 13, 9:35 PM
epriestley added a commit to T13000: Sustained MySQL I/O overwhelmed db009 / huge Ferret engine ngrams table: rPc5e8de945020: Make `bin/storage dump` insert CREATE DATABASE and USE statements.
Fri, Oct 13, 9:35 PM · Ops, Phacility
epriestley closed D18707: Make `bin/storage dump` insert CREATE DATABASE and USE statements.
Fri, Oct 13, 9:35 PM
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

I used ab to verify the rate limiting code here, and got killed after a bit of abuse with ab -c 10 -n 200 ...:

Fri, Oct 13, 9:25 PM · Ops, Phacility
epriestley set the image for Aphront to F5221631: profile.
Fri, Oct 13, 9:13 PM
epriestley added a revision to T13000: Sustained MySQL I/O overwhelmed db009 / huge Ferret engine ngrams table: D18707: Make `bin/storage dump` insert CREATE DATABASE and USE statements.
Fri, Oct 13, 8:53 PM · Ops, Phacility
epriestley created D18707: Make `bin/storage dump` insert CREATE DATABASE and USE statements.
Fri, Oct 13, 8:53 PM
epriestley added a commit to T13003: `admin.phacility.com` is receiving a huge volume of "leafweb" traffic: Unknown Object (Diffusion Commit).
Fri, Oct 13, 8:13 PM · Ops, Phacility
epriestley committed rPacb145b3d7ad: Allow duplicates and merged-in tasks to be queried with `edge.search` (authored by epriestley).
Allow duplicates and merged-in tasks to be queried with `edge.search`
Fri, Oct 13, 8:13 PM
epriestley closed D18706: Allow duplicates and merged-in tasks to be queried with `edge.search`.
Fri, Oct 13, 8:12 PM
epriestley committed rP3f53718d107f: Modularize rate/connection limits in Phabricator (authored by epriestley).
Modularize rate/connection limits in Phabricator
Fri, Oct 13, 8:12 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rP3f53718d107f: Modularize rate/connection limits in Phabricator.
Fri, Oct 13, 8:12 PM · Ops, Phacility
epriestley closed D18703: Modularize rate/connection limits in Phabricator.
Fri, Oct 13, 8:12 PM
epriestley committed rP9777c66576ff: Allow Phabricator to run with "enable_post_data_reading" disabled (authored by epriestley).
Allow Phabricator to run with "enable_post_data_reading" disabled
Fri, Oct 13, 8:11 PM
epriestley closed D18702: Allow Phabricator to run with "enable_post_data_reading" disabled.
Fri, Oct 13, 8:11 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rP9777c66576ff: Allow Phabricator to run with "enable_post_data_reading" disabled.
Fri, Oct 13, 8:11 PM · Ops, Phacility
epriestley updated the diff for D18702: Allow Phabricator to run with "enable_post_data_reading" disabled.
  • Rewrite explanatory comment.
Fri, Oct 13, 8:05 PM
epriestley added a comment to D18703: Modularize rate/connection limits in Phabricator.

The previous version of this code has been active in production on secure since D17761, and is currently active, and we haven't seen issues. (You can use ConfigCache Status to review APCu usage. Whichever secure host I just hit is using 2MB, which suggests there's no leak.)

Fri, Oct 13, 8:03 PM
epriestley removed edges from rPHUa6e490ef7ee0: Fix a bug which prevented Conduit futures from having parallelism limited…: D17607: Part 2, D17749: Fixes T4815.
Fri, Oct 13, 2:43 PM
epriestley removed 1 commit(s) for D17607: Part 2: rPHUa6e490ef7ee0: Fix a bug which prevented Conduit futures from having parallelism limited….
Fri, Oct 13, 2:43 PM
epriestley removed 1 commit(s) for D17749: Fixes T4815: rPHUa6e490ef7ee0: Fix a bug which prevented Conduit futures from having parallelism limited….
Fri, Oct 13, 2:43 PM

Thu, Oct 12

epriestley updated the diff for D18706: Allow duplicates and merged-in tasks to be queried with `edge.search`.
  • Rebase on top of a cleaner build.
Thu, Oct 12, 11:43 PM
epriestley updated the diff for D18703: Modularize rate/connection limits in Phabricator.
  • Fix unit test path to PhabricatorStartup.php.
Thu, Oct 12, 11:33 PM
epriestley created D18706: Allow duplicates and merged-in tasks to be queried with `edge.search`.
Thu, Oct 12, 11:29 PM
epriestley added a revision to T13003: `admin.phacility.com` is receiving a huge volume of "leafweb" traffic: Unknown Object (Differential Revision).
Thu, Oct 12, 10:59 PM · Ops, Phacility
epriestley committed rPHUa6e490ef7ee0: Fix a bug which prevented Conduit futures from having parallelism limited… (authored by epriestley).
Fix a bug which prevented Conduit futures from having parallelism limited…
Thu, Oct 12, 7:37 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rPHUa6e490ef7ee0: Fix a bug which prevented Conduit futures from having parallelism limited….
Thu, Oct 12, 7:37 PM · Ops, Phacility
epriestley closed D18704: Fix a bug which prevented Conduit futures from having parallelism limited effectively.
Thu, Oct 12, 7:37 PM
epriestley committed rPHU260dcbd6b093: Stream "multipart/form-data" file uploads to disk (authored by epriestley).
Stream "multipart/form-data" file uploads to disk
Thu, Oct 12, 7:36 PM
epriestley closed D18701: Stream "multipart/form-data" file uploads to disk.
Thu, Oct 12, 7:36 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rPHU260dcbd6b093: Stream "multipart/form-data" file uploads to disk.
Thu, Oct 12, 7:36 PM · Ops, Phacility
epriestley committed rPHUf46b445bdc5b: Add a rough "multipart/form-data" stream parser (authored by epriestley).
Add a rough "multipart/form-data" stream parser
Thu, Oct 12, 7:34 PM
epriestley closed D18700: Add a rough "multipart/form-data" stream parser.
Thu, Oct 12, 7:34 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rPHUf46b445bdc5b: Add a rough "multipart/form-data" stream parser.
Thu, Oct 12, 7:34 PM · Ops, Phacility
epriestley committed rPHU68324a95db7e: Add a rough HTTP header value parser (authored by epriestley).
Add a rough HTTP header value parser
Thu, Oct 12, 7:34 PM
epriestley closed D18699: Add a rough HTTP header value parser.
Thu, Oct 12, 7:34 PM
epriestley added a commit to T13008: Process slot exhaustion in Phacility web tier: rPHU68324a95db7e: Add a rough HTTP header value parser.
Thu, Oct 12, 7:34 PM · Ops, Phacility
epriestley created D18704: Fix a bug which prevented Conduit futures from having parallelism limited effectively.
Thu, Oct 12, 7:18 PM
epriestley added a revision to T13008: Process slot exhaustion in Phacility web tier: D18704: Fix a bug which prevented Conduit futures from having parallelism limited effectively.
Thu, Oct 12, 7:18 PM · Ops, Phacility
epriestley added a comment to T13008: Process slot exhaustion in Phacility web tier.

There's also a bug with arc here. It's supposed to limit simultaneous uploads to 4:

Thu, Oct 12, 7:04 PM · Ops, Phacility
epriestley added a comment to T12857: Temporary directory fullness can cause daemon issues?.

Tangentially related here and to T12611, the traffic volume from the ongoing "attack" in T13003 filled /tmp on admin.phacility.com. I pruned some old logs for now, but the mitigations in T12611 (e.g., separate log volumes) would resolve this properly.

Thu, Oct 12, 3:38 PM · Diffusion, Ops, Daemons, Phacility

Wed, Oct 11

epriestley created D18703: Modularize rate/connection limits in Phabricator.
Wed, Oct 11, 11:39 PM
epriestley added a dependent revision for D18702: Allow Phabricator to run with "enable_post_data_reading" disabled: D18703: Modularize rate/connection limits in Phabricator.
Wed, Oct 11, 11:39 PM
epriestley added a revision to T13008: Process slot exhaustion in Phacility web tier: D18703: Modularize rate/connection limits in Phabricator.
Wed, Oct 11, 11:39 PM · Ops, Phacility
epriestley added a comment to D18701: Stream "multipart/form-data" file uploads to disk.

Yeah, the default behavior for PHP is to write multipart/form-data uploads to tempfiles on disk.

Wed, Oct 11, 8:48 PM
epriestley committed rP821c7ac8334f: For backup persitsence, mark the "common ngrams" table as a data table, not an… (authored by epriestley).
For backup persitsence, mark the "common ngrams" table as a data table, not an…
Wed, Oct 11, 6:07 PM
epriestley added a commit to T13000: Sustained MySQL I/O overwhelmed db009 / huge Ferret engine ngrams table: rP821c7ac8334f: For backup persitsence, mark the "common ngrams" table as a data table, not an….
Wed, Oct 11, 6:07 PM · Ops, Phacility
epriestley closed D18696: For backup persitsence, mark the "common ngrams" table as a data table, not an index table.
Wed, Oct 11, 6:07 PM

Tue, Oct 10

epriestley created D18702: Allow Phabricator to run with "enable_post_data_reading" disabled.
Tue, Oct 10, 10:26 PM
epriestley added a dependent revision for D18701: Stream "multipart/form-data" file uploads to disk: D18702: Allow Phabricator to run with "enable_post_data_reading" disabled.
Tue, Oct 10, 10:26 PM
epriestley added a revision to T13008: Process slot exhaustion in Phacility web tier: D18702: Allow Phabricator to run with "enable_post_data_reading" disabled.
Tue, Oct 10, 10:26 PM · Ops, Phacility
epriestley added a revision to T13008: Process slot exhaustion in Phacility web tier: D18701: Stream "multipart/form-data" file uploads to disk.
Tue, Oct 10, 10:17 PM · Ops, Phacility
epriestley updated the summary of D18701: Stream "multipart/form-data" file uploads to disk.
Tue, Oct 10, 10:17 PM
epriestley created D18701: Stream "multipart/form-data" file uploads to disk.
Tue, Oct 10, 10:16 PM
epriestley added a dependent revision for D18700: Add a rough "multipart/form-data" stream parser: D18701: Stream "multipart/form-data" file uploads to disk.
Tue, Oct 10, 10:16 PM
epriestley created D18700: Add a rough "multipart/form-data" stream parser.
Tue, Oct 10, 8:36 PM
epriestley added a revision to T13008: Process slot exhaustion in Phacility web tier: D18700: Add a rough "multipart/form-data" stream parser.
Tue, Oct 10, 8:36 PM · Ops, Phacility
epriestley added a dependent revision for D18699: Add a rough HTTP header value parser: D18700: Add a rough "multipart/form-data" stream parser.
Tue, Oct 10, 8:36 PM
epriestley added a comment to D18699: Add a rough HTTP header value parser.

For cases like Animal: zebra we'll end up in state "key" (since we never saw a ; or =), which is okay.

Tue, Oct 10, 8:32 PM