Write Phabricator HTTP and SSH logs in the production cluster
Open, NormalPublic
Actions

Assigned To

None

Authored By

	epriestley
	Apr 20 2017, 9:20 PM

Description

See discussion in T12605. I want to configure log.access.path and log.ssh.path so we generate meaningful logs in production.

I identified the problem instance and user in T12605 through huge leaps of guesswork which no one who is less experienced could reasonably have been expected to perform, and still got lucky. Configuring these will let anyone plainly read values out of sensible logfiles and skip all this guesswork.

Revisions and Commits

	Restricted Differential Revision	Restricted Diffusion Commit
	Restricted Differential Revision	Restricted Diffusion Commit
	Restricted Differential Revision	Restricted Diffusion Commit
	Restricted Differential Revision	Restricted Diffusion Commit
	Restricted Differential Revision	Restricted Diffusion Commit
rPHU libphutil
	D17777	rPHU904f81a31b74 Make "fail quietly" flag in DeferredLog a little more quiet
rP Phabricator
	D17776	rPd0e6bf831d6a Add "%I" (instance name) to application log formats

Related Objects
Search...

		Status	Assigned	Task
				Unknown Object (Maniphest Task)
		Open	None	T12611 Write Phabricator HTTP and SSH logs in the production cluster

Event Timeline

epriestley created this task.Apr 20 2017, 9:20 PM

epriestley created this object in space Restricted Space.

epriestley created this object with edit policy "All Users".

I'm going to declassify this too since there's nothing private.

epriestley shifted this object from the Restricted Space space to the S1 Core space.Apr 21 2017, 7:21 PM

Herald added a subscriber: eadler. · View Herald TranscriptApr 21 2017, 7:21 PM

epriestley added a revision: D17776: Add "%I" (instance name) to application log formats.Apr 23 2017, 2:35 PM

epriestley added a revision: D17777: Make "fail quietly" flag in DeferredLog a little more quiet.

epriestley added a revision: Restricted Differential Revision.Apr 23 2017, 2:47 PM

epriestley added a revision: Restricted Differential Revision.Apr 23 2017, 2:57 PM

epriestley added a commit: rPHU904f81a31b74: Make "fail quietly" flag in DeferredLog a little more quiet.Apr 23 2017, 6:01 PM

epriestley added a commit: rPd0e6bf831d6a: Add "%I" (instance name) to application log formats.Apr 23 2017, 6:07 PM

epriestley added a commit: Restricted Diffusion Commit.Apr 23 2017, 6:27 PM

epriestley added a commit: Restricted Diffusion Commit.Apr 23 2017, 6:34 PM

The Aphlict logs should probably get consolidated here too, they're set up a little weird and don't currently rotate during normal deployments.

Specifically, I'd like to make these changes:

Put the Aphlict log in log/ with the same organization as everything else, and have it share the same rotation code. It would currently conflict with a hypothetical instance named aphlict.phaclity.com.
Put log volumes on more devices. admin, web and notify-class devices currently serve logged traffic but do not have dedicated log volumes.
- bin/provision should be aware of these volumes, and ideally should be used to retroactively provision and attach them.
Enable the SSH and HTTP application logs on the web, repo and admin tiers.
Move Apache logs to log/, so log accumulation is less able to endanger other processes by filling up disks.
Probably put them on the same rotation infrastructure as everything else, although logrotate currently interacts here.
Put notify in the regular push rotation. It currently gets deployed irregularly.