Page MenuHomePhabricator

Transaction edge storage is inefficient / an instance has 130GB of commit cross-refs
Closed, ResolvedPublic

Description

When we write an edge transaction to the database, we currently write the entire old and new edge lists into the record.

For example, if the old project list was "A, B, C, D" and the new project list is "A, C, D, E", we write the complete lists, for a total of 8 PHIDs. Today, all readers compute changes from the lists anyway, so writing "A", "C" and "D" to both lists has no effect. The transaction has identical behavior if we write just "B" and "E", respectively -- it will still render as "alice changed projects; added: E; removed: B.".

This doesn't seem like it should be a very big deal since we're just talking about a handful of extra PHIDs, but some repositories do merges in a way that causes the same commits to be mentioned hundreds and hundreds of times. When we add the 101st mention, we're writing 201 PHIDs. When we add the 1001st mention, we're writing 2001 PHIDs. We have one hosted instance with approximately 130GB of "commit X mentioned commit Y" data in the repository_audit.audit_transaction table because of this.

(This is a test instance which is importing an especially large, well-known repository so no actual work is directly impacted, although other instances on the same shard may be suffering.)

This is also made worse because the actual storage format is very verbose:

"PHID-CMIT-xxx": {
  "src":"PHID-CMIT-yyy",
  "type":"51",
  "dst":"PHID-CMIT-xxx",
  "dateCreated":"123456",
  "seq":"0",
  "dataID":null,
  "data":[]
}
  • We no longer use dataID or data, and these can be removed.
  • I think there is no value in writing seq. We don't rely on edge ordering and don't support reordering. If we did in the future, we could treat "no seq data present means it was added at the end" safely.
  • There is no value in writing dateCreated, since this is always the same as the transaction date.
  • The type is always the same for all transactions in a group, and always present in edge:type metadata.
  • src is always the object PHID.
  • dst is always the dictionary key (the other end of the edge).

So I think we can write something like this record instead:

{
  "dst": [...]
}

...where the list is just modified PHIDs. This could probably even just be [...] but if we wrap it in {"dst":[...]} we might be a little more future-proof.

I'm planning to:

  • Put a translation layer into the edge transaction handling.
  • Define all reads in terms of the translation layer; get test coverage on the reads.
  • Add support for a new compact format.
  • Start writing the compact format.
  • Build a tool for compacting old rows and run it in production.

  • I removed a piece of code which hid very old metadata-only edge transactions in Differential (they used to be written as a side effect of accepting a revision). Any eventual migration should probably just delete these transactions since they've never had any user-visible effect (when written, they were redundant with associated "accepted this revision" transactions).

Event Timeline

epriestley triaged this task as Normal priority.Jan 27 2018, 12:59 PM
epriestley created this task.
epriestley updated the task description. (Show Details)Jan 27 2018, 1:03 PM
epriestley updated the task description. (Show Details)Jan 27 2018, 1:21 PM
epriestley added a comment.EditedJan 27 2018, 1:25 PM

Bad news: data still has one reader/writer in the Asana-to-Revision linking implementation. So we can't completely get rid of that yet.

My plan is to pick those to stable, then compact-edges here on secure, then compact-edges on the affected 130GB instance. There's some value in doing this sooner rather than later because the backups for 130GB of edge data are having some issues. The instance is a free test instance so this isn't a huge concern, but I'd sleep better if it was running smoothly. If you don't run compact-edges I think the worst those changes could really do is cause some kind of temporary display bug with new transactions, so the risk should be pretty small.

After that, I'll see if I can clean things up a bit more for the actual release.

(Pushing this to secure, stuff might be funky for a minute while I gently massage the database.)

Editing some edges on the new code as a sanity check before I compact things.

I double checked that our backups are working.

I ran bin/storage optimize, on the secure masters and got this storage probe output:

secure001 probe
ubuntu@secure001:/core$ PHABRICATOR_INSTANCE=secure ./lib/phabricator/bin/storage probe --host secure001.phacility.net
Analyzing table sizes (this may take a moment)...
secure_phlux                                          0.1 MB  0.0%
    phlux_variable                                    0.0 MB  0.0%
    phlux_transaction                                 0.0 MB  0.0%
secure_policy                                         0.1 MB  0.0%
    policy                                            0.1 MB  0.0%
secure_phrequent                                      0.1 MB  0.0%
    phrequent_usertime                                0.1 MB  0.0%
secure_doorkeeper                                     0.1 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    doorkeeper_externalobject                         0.1 MB  0.0%
secure_application                                    0.1 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    application_application                           0.0 MB  0.0%
    application_applicationtransaction                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_meta_data                                      0.1 MB  0.0%
    hoststate                                         0.0 MB  0.0%
    patch_status                                      0.1 MB  0.0%
secure_spaces                                         0.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    spaces_namespacetransaction                       0.0 MB  0.0%
    spaces_namespace                                  0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_phragment                                      0.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phragment_snapshotchild                           0.0 MB  0.0%
    phragment_snapshot                                0.0 MB  0.0%
    phragment_fragment                                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phragment_fragmentversion                         0.0 MB  0.0%
secure_config                                         0.3 MB  0.0%
    config_manualactivity                             0.0 MB  0.0%
    config_entry                                      0.0 MB  0.0%
    config_transaction                                0.2 MB  0.0%
secure_countdown                                      0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    countdown_transaction_comment                     0.0 MB  0.0%
    countdown_transaction                             0.1 MB  0.0%
    countdown                                         0.1 MB  0.0%
secure_oauth_server                                   0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    oauth_server_oauthserveraccesstoken               0.0 MB  0.0%
    oauth_server_oauthserverclient                    0.0 MB  0.0%
    oauth_server_oauthclientauthorization             0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    oauth_server_transaction                          0.0 MB  0.0%
    oauth_server_oauthserverauthorizationcode         0.1 MB  0.0%
secure_phurl                                          0.4 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phurl_urltransaction_comment                      0.0 MB  0.0%
    phurl_phurlname_ngrams                            0.1 MB  0.0%
    phurl_url                                         0.1 MB  0.0%
    phurl_urltransaction                              0.1 MB  0.0%
secure_releeph                                        0.4 MB  0.0%
    releeph_requesttransaction_comment                0.0 MB  0.0%
    releeph_requesttransaction                        0.0 MB  0.0%
    releeph_project                                   0.0 MB  0.0%
    releeph_branchtransaction                         0.0 MB  0.0%
    releeph_producttransaction                        0.0 MB  0.0%
    releeph_request                                   0.1 MB  0.0%
    releeph_branch                                    0.1 MB  0.0%
secure_slowvote                                       0.4 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    slowvote_option                                   0.0 MB  0.0%
    slowvote_transaction_comment                      0.0 MB  0.0%
    slowvote_poll                                     0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    slowvote_transaction                              0.1 MB  0.0%
    slowvote_choice                                   0.1 MB  0.0%
secure_packages                                       0.5 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    packages_publishertransaction                     0.0 MB  0.0%
    packages_version                                  0.0 MB  0.0%
    packages_versiontransaction                       0.0 MB  0.0%
    packages_publishername_ngrams                     0.0 MB  0.0%
    packages_versionname_ngrams                       0.0 MB  0.0%
    packages_publisher                                0.0 MB  0.0%
    packages_package                                  0.0 MB  0.0%
    packages_packagename_ngrams                       0.0 MB  0.0%
    packages_packagetransaction                       0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_xhprof                                         0.5 MB  0.0%
    xhprof_sample                                     0.5 MB  0.0%
secure_owners                                         0.7 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    owners_package_ffield                             0.0 MB  0.0%
    owners_package                                    0.0 MB  0.0%
    owners_customfieldstorage                         0.0 MB  0.0%
    owners_path                                       0.0 MB  0.0%
    owners_package_fngrams                            0.0 MB  0.0%
    owners_package_fngrams_common                     0.0 MB  0.0%
    owners_owner                                      0.0 MB  0.0%
    owners_customfieldnumericindex                    0.0 MB  0.0%
    owners_customfieldstringindex                     0.0 MB  0.0%
    owners_name_ngrams                                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    owners_package_fdocument                          0.1 MB  0.0%
    owners_packagetransaction                         0.1 MB  0.0%
secure_nuance                                         0.7 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    nuance_itemcommand                                0.0 MB  0.0%
    nuance_queue                                      0.0 MB  0.0%
    nuance_sourcename_ngrams                          0.0 MB  0.0%
    nuance_source                                     0.0 MB  0.0%
    nuance_sourcetransaction                          0.0 MB  0.0%
    nuance_sourcetransaction_comment                  0.0 MB  0.0%
    nuance_queuetransaction_comment                   0.0 MB  0.0%
    nuance_itemtransaction_comment                    0.0 MB  0.0%
    nuance_importcursordata                           0.0 MB  0.0%
    nuance_queuetransaction                           0.0 MB  0.0%
    nuance_itemtransaction                            0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    nuance_item                                       0.1 MB  0.0%
secure_passphrase                                     0.7 MB  0.0%
    passphrase_secret                                 0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    passphrase_credential_fngrams_common              0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    passphrase_credential_ffield                      0.1 MB  0.0%
    passphrase_credential                             0.1 MB  0.0%
    passphrase_credential_fdocument                   0.1 MB  0.0%
    passphrase_credentialtransaction                  0.1 MB  0.0%
    passphrase_credential_fngrams                     0.2 MB  0.0%
secure_fund                                           0.7 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    fund_initiative_ffield                            0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    fund_initiative_fngrams_common                    0.0 MB  0.0%
    fund_initiativetransaction_comment                0.0 MB  0.0%
    fund_initiative_fngrams                           0.0 MB  0.0%
    fund_initiative                                   0.1 MB  0.0%
    fund_initiative_fdocument                         0.1 MB  0.0%
    fund_backer                                       0.1 MB  0.0%
    fund_initiativetransaction                        0.1 MB  0.0%
    fund_backertransaction                            0.1 MB  0.0%
secure_flag                                           0.7 MB  0.0%
    flag                                              0.7 MB  0.0%
secure_almanac                                        0.9 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    almanac_network                                   0.0 MB  0.0%
    almanac_property                                  0.0 MB  0.0%
    almanac_networktransaction                        0.0 MB  0.0%
    almanac_networkname_ngrams                        0.0 MB  0.0%
    almanac_servicetransaction                        0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    almanac_servicename_ngrams                        0.0 MB  0.0%
    almanac_namespacetransaction                      0.0 MB  0.0%
    almanac_devicetransaction                         0.0 MB  0.0%
    almanac_devicename_ngrams                         0.0 MB  0.0%
    almanac_namespacename_ngrams                      0.0 MB  0.0%
    almanac_bindingtransaction                        0.0 MB  0.0%
    almanac_interface                                 0.1 MB  0.0%
    almanac_namespace                                 0.1 MB  0.0%
    almanac_device                                    0.1 MB  0.0%
    almanac_service                                   0.1 MB  0.0%
    almanac_binding                                   0.1 MB  0.0%
secure_token                                          1.8 MB  0.0%
    token_token                                       0.1 MB  0.0%
    token_count                                       0.4 MB  0.0%
    token_given                                       1.4 MB  0.0%
secure_badges                                         1.8 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    badges_transaction_comment                        0.0 MB  0.0%
    badges_badgename_ngrams                           0.0 MB  0.0%
    badges_badge                                      0.0 MB  0.0%
    badges_award                                      0.1 MB  0.0%
    badges_transaction                                1.5 MB  0.0%
secure_legalpad                                       2.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    legalpad_transaction_comment                      0.1 MB  0.0%
    legalpad_document                                 0.1 MB  0.0%
    edge                                              0.1 MB  0.0%
    legalpad_documentbody                             0.2 MB  0.0%
    legalpad_documentsignature                        0.2 MB  0.0%
    legalpad_transaction                              1.5 MB  0.0%
secure_phortune                                       3.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phortune_merchant                                 0.0 MB  0.0%
    phortune_paymentproviderconfigtransaction         0.0 MB  0.0%
    phortune_product                                  0.0 MB  0.0%
    phortune_paymentproviderconfig                    0.0 MB  0.0%
    phortune_merchanttransaction                      0.0 MB  0.0%
    phortune_paymentmethod                            0.1 MB  0.0%
    phortune_subscription                             0.1 MB  0.0%
    phortune_charge                                   0.1 MB  0.0%
    phortune_purchase                                 0.1 MB  0.0%
    phortune_carttransaction                          0.1 MB  0.0%
    phortune_cart                                     0.1 MB  0.0%
    phortune_account                                  0.2 MB  0.0%
    edge                                              0.4 MB  0.0%
    phortune_accounttransaction                       1.8 MB  0.0%
secure_auth                                           4.8 MB  0.0%
    auth_hmackey                                      0.0 MB  0.0%
    auth_passwordtransaction                          0.0 MB  0.0%
    auth_providerconfigtransaction                    0.0 MB  0.0%
    auth_factorconfig                                 0.0 MB  0.0%
    auth_temporarytoken                               0.1 MB  0.0%
    auth_providerconfig                               0.1 MB  0.0%
    auth_sshkeytransaction                            0.3 MB  0.0%
    auth_sshkey                                       1.8 MB  0.0%
    auth_password                                     2.4 MB  0.0%
secure_pastebin                                       5.5 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    pastebin_pastetransaction_comment                 0.3 MB  0.0%
    edge                                              0.8 MB  0.0%
    pastebin_paste                                    1.0 MB  0.0%
    pastebin_pastetransaction                         3.3 MB  0.0%
secure_dashboard                                      8.6 MB  0.1%
    edgedata                                          0.0 MB  0.0%
    dashboard_install                                 0.0 MB  0.0%
    dashboard                                         0.1 MB  0.0%
    dashboard_panel                                   0.3 MB  0.0%
    dashboard_dashboard_ngrams                        0.4 MB  0.0%
    edge                                              0.6 MB  0.0%
    dashboard_dashboardpanel_ngrams                   0.7 MB  0.0%
    dashboard_transaction                             1.7 MB  0.0%
    dashboard_paneltransaction                        4.8 MB  0.0%
secure_phame                                          9.8 MB  0.1%
    edgedata                                          0.0 MB  0.0%
    phame_blog_ffield                                 0.0 MB  0.0%
    phame_blog_fngrams_common                         0.0 MB  0.0%
    phame_post_fngrams_common                         0.0 MB  0.0%
    phame_blog                                        0.0 MB  0.0%
    phame_blog_fngrams                                0.0 MB  0.0%
    phame_blog_fdocument                              0.1 MB  0.0%
    phame_posttransaction_comment                     0.1 MB  0.0%
    phame_post_fdocument                              0.1 MB  0.0%
    phame_blogtransaction                             0.1 MB  0.0%
    edge                                              0.2 MB  0.0%
    phame_post                                        0.3 MB  0.0%
    phame_posttransaction                             1.6 MB  0.0%
    phame_post_ffield                                 2.5 MB  0.0%
    phame_post_fngrams                                4.5 MB  0.0%
secure_pholio                                        11.4 MB  0.1%
    edgedata                                          0.0 MB  0.0%
    pholio_mock_fngrams_common                        0.0 MB  0.0%
    pholio_mock_fdocument                             0.1 MB  0.0%
    pholio_mock                                       0.2 MB  0.0%
    pholio_image                                      0.2 MB  0.0%
    edge                                              0.3 MB  0.0%
    pholio_transaction_comment                        0.6 MB  0.0%
    pholio_transaction                                1.8 MB  0.0%
    pholio_mock_ffield                                2.6 MB  0.0%
    pholio_mock_fngrams                               5.5 MB  0.0%
secure_ponder                                        14.7 MB  0.1%
    ponder_question_fngrams_common                    0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    ponder_question_fdocument                         0.1 MB  0.0%
    ponder_answertransaction_comment                  0.4 MB  0.0%
    ponder_questiontransaction_comment                0.5 MB  0.0%
    ponder_answer                                     0.5 MB  0.0%
    edge                                              0.5 MB  0.0%
    ponder_question_ffield                            1.5 MB  0.0%
    ponder_question                                   1.7 MB  0.0%
    ponder_answertransaction                          1.8 MB  0.0%
    ponder_questiontransaction                        3.2 MB  0.0%
    ponder_question_fngrams                           4.5 MB  0.0%
secure_drydock                                       15.0 MB  0.1%
    edgedata                                          0.0 MB  0.0%
    drydock_blueprint                                 0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    drydock_blueprintname_ngrams                      0.0 MB  0.0%
    drydock_slotlock                                  0.0 MB  0.0%
    drydock_blueprinttransaction                      0.0 MB  0.0%
    drydock_authorization                             0.1 MB  0.0%
    drydock_resource                                  0.1 MB  0.0%
    drydock_repositoryoperation                       0.2 MB  0.0%
    drydock_log                                       0.6 MB  0.0%
    drydock_command                                   3.0 MB  0.0%
    drydock_lease                                    10.8 MB  0.1%
secure_system                                        16.2 MB  0.1%
    system_actionlog                                  0.2 MB  0.0%
    system_destructionlog                            16.0 MB  0.1%
secure_xhpast                                        23.5 MB  0.2%
    xhpast_parsetree                                 23.5 MB  0.2%
secure_daemon                                        26.6 MB  0.2%
    daemon_log                                        0.2 MB  0.0%
    daemon_logevent                                  26.4 MB  0.2%
secure_worker                                        26.6 MB  0.2%
    lisk_counter                                      0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    worker_bulkjobtransaction                         0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    worker_triggerevent                               0.0 MB  0.0%
    worker_bulktask                                   0.0 MB  0.0%
    worker_trigger                                    0.0 MB  0.0%
    worker_bulkjob                                    0.1 MB  0.0%
    worker_activetask                                 0.1 MB  0.0%
    worker_taskdata                                   9.5 MB  0.1%
    worker_archivetask                               16.6 MB  0.1%
secure_fact                                          31.2 MB  0.2%
    fact_aggregate                                    0.0 MB  0.0%
    fact_cursor                                       0.0 MB  0.0%
    fact_raw                                         31.1 MB  0.2%
secure_chatlog                                       36.1 MB  0.3%
    chatlog_channel                                   0.0 MB  0.0%
    chatlog_event                                    36.1 MB  0.3%
secure_audit                                         42.6 MB  0.3%
    audit_transaction_comment                         1.0 MB  0.0%
    audit_transaction                                41.6 MB  0.3%
secure_draft                                         58.2 MB  0.4%
    draft                                            19.1 MB  0.1%
    draft_versioneddraft                             39.1 MB  0.3%
secure_project                                       61.2 MB  0.4%
    project_customfieldnumericindex                   0.0 MB  0.0%
    project_customfieldstringindex                    0.0 MB  0.0%
    project_project_fngrams_common                    0.0 MB  0.0%
    project_slug                                      0.1 MB  0.0%
    project_datasourcetoken                           0.1 MB  0.0%
    project_project_fdocument                         0.1 MB  0.0%
    project_columntransaction                         0.2 MB  0.0%
    project                                           0.2 MB  0.0%
    project_customfieldstorage                        0.2 MB  0.0%
    project_column                                    0.3 MB  0.0%
    project_project_ffield                            0.3 MB  0.0%
    edgedata                                          0.5 MB  0.0%
    project_project_fngrams                           0.9 MB  0.0%
    project_columnposition                           10.1 MB  0.1%
    edge                                             11.6 MB  0.1%
    project_transaction                              36.5 MB  0.3%
secure_phriction                                     68.0 MB  0.5%
    edgedata                                          0.0 MB  0.0%
    phriction_transaction_comment                     0.0 MB  0.0%
    phriction_document_fngrams_common                 0.0 MB  0.0%
    phriction_document_fdocument                      0.2 MB  0.0%
    phriction_document                                0.3 MB  0.0%
    edge                                              0.6 MB  0.0%
    phriction_content                                11.9 MB  0.1%
    phriction_document_ffield                        12.6 MB  0.1%
    phriction_document_fngrams                       18.5 MB  0.1%
    phriction_transaction                            23.8 MB  0.2%
secure_calendar                                      76.0 MB  0.5%
    edgedata                                          0.0 MB  0.0%
    calendar_notification                             0.0 MB  0.0%
    calendar_import                                   0.0 MB  0.0%
    calendar_externalinvitee                          0.0 MB  0.0%
    calendar_importtransaction                        0.0 MB  0.0%
    calendar_event_fngrams_common                     0.0 MB  0.0%
    calendar_exporttransaction                        0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    calendar_export                                   0.1 MB  0.0%
    calendar_eventtransaction_comment                 0.1 MB  0.0%
    calendar_eventinvitee                             0.1 MB  0.0%
    calendar_event_fdocument                          0.5 MB  0.0%
    calendar_event                                    1.8 MB  0.0%
    calendar_eventtransaction                         1.8 MB  0.0%
    calendar_event_ffield                             2.7 MB  0.0%
    calendar_event_fngrams                           11.5 MB  0.1%
    calendar_importlog                               57.1 MB  0.4%
secure_multimeter                                    84.6 MB  0.6%
    multimeter_host                                   0.0 MB  0.0%
    multimeter_context                                0.1 MB  0.0%
    multimeter_label                                  0.1 MB  0.0%
    multimeter_viewer                                 0.7 MB  0.0%
    multimeter_event                                 83.6 MB  0.6%
secure_diviner                                       88.4 MB  0.6%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    diviner_livebooktransaction                       0.0 MB  0.0%
    diviner_livebook                                  0.0 MB  0.0%
    diviner_livesymbol                               43.2 MB  0.3%
    diviner_liveatom                                 45.1 MB  0.3%
secure_herald                                       158.3 MB  1.1%
    edgedata                                          0.0 MB  0.0%
    herald_action                                     0.0 MB  0.0%
    herald_ruletransaction_comment                    0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    herald_condition                                  0.1 MB  0.0%
    herald_ruletransaction                            0.1 MB  0.0%
    herald_rule                                       0.1 MB  0.0%
    herald_savedheader                                3.5 MB  0.0%
    herald_ruleapplied                                7.0 MB  0.0%
    herald_transcript                               147.3 MB  1.0%
secure_conpherence                                  185.5 MB  1.3%
    edgedata                                          0.0 MB  0.0%
    conpherence_thread                                0.6 MB  0.0%
    conpherence_threadtitle_ngrams                    0.9 MB  0.0%
    conpherence_participant                           1.2 MB  0.0%
    edge                                              2.3 MB  0.0%
    conpherence_index                                21.8 MB  0.2%
    conpherence_transaction_comment                  32.1 MB  0.2%
    conpherence_transaction                         126.6 MB  0.9%
secure_feed                                         221.1 MB  1.5%
    feed_storynotification                            6.0 MB  0.0%
    feed_storyreference                              88.4 MB  0.6%
    feed_storydata                                  126.7 MB  0.9%
secure_cache                                        228.7 MB  1.6%
    cache_general                                    40.9 MB  0.3%
    cache_markupcache                               187.8 MB  1.3%
secure_conduit                                      278.2 MB  1.9%
    conduit_certificatetoken                          0.3 MB  0.0%
    conduit_token                                     0.3 MB  0.0%
    conduit_methodcalllog                           277.6 MB  1.9%
secure_user                                         280.7 MB  2.0%
    user_configuredcustomfieldstorage                 0.0 MB  0.0%
    user_customfieldstringindex                       0.0 MB  0.0%
    user_user_fngrams_common                          0.0 MB  0.0%
    user_customfieldnumericindex                      0.0 MB  0.0%
    user_authinvite                                   0.1 MB  0.0%
    phabricator_session                               0.1 MB  0.0%
    user_transaction                                  1.8 MB  0.0%
    edgedata                                          2.5 MB  0.0%
    user_profile                                      3.0 MB  0.0%
    user_preferences                                  5.6 MB  0.0%
    user_preferencestransaction                       6.5 MB  0.0%
    user_log                                          6.7 MB  0.0%
    user_user_fdocument                              11.0 MB  0.1%
    user_email                                       11.5 MB  0.1%
    user_nametoken                                   12.5 MB  0.1%
    user_user_ffield                                 23.0 MB  0.2%
    user_externalaccount                             27.1 MB  0.2%
    edge                                             27.7 MB  0.2%
    user                                             30.6 MB  0.2%
    user_cache                                       35.1 MB  0.2%
    user_user_fngrams                                75.7 MB  0.5%
secure_search                                       566.0 MB  4.0%
    edgedata                                          0.0 MB  0.0%
    stopwords                                         0.0 MB  0.0%
    search_namedqueryconfig                           0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    search_editengineconfiguration                    0.1 MB  0.0%
    search_namedquery                                 0.1 MB  0.0%
    search_editengineconfigurationtransaction         0.1 MB  0.0%
    search_profilepanelconfiguration                  0.2 MB  0.0%
    search_profilepanelconfigurationtransaction       0.5 MB  0.0%
    search_document                                  33.7 MB  0.2%
    search_indexversion                              61.2 MB  0.4%
    search_documentrelationship                     123.4 MB  0.9%
    search_savedquery                               147.2 MB  1.0%
    search_documentfield                            199.5 MB  1.4%
secure_metamta                                      750.5 MB  5.3%
    edgedata                                          0.0 MB  0.0%
    sms                                               0.0 MB  0.0%
    metamta_applicationemailtransaction               0.0 MB  0.0%
    metamta_applicationemail                          0.1 MB  0.0%
    edge                                             10.5 MB  0.1%
    metamta_receivedmail                             34.8 MB  0.2%
    metamta_mail                                    705.0 MB  4.9%
secure_maniphest                                    968.8 MB  6.8%
    edgedata                                          0.0 MB  0.0%
    maniphest_customfieldstorage                      0.0 MB  0.0%
    maniphest_customfieldnumericindex                 0.0 MB  0.0%
    maniphest_customfieldstringindex                  0.0 MB  0.0%
    maniphest_task_fngrams_common                     0.1 MB  0.0%
    maniphest_nameindex                               0.3 MB  0.0%
    maniphest_task_fdocument                          7.7 MB  0.1%
    maniphest_task                                   26.3 MB  0.2%
    maniphest_transaction_comment                    43.6 MB  0.3%
    edge                                             46.8 MB  0.3%
    maniphest_task_fngrams                          270.9 MB  1.9%
    maniphest_task_ffield                           273.1 MB  1.9%
    maniphest_transaction                           299.8 MB  2.1%
secure_harbormaster                                 994.9 MB  7.0%
    lisk_counter                                      0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    harbormaster_object                               0.0 MB  0.0%
    harbormaster_buildcommand                         0.0 MB  0.0%
    harbormaster_scratchtable                         0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    harbormaster_buildsteptransaction                 0.0 MB  0.0%
    harbormaster_buildplantransaction                 0.0 MB  0.0%
    harbormaster_buildplanname_ngrams                 0.0 MB  0.0%
    harbormaster_buildstep                            0.1 MB  0.0%
    harbormaster_buildplan                            0.1 MB  0.0%
    harbormaster_buildabletransaction                 0.1 MB  0.0%
    harbormaster_buildtransaction                     0.1 MB  0.0%
    harbormaster_buildlintmessage                     2.8 MB  0.0%
    harbormaster_buildmessage                         6.0 MB  0.0%
    harbormaster_buildartifact                        6.6 MB  0.0%
    harbormaster_buildlog                             7.5 MB  0.1%
    harbormaster_buildable                            8.3 MB  0.1%
    harbormaster_build                               15.6 MB  0.1%
    harbormaster_buildtarget                         42.6 MB  0.3%
    harbormaster_buildlogchunk                       70.8 MB  0.5%
    harbormaster_buildunitmessage                   834.0 MB  5.8%
secure_file                                       1,841.7 MB  12.9%
    edgedata                                          0.0 MB  0.0%
    file_filename_ngrams                              0.0 MB  0.0%
    macro_transaction_comment                         0.0 MB  0.0%
    file_externalrequest                              0.1 MB  0.0%
    file_transaction                                  0.1 MB  0.0%
    file_imagemacro                                   0.1 MB  0.0%
    macro_transaction                                 0.2 MB  0.0%
    file_transaction_comment                          0.3 MB  0.0%
    file_chunk                                        0.8 MB  0.0%
    file_transformedfile                             10.5 MB  0.1%
    file                                            118.9 MB  0.8%
    edge                                            512.0 MB  3.6%
    file_storageblob                              1,198.6 MB  8.4%
secure_repository                                 2,336.4 MB  16.4%
    edgedata                                          0.0 MB  0.0%
    repository_filesystem                             0.0 MB  0.0%
    repository_gitlfsref                              0.0 MB  0.0%
    repository_statusmessage                          0.0 MB  0.0%
    repository_coverage                               0.0 MB  0.0%
    repository_summary                                0.0 MB  0.0%
    repository_commithint                             0.0 MB  0.0%
    repository_branch                                 0.0 MB  0.0%
    repository_workingcopyversion                     0.0 MB  0.0%
    repository_repository_fngrams_common              0.0 MB  0.0%
    repository_commit_fngrams_common                  0.0 MB  0.0%
    repository_uritransaction                         0.0 MB  0.0%
    repository_mirror                                 0.0 MB  0.0%
    repository_uriindex                               0.0 MB  0.0%
    repository_repository_ffield                      0.1 MB  0.0%
    repository_uri                                    0.1 MB  0.0%
    repository_repository_fdocument                   0.1 MB  0.0%
    repository_repository_fngrams                     0.1 MB  0.0%
    repository                                        0.1 MB  0.0%
    repository_transaction                            0.1 MB  0.0%
    repository_lintmessage                            0.5 MB  0.0%
    repository_auditrequest                           0.9 MB  0.0%
    repository_oldref                                 1.9 MB  0.0%
    repository_refposition                            3.0 MB  0.0%
    repository_parents                                4.5 MB  0.0%
    repository_refcursor                              5.5 MB  0.0%
    repository_pushevent                              6.5 MB  0.0%
    repository_path                                   7.0 MB  0.0%
    repository_symbol                                 9.0 MB  0.1%
    repository_commit_fdocument                      11.9 MB  0.1%
    repository_commit                                23.1 MB  0.2%
    repository_pullevent                             26.0 MB  0.2%
    repository_commitdata                            36.0 MB  0.3%
    repository_pathchange                            47.1 MB  0.3%
    repository_pushlog                               49.7 MB  0.3%
    repository_commit_ffield                        271.2 MB  1.9%
    repository_commit_fngrams                       436.1 MB  3.1%
    edge                                          1,395.0 MB  9.8%
secure_differential                               4,741.7 MB  33.2%
    differential_hunk                                 0.0 MB  0.0%
    differential_customfieldnumericindex              0.0 MB  0.0%
    differential_customfieldstringindex               0.0 MB  0.0%
    differential_revision_fngrams_common              0.1 MB  0.0%
    differential_hiddencomment                        0.1 MB  0.0%
    differential_customfieldstorage                   1.0 MB  0.0%
    edgedata                                          3.5 MB  0.0%
    differential_difftransaction                      4.5 MB  0.0%
    differential_commit                               5.0 MB  0.0%
    differential_revision_fdocument                   8.0 MB  0.1%
    differential_revisionhash                         9.5 MB  0.1%
    differential_reviewer                            13.6 MB  0.1%
    differential_diff                                19.1 MB  0.1%
    differential_revision                            20.6 MB  0.1%
    differential_affectedpath                        31.5 MB  0.2%
    differential_transaction_comment                 50.2 MB  0.4%
    differential_transaction                        126.8 MB  0.9%
    differential_changeset                          137.1 MB  1.0%
    differential_revision_ffield                    182.1 MB  1.3%
    differential_revision_fngrams                   293.0 MB  2.1%
    differential_diffproperty                       316.1 MB  2.2%
    edge                                            362.0 MB  2.5%
    differential_hunk_modern                      1,466.0 MB  10.3%
    differential_changeset_parse_cache            1,691.7 MB  11.8%
TOTAL                                            14,279.2 MB  100.0%
secure003 probe
ubuntu@secure001:/core$ PHABRICATOR_INSTANCE=secure ./lib/phabricator/bin/storage probe --host secure003.phacility.net
Analyzing table sizes (this may take a moment)...
secure_xhpast                                         0.0 MB  0.0%
    xhpast_parsetree                                  0.0 MB  0.0%
secure_phrequent                                      0.0 MB  0.0%
    phrequent_usertime                                0.0 MB  0.0%
secure_policy                                         0.0 MB  0.0%
    policy                                            0.0 MB  0.0%
secure_xhprof                                         0.0 MB  0.0%
    xhprof_sample                                     0.0 MB  0.0%
secure_flag                                           0.0 MB  0.0%
    flag                                              0.0 MB  0.0%
secure_chatlog                                        0.1 MB  0.0%
    chatlog_event                                     0.0 MB  0.0%
    chatlog_channel                                   0.0 MB  0.0%
secure_draft                                          0.1 MB  0.0%
    draft_versioneddraft                              0.0 MB  0.0%
    draft                                             0.0 MB  0.0%
secure_system                                         0.1 MB  0.0%
    system_destructionlog                             0.0 MB  0.0%
    system_actionlog                                  0.0 MB  0.0%
secure_phlux                                          0.1 MB  0.0%
    phlux_variable                                    0.0 MB  0.0%
    phlux_transaction                                 0.0 MB  0.0%
secure_cache                                          0.1 MB  0.0%
    cache_markupcache                                 0.0 MB  0.0%
    cache_general                                     0.1 MB  0.0%
secure_daemon                                         0.1 MB  0.0%
    daemon_logevent                                   0.0 MB  0.0%
    daemon_log                                        0.1 MB  0.0%
secure_doorkeeper                                     0.1 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    doorkeeper_externalobject                         0.1 MB  0.0%
secure_config                                         0.1 MB  0.0%
    config_manualactivity                             0.0 MB  0.0%
    config_transaction                                0.0 MB  0.0%
    config_entry                                      0.0 MB  0.0%
secure_fact                                           0.1 MB  0.0%
    fact_aggregate                                    0.0 MB  0.0%
    fact_cursor                                       0.0 MB  0.0%
    fact_raw                                          0.1 MB  0.0%
secure_feed                                           0.1 MB  0.0%
    feed_storyreference                               0.0 MB  0.0%
    feed_storydata                                    0.0 MB  0.0%
    feed_storynotification                            0.1 MB  0.0%
secure_application                                    0.1 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    application_application                           0.0 MB  0.0%
    application_applicationtransaction                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_meta_data                                      0.1 MB  0.0%
    hoststate                                         0.0 MB  0.0%
    patch_status                                      0.1 MB  0.0%
secure_audit                                          0.2 MB  0.0%
    audit_transaction                                 0.0 MB  0.0%
    audit_transaction_comment                         0.1 MB  0.0%
secure_spaces                                         0.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    spaces_namespacetransaction                       0.0 MB  0.0%
    spaces_namespace                                  0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_token                                          0.2 MB  0.0%
    token_count                                       0.0 MB  0.0%
    token_token                                       0.1 MB  0.0%
    token_given                                       0.1 MB  0.0%
secure_multimeter                                     0.2 MB  0.0%
    multimeter_viewer                                 0.0 MB  0.0%
    multimeter_label                                  0.0 MB  0.0%
    multimeter_host                                   0.0 MB  0.0%
    multimeter_context                                0.0 MB  0.0%
    multimeter_event                                  0.0 MB  0.0%
secure_conduit                                        0.2 MB  0.0%
    conduit_certificatetoken                          0.0 MB  0.0%
    conduit_token                                     0.1 MB  0.0%
    conduit_methodcalllog                             0.1 MB  0.0%
secure_countdown                                      0.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    countdown_transaction_comment                     0.0 MB  0.0%
    countdown_transaction                             0.0 MB  0.0%
    countdown                                         0.1 MB  0.0%
secure_phragment                                      0.2 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phragment_snapshotchild                           0.0 MB  0.0%
    phragment_snapshot                                0.0 MB  0.0%
    phragment_fragment                                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phragment_fragmentversion                         0.0 MB  0.0%
secure_pastebin                                       0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    pastebin_pastetransaction                         0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    pastebin_pastetransaction_comment                 0.0 MB  0.0%
    pastebin_paste                                    0.1 MB  0.0%
secure_oauth_server                                   0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    oauth_server_oauthserverauthorizationcode         0.0 MB  0.0%
    oauth_server_oauthserveraccesstoken               0.0 MB  0.0%
    oauth_server_transaction                          0.0 MB  0.0%
    oauth_server_oauthserverclient                    0.0 MB  0.0%
    oauth_server_oauthclientauthorization             0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_slowvote                                       0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    slowvote_option                                   0.0 MB  0.0%
    slowvote_transaction_comment                      0.0 MB  0.0%
    slowvote_transaction                              0.0 MB  0.0%
    slowvote_poll                                     0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    slowvote_choice                                   0.0 MB  0.0%
secure_phurl                                          0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phurl_urltransaction_comment                      0.0 MB  0.0%
    phurl_urltransaction                              0.0 MB  0.0%
    phurl_phurlname_ngrams                            0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phurl_url                                         0.1 MB  0.0%
secure_badges                                         0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    badges_transaction_comment                        0.0 MB  0.0%
    badges_transaction                                0.0 MB  0.0%
    badges_badge                                      0.0 MB  0.0%
    badges_badgename_ngrams                           0.0 MB  0.0%
    badges_award                                      0.0 MB  0.0%
secure_conpherence                                    0.3 MB  0.0%
    conpherence_index                                 0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    conpherence_thread                                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    conpherence_threadtitle_ngrams                    0.0 MB  0.0%
    conpherence_participant                           0.0 MB  0.0%
    conpherence_transaction                           0.0 MB  0.0%
    conpherence_transaction_comment                   0.1 MB  0.0%
secure_diviner                                        0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    diviner_liveatom                                  0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    diviner_livebooktransaction                       0.0 MB  0.0%
    diviner_livebook                                  0.0 MB  0.0%
    diviner_livesymbol                                0.1 MB  0.0%
secure_dashboard                                      0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    dashboard                                         0.0 MB  0.0%
    dashboard_install                                 0.0 MB  0.0%
    dashboard_panel                                   0.0 MB  0.0%
    dashboard_dashboard_ngrams                        0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    dashboard_transaction                             0.0 MB  0.0%
    dashboard_paneltransaction                        0.0 MB  0.0%
    dashboard_dashboardpanel_ngrams                   0.0 MB  0.0%
secure_legalpad                                       0.3 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    legalpad_transaction                              0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    legalpad_documentbody                             0.0 MB  0.0%
    legalpad_transaction_comment                      0.1 MB  0.0%
    legalpad_document                                 0.1 MB  0.0%
    legalpad_documentsignature                        0.1 MB  0.0%
secure_releeph                                        0.4 MB  0.0%
    releeph_requesttransaction_comment                0.0 MB  0.0%
    releeph_requesttransaction                        0.0 MB  0.0%
    releeph_project                                   0.0 MB  0.0%
    releeph_branchtransaction                         0.0 MB  0.0%
    releeph_producttransaction                        0.0 MB  0.0%
    releeph_request                                   0.1 MB  0.0%
    releeph_branch                                    0.1 MB  0.0%
secure_metamta                                        0.4 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    sms                                               0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    metamta_applicationemailtransaction               0.0 MB  0.0%
    metamta_applicationemail                          0.1 MB  0.0%
    metamta_receivedmail                              0.1 MB  0.0%
    metamta_mail                                      0.1 MB  0.0%
secure_herald                                         0.4 MB  0.0%
    herald_savedheader                                0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    herald_ruleapplied                                0.0 MB  0.0%
    herald_action                                     0.0 MB  0.0%
    herald_condition                                  0.0 MB  0.0%
    herald_ruletransaction_comment                    0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    herald_ruletransaction                            0.0 MB  0.0%
    herald_transcript                                 0.1 MB  0.0%
    herald_rule                                       0.1 MB  0.0%
secure_passphrase                                     0.4 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    passphrase_secret                                 0.0 MB  0.0%
    passphrase_credential_ffield                      0.0 MB  0.0%
    passphrase_credentialtransaction                  0.0 MB  0.0%
    passphrase_credential_fngrams_common              0.0 MB  0.0%
    passphrase_credential_fngrams                     0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    passphrase_credential                             0.1 MB  0.0%
    passphrase_credential_fdocument                   0.1 MB  0.0%
secure_auth                                           0.5 MB  0.0%
    auth_hmackey                                      0.0 MB  0.0%
    auth_providerconfigtransaction                    0.0 MB  0.0%
    auth_factorconfig                                 0.0 MB  0.0%
    auth_passwordtransaction                          0.0 MB  0.0%
    auth_sshkeytransaction                            0.0 MB  0.0%
    auth_password                                     0.0 MB  0.0%
    auth_temporarytoken                               0.1 MB  0.0%
    auth_providerconfig                               0.1 MB  0.0%
    auth_sshkey                                       0.1 MB  0.0%
secure_packages                                       0.5 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    packages_publishertransaction                     0.0 MB  0.0%
    packages_version                                  0.0 MB  0.0%
    packages_versiontransaction                       0.0 MB  0.0%
    packages_publishername_ngrams                     0.0 MB  0.0%
    packages_versionname_ngrams                       0.0 MB  0.0%
    packages_publisher                                0.0 MB  0.0%
    packages_package                                  0.0 MB  0.0%
    packages_packagename_ngrams                       0.0 MB  0.0%
    packages_packagetransaction                       0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
secure_pholio                                         0.5 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    pholio_mock_ffield                                0.0 MB  0.0%
    pholio_mock_fngrams                               0.0 MB  0.0%
    pholio_transaction                                0.0 MB  0.0%
    pholio_mock_fngrams_common                        0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    pholio_image                                      0.0 MB  0.0%
    pholio_mock                                       0.1 MB  0.0%
    pholio_transaction_comment                        0.1 MB  0.0%
    pholio_mock_fdocument                             0.1 MB  0.0%
secure_phriction                                      0.5 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phriction_document_ffield                         0.0 MB  0.0%
    phriction_document_fngrams                        0.0 MB  0.0%
    phriction_transaction                             0.0 MB  0.0%
    phriction_document_fngrams_common                 0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phriction_transaction_comment                     0.0 MB  0.0%
    phriction_content                                 0.1 MB  0.0%
    phriction_document                                0.1 MB  0.0%
    phriction_document_fdocument                      0.1 MB  0.0%
secure_search                                         0.5 MB  0.0%
    search_documentfield                              0.0 MB  0.0%
    stopwords                                         0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    search_namedquery                                 0.0 MB  0.0%
    search_indexversion                               0.0 MB  0.0%
    search_savedquery                                 0.0 MB  0.0%
    search_namedqueryconfig                           0.0 MB  0.0%
    search_profilepanelconfigurationtransaction       0.0 MB  0.0%
    search_profilepanelconfiguration                  0.0 MB  0.0%
    search_editengineconfigurationtransaction         0.0 MB  0.0%
    search_document                                   0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    search_documentrelationship                       0.1 MB  0.0%
    search_editengineconfiguration                    0.1 MB  0.0%
secure_fund                                           0.5 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    fund_initiative_ffield                            0.0 MB  0.0%
    fund_initiative_fngrams                           0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    fund_backertransaction                            0.0 MB  0.0%
    fund_initiativetransaction                        0.0 MB  0.0%
    fund_initiative_fngrams_common                    0.0 MB  0.0%
    fund_initiativetransaction_comment                0.0 MB  0.0%
    fund_initiative                                   0.1 MB  0.0%
    fund_backer                                       0.1 MB  0.0%
    fund_initiative_fdocument                         0.1 MB  0.0%
secure_worker                                         0.6 MB  0.0%
    worker_taskdata                                   0.0 MB  0.0%
    lisk_counter                                      0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    worker_bulktask                                   0.0 MB  0.0%
    worker_bulkjobtransaction                         0.0 MB  0.0%
    worker_triggerevent                               0.0 MB  0.0%
    worker_trigger                                    0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    worker_bulkjob                                    0.1 MB  0.0%
    worker_archivetask                                0.1 MB  0.0%
    worker_activetask                                 0.1 MB  0.0%
secure_owners                                         0.6 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    owners_package                                    0.0 MB  0.0%
    owners_package_ffield                             0.0 MB  0.0%
    owners_customfieldstorage                         0.0 MB  0.0%
    owners_path                                       0.0 MB  0.0%
    owners_package_fngrams                            0.0 MB  0.0%
    owners_package_fngrams_common                     0.0 MB  0.0%
    owners_packagetransaction                         0.0 MB  0.0%
    owners_owner                                      0.0 MB  0.0%
    owners_customfieldnumericindex                    0.0 MB  0.0%
    owners_customfieldstringindex                     0.0 MB  0.0%
    owners_name_ngrams                                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    owners_package_fdocument                          0.1 MB  0.0%
secure_drydock                                        0.6 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    drydock_blueprint                                 0.0 MB  0.0%
    drydock_command                                   0.0 MB  0.0%
    drydock_slotlock                                  0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    drydock_blueprinttransaction                      0.0 MB  0.0%
    drydock_blueprintname_ngrams                      0.0 MB  0.0%
    drydock_resource                                  0.1 MB  0.0%
    drydock_lease                                     0.1 MB  0.0%
    drydock_repositoryoperation                       0.1 MB  0.0%
    drydock_authorization                             0.1 MB  0.0%
    drydock_log                                       0.1 MB  0.0%
secure_ponder                                         0.6 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    ponder_question_ffield                            0.0 MB  0.0%
    ponder_questiontransaction                        0.0 MB  0.0%
    ponder_question_fngrams                           0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    ponder_answertransaction_comment                  0.0 MB  0.0%
    ponder_questiontransaction_comment                0.0 MB  0.0%
    ponder_answertransaction                          0.0 MB  0.0%
    ponder_question_fngrams_common                    0.0 MB  0.0%
    ponder_question                                   0.1 MB  0.0%
    ponder_answer                                     0.1 MB  0.0%
    ponder_question_fdocument                         0.1 MB  0.0%
secure_nuance                                         0.7 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    nuance_itemcommand                                0.0 MB  0.0%
    nuance_queue                                      0.0 MB  0.0%
    nuance_sourcename_ngrams                          0.0 MB  0.0%
    nuance_source                                     0.0 MB  0.0%
    nuance_sourcetransaction                          0.0 MB  0.0%
    nuance_sourcetransaction_comment                  0.0 MB  0.0%
    nuance_queuetransaction_comment                   0.0 MB  0.0%
    nuance_itemtransaction_comment                    0.0 MB  0.0%
    nuance_importcursordata                           0.0 MB  0.0%
    nuance_queuetransaction                           0.0 MB  0.0%
    nuance_itemtransaction                            0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    nuance_item                                       0.1 MB  0.0%
secure_phame                                          0.7 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phame_blog_ffield                                 0.0 MB  0.0%
    phame_post_ffield                                 0.0 MB  0.0%
    phame_post_fngrams_common                         0.0 MB  0.0%
    phame_post_fngrams                                0.0 MB  0.0%
    phame_posttransaction                             0.0 MB  0.0%
    phame_posttransaction_comment                     0.0 MB  0.0%
    phame_post                                        0.0 MB  0.0%
    phame_blogtransaction                             0.0 MB  0.0%
    phame_blog                                        0.0 MB  0.0%
    phame_blog_fngrams                                0.0 MB  0.0%
    phame_blog_fngrams_common                         0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phame_blog_fdocument                              0.1 MB  0.0%
    phame_post_fdocument                              0.1 MB  0.0%
secure_phortune                                       0.8 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    phortune_account                                  0.0 MB  0.0%
    phortune_merchant                                 0.0 MB  0.0%
    phortune_paymentproviderconfig                    0.0 MB  0.0%
    phortune_paymentproviderconfigtransaction         0.0 MB  0.0%
    phortune_product                                  0.0 MB  0.0%
    phortune_purchase                                 0.0 MB  0.0%
    phortune_accounttransaction                       0.0 MB  0.0%
    phortune_merchanttransaction                      0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    phortune_carttransaction                          0.0 MB  0.0%
    phortune_paymentmethod                            0.1 MB  0.0%
    phortune_cart                                     0.1 MB  0.0%
    phortune_subscription                             0.1 MB  0.0%
    phortune_charge                                   0.1 MB  0.0%
secure_maniphest                                      0.8 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    maniphest_customfieldstorage                      0.0 MB  0.0%
    maniphest_task_ffield                             0.0 MB  0.0%
    maniphest_task_fngrams                            0.0 MB  0.0%
    maniphest_transaction                             0.0 MB  0.0%
    maniphest_task_fngrams_common                     0.0 MB  0.0%
    maniphest_nameindex                               0.0 MB  0.0%
    maniphest_customfieldnumericindex                 0.0 MB  0.0%
    maniphest_customfieldstringindex                  0.0 MB  0.0%
    maniphest_transaction_comment                     0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    maniphest_task_fdocument                          0.1 MB  0.0%
    maniphest_task                                    0.2 MB  0.0%
secure_calendar                                       0.8 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    calendar_importlog                                0.0 MB  0.0%
    calendar_event_ffield                             0.0 MB  0.0%
    calendar_notification                             0.0 MB  0.0%
    calendar_externalinvitee                          0.0 MB  0.0%
    calendar_importtransaction                        0.0 MB  0.0%
    calendar_exporttransaction                        0.0 MB  0.0%
    calendar_import                                   0.0 MB  0.0%
    calendar_eventtransaction                         0.0 MB  0.0%
    calendar_event_fngrams                            0.0 MB  0.0%
    calendar_event_fngrams_common                     0.0 MB  0.0%
    calendar_eventinvitee                             0.0 MB  0.0%
    calendar_eventtransaction_comment                 0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    calendar_export                                   0.1 MB  0.0%
    calendar_event_fdocument                          0.1 MB  0.0%
    calendar_event                                    0.1 MB  0.0%
secure_project                                        0.9 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    project_project_ffield                            0.0 MB  0.0%
    project_customfieldstorage                        0.0 MB  0.0%
    project_project_fngrams                           0.0 MB  0.0%
    project_project_fngrams_common                    0.0 MB  0.0%
    project_slug                                      0.0 MB  0.0%
    project_transaction                               0.0 MB  0.0%
    project_customfieldstringindex                    0.0 MB  0.0%
    project_datasourcetoken                           0.0 MB  0.0%
    project_columntransaction                         0.0 MB  0.0%
    project_customfieldnumericindex                   0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    project_columnposition                            0.1 MB  0.0%
    project_column                                    0.1 MB  0.0%
    project_project_fdocument                         0.1 MB  0.0%
    project                                           0.1 MB  0.0%
secure_almanac                                        0.9 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    almanac_network                                   0.0 MB  0.0%
    almanac_property                                  0.0 MB  0.0%
    almanac_networktransaction                        0.0 MB  0.0%
    almanac_networkname_ngrams                        0.0 MB  0.0%
    almanac_servicetransaction                        0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    almanac_servicename_ngrams                        0.0 MB  0.0%
    almanac_namespacetransaction                      0.0 MB  0.0%
    almanac_devicetransaction                         0.0 MB  0.0%
    almanac_devicename_ngrams                         0.0 MB  0.0%
    almanac_namespacename_ngrams                      0.0 MB  0.0%
    almanac_bindingtransaction                        0.0 MB  0.0%
    almanac_interface                                 0.1 MB  0.0%
    almanac_namespace                                 0.1 MB  0.0%
    almanac_device                                    0.1 MB  0.0%
    almanac_service                                   0.1 MB  0.0%
    almanac_binding                                   0.1 MB  0.0%
secure_harbormaster                                   1.0 MB  0.0%
    lisk_counter                                      0.0 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    harbormaster_buildlogchunk                        0.0 MB  0.0%
    harbormaster_buildunitmessage                     0.0 MB  0.0%
    harbormaster_scratchtable                         0.0 MB  0.0%
    harbormaster_buildlintmessage                     0.0 MB  0.0%
    harbormaster_object                               0.0 MB  0.0%
    harbormaster_buildmessage                         0.0 MB  0.0%
    harbormaster_buildcommand                         0.0 MB  0.0%
    harbormaster_buildlog                             0.0 MB  0.0%
    harbormaster_buildsteptransaction                 0.0 MB  0.0%
    harbormaster_buildtarget                          0.0 MB  0.0%
    harbormaster_buildplantransaction                 0.0 MB  0.0%
    harbormaster_buildplanname_ngrams                 0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    harbormaster_buildabletransaction                 0.0 MB  0.0%
    harbormaster_buildtransaction                     0.0 MB  0.0%
    harbormaster_buildstep                            0.1 MB  0.0%
    harbormaster_buildable                            0.1 MB  0.0%
    harbormaster_buildplan                            0.1 MB  0.0%
    harbormaster_buildartifact                        0.1 MB  0.0%
    harbormaster_build                                0.1 MB  0.0%
secure_user                                           1.1 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    user_profile                                      0.0 MB  0.0%
    user_user_ffield                                  0.0 MB  0.0%
    user_configuredcustomfieldstorage                 0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    user_user_fngrams_common                          0.0 MB  0.0%
    user_nametoken                                    0.0 MB  0.0%
    user_email                                        0.0 MB  0.0%
    user_customfieldstringindex                       0.0 MB  0.0%
    user_transaction                                  0.0 MB  0.0%
    user_preferencestransaction                       0.0 MB  0.0%
    user_customfieldnumericindex                      0.0 MB  0.0%
    user_user_fngrams                                 0.0 MB  0.0%
    user_externalaccount                              0.1 MB  0.0%
    user_authinvite                                   0.1 MB  0.0%
    user_preferences                                  0.1 MB  0.0%
    user_cache                                        0.1 MB  0.0%
    phabricator_session                               0.1 MB  0.0%
    user                                              0.1 MB  0.0%
    user_user_fdocument                               0.1 MB  0.0%
    user_log                                          0.1 MB  0.0%
secure_differential                                   1.1 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    differential_diffproperty                         0.0 MB  0.0%
    differential_revision_ffield                      0.0 MB  0.0%
    differential_customfieldstorage                   0.0 MB  0.0%
    differential_hunk                                 0.0 MB  0.0%
    differential_changeset                            0.0 MB  0.0%
    differential_changeset_parse_cache                0.0 MB  0.0%
    differential_commit                               0.0 MB  0.0%
    differential_transaction                          0.0 MB  0.0%
    differential_revisionhash                         0.0 MB  0.0%
    differential_revision_fngrams                     0.0 MB  0.0%
    differential_revision_fngrams_common              0.0 MB  0.0%
    differential_customfieldnumericindex              0.0 MB  0.0%
    differential_hunk_modern                          0.0 MB  0.0%
    differential_difftransaction                      0.0 MB  0.0%
    differential_customfieldstringindex               0.0 MB  0.0%
    differential_reviewer                             0.0 MB  0.0%
    differential_hiddencomment                        0.0 MB  0.0%
    differential_affectedpath                         0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    differential_diff                                 0.1 MB  0.0%
    differential_revision                             0.1 MB  0.0%
    differential_revision_fdocument                   0.1 MB  0.0%
    differential_transaction_comment                  0.1 MB  0.0%
secure_repository                                     1.9 MB  0.0%
    edgedata                                          0.0 MB  0.0%
    repository_statusmessage                          0.0 MB  0.0%
    repository_summary                                0.0 MB  0.0%
    repository_filesystem                             0.0 MB  0.0%
    repository_oldref                                 0.0 MB  0.0%
    repository_gitlfsref                              0.0 MB  0.0%
    repository_path                                   0.0 MB  0.0%
    repository_refposition                            0.0 MB  0.0%
    repository_repository_ffield                      0.0 MB  0.0%
    repository_pathchange                             0.0 MB  0.0%
    repository_coverage                               0.0 MB  0.0%
    repository_commithint                             0.0 MB  0.0%
    repository_commit_ffield                          0.0 MB  0.0%
    repository_branch                                 0.0 MB  0.0%
    repository_workingcopyversion                     0.0 MB  0.0%
    repository_symbol                                 0.0 MB  0.0%
    repository_commitdata                             0.0 MB  0.0%
    repository_repository_fngrams                     0.0 MB  0.0%
    repository_uritransaction                         0.0 MB  0.0%
    repository_uri                                    0.0 MB  0.0%
    repository_transaction                            0.0 MB  0.0%
    repository_refcursor                              0.0 MB  0.0%
    repository_repository_fngrams_common              0.0 MB  0.0%
    repository_uriindex                               0.0 MB  0.0%
    repository_parents                                0.0 MB  0.0%
    edge                                              0.0 MB  0.0%
    repository_mirror                                 0.0 MB  0.0%
    repository_commit_fngrams                         0.0 MB  0.0%
    repository_pushevent                              0.0 MB  0.0%
    repository_commit_fngrams_common                  0.0 MB  0.0%
    repository_auditrequest                           0.1 MB  0.0%
    repository_lintmessage                            0.1 MB  0.0%
    repository_pullevent                              0.1 MB  0.0%
    repository_repository_fdocument                   0.1 MB  0.0%
    repository_commit_fdocument                       0.1 MB  0.0%
    repository_pushlog                                0.1 MB  0.0%
    repository                                        0.1 MB  0.0%
    repository_commit                                 0.1 MB  0.0%
secure_file                                       5,534.7 MB  99.6%
    edgedata                                          0.0 MB  0.0%
    macro_transaction_comment                         0.0 MB  0.0%
    file_externalrequest                              0.1 MB  0.0%
    file_imagemacro                                   0.1 MB  0.0%
    file_transaction                                  0.1 MB  0.0%
    macro_transaction                                 0.2 MB  0.0%
    file_transaction_comment                          0.3 MB  0.0%
    file_chunk                                        2.5 MB  0.0%
    file_transformedfile                             11.5 MB  0.2%
    file                                            175.0 MB  3.1%
    file_filename_ngrams                            366.0 MB  6.6%
    edge                                          1,292.0 MB  23.2%
    file_storageblob                              3,686.7 MB  66.3%
TOTAL                                             5,557.9 MB  100.0%

Our *_transaction tables generally aren't that big (I expect few installs have the kind of pathological data that the instance in question does), although these have a little meat on them:

differential_transaction                        126.8 MB  0.9%
audit_transaction                                41.6 MB  0.3%
conpherence_transaction                         126.6 MB  0.9%
maniphest_transaction                           299.8 MB  2.1%

I'm going to run bin/garbage compact-edges now.

Took like 3-ish minutes and did this:

bin/garbage compact-edges
$ PHABRICATOR_INSTANCE=secure ./lib/phabricator/bin/garbage compact-edges                       
Rebuilding transactions for "AlmanacNamespaceTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "AlmanacNetworkTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "AlmanacBindingTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "AlmanacDeviceTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "AlmanacServiceTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "DifferentialDiffTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "DivinerLiveBookTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "HarbormasterBuildPlanTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "HarbormasterBuildStepTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "HarbormasterBuildTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "HarbormasterBuildableTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "HeraldRuleTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorAuthProviderConfigTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorAuthSSHKeyTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorConfigTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorDashboardPanelTransaction"...
Done, compacted 890 edge transactions.
Rebuilding transactions for "PhabricatorDashboardTransaction"...
Done, compacted 689 edge transactions.
Rebuilding transactions for "PhabricatorEditEngineConfigurationTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorMetaMTAApplicationEmailTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "ConpherenceTransaction"...
Done, compacted 1,327 edge transactions.
Rebuilding transactions for "DifferentialTransaction"...
Done, compacted 38,192 edge transactions.
Rebuilding transactions for "DrydockBlueprintTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "FundBackerTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "FundInitiativeTransaction"...
Done, compacted 16 edge transactions.
Rebuilding transactions for "LegalpadTransaction"...
Done, compacted 1 edge transactions.
Rebuilding transactions for "ManiphestTransaction"...
Done, compacted 45,158 edge transactions.
Rebuilding transactions for "NuanceItemTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "NuanceQueueTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "NuanceSourceTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PassphraseCredentialTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorApplicationApplicationTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorAuditTransaction"...
Done, compacted 10,693 edge transactions.
Rebuilding transactions for "PhabricatorAuthPasswordTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorBadgesTransaction"...
Done, compacted 80 edge transactions.
Rebuilding transactions for "PhabricatorCalendarEventTransaction"...
Done, compacted 51 edge transactions.
Rebuilding transactions for "PhabricatorCalendarExportTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorCalendarImportTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorCountdownTransaction"...
Done, compacted 1 edge transactions.
Rebuilding transactions for "PhabricatorFileTransaction"...
Done, compacted 4 edge transactions.
Rebuilding transactions for "PhabricatorMacroTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorOwnersPackageTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorPackagesPackageTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorPackagesPublisherTransaction"...
Done, compacted 1 edge transactions.
Rebuilding transactions for "PhabricatorPackagesVersionTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorPasteTransaction"...
Done, compacted 296 edge transactions.
Rebuilding transactions for "PhabricatorPhurlURLTransaction"...
Done, compacted 9 edge transactions.
Rebuilding transactions for "PhabricatorProjectTransaction"...
Done, compacted 979 edge transactions.
Rebuilding transactions for "PhabricatorSlowvoteTransaction"...
Done, compacted 2 edge transactions.
Rebuilding transactions for "PhabricatorSpacesNamespaceTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhameBlogTransaction"...
Done, compacted 1 edge transactions.
Rebuilding transactions for "PhamePostTransaction"...
Done, compacted 42 edge transactions.
Rebuilding transactions for "PholioTransaction"...
Done, compacted 176 edge transactions.
Rebuilding transactions for "PhortuneAccountTransaction"...
Done, compacted 967 edge transactions.
Rebuilding transactions for "PhortuneMerchantTransaction"...
Done, compacted 4 edge transactions.
Rebuilding transactions for "PhrictionTransaction"...
Done, compacted 192 edge transactions.
Rebuilding transactions for "PonderAnswerTransaction"...
Done, compacted 130 edge transactions.
Rebuilding transactions for "PonderQuestionTransaction"...
Done, compacted 450 edge transactions.
Rebuilding transactions for "PhabricatorOAuthServerTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorProfileMenuItemConfigurationTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorProjectColumnTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorRepositoryTransaction"...
Done, compacted 5 edge transactions.
Rebuilding transactions for "PhabricatorRepositoryURITransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorUserPreferencesTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorUserTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhabricatorWorkerBulkJobTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhluxTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhortuneCartTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "PhortunePaymentProviderConfigTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "ReleephBranchTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "ReleephProductTransaction"...
Done, compacted 0 edge transactions.
Rebuilding transactions for "ReleephRequestTransaction"...
Done, compacted 0 edge transactions.

I'm going to optimize + probe secure001 now and see if any of the tables above shrunk. I'm expecting a very modest effect combined with zero user visible changes in the UI despite throwing away a bunch of data.

epriestley added a comment.EditedJan 29 2018, 11:04 PM

Even on our fairly normal data, the effect was a little bit more dramatic than I'd expected:

differential_transaction                        112.8 MB  0.8%
audit_transaction                                36.6 MB  0.3%
conpherence_transaction                          53.6 MB  0.4%
maniphest_transaction                           202.8 MB  1.4%

It looks like Differential shrank by about 10%, Maniphest shrank by about 30%, and Conpherence shrank by almost 60%.

If we don't catch any weirdness here on secure soon, I'm going to cherry-pick this stuff to production and compact the problem instance either later tonight or tomorrow morning.

web004 died abruptly so I'm going to fix that and deploy these changes at the same time.

I pulled web004-old out of lb001 and launched the new web004.

web004 is deploying now.

Once it deploys, I'll upgrade web001-3 and then put web004 into the LB pool. After that, I'll upgrade the repo pool. There are no schema changes so db does not need an upgrade.

Pool is full again, repo is upgrading, edges are compacting on the instance shard.

The compaction completed overnight. I'm optimizing the tables now.

...
 OPTIMIZE  Optimizing table "<instance>_audit"."audit_transaction"...
  DONE  Compacted table by 139 GB in 910,219ms.
...

I just let that run for a while but it finished at some point:

OPTIMIZED  Completed optimizations, reclaimed 141 GB of disk space.

The total instance database size is now ~14GB, down from ~150GB.

I haven't caught any issues with this change more broadly, either, so I think this is largely resolved. (I'm planning to file a followup with cleanup plans before closing this out, though.)

epriestley closed this task as Resolved.EditedJan 31 2018, 2:40 PM

See T13056 for followup.