Page MenuHomePhabricator
Create Task
Maniphest T13206

Diviner publishing can fail resoundingly with unusual file permissions [was: User documentation on "secure" is occasionally unpublishing]
Open, LowPublic
Actions

Description

Twice now, the user documentation has unpublished on secure. In this state, bin/diviner generate --book ... reports that the documentation is published, and bin/diviner generate --book ... --clean fixes the issue.

Neither Diviner nor the cronjob driving documentation generation have been touched in a long time, so this is a bit mystifying at first glance.

Related Objects

Mentioned In
Z1336: General Chat

Event Timeline

epriestley triaged this task as Normal priority.Sep 27 2018, 1:01 PM
epriestley created this task.
epriestley added a comment.Oct 5 2018, 1:14 PM

The "Contributor Introduction" document, specifically, seems to have unpublished. Let me see what I can dig up about the current database state.

epriestley added a comment.Oct 5 2018, 1:27 PM

Oh, the contrib stuff got separated into a different book and has fully unpublished:

Screen Shot 2018-10-05 at 6.20.03 AM.png (838×1 px, 82 KB)

So this isn't new behavior, and presumably nothing about the "User" book is unique.

The book itself is in good shape:

mysql> select * from diviner_livebook where phid = 'PHID-BOOK-ux3r2a54misiq6swn7ze'\G
*************************** 1. row ***************************
               id: 3
             phid: PHID-BOOK-ux3r2a54misiq6swn7ze
             name: phabcontrib
   repositoryPHID: NULL
       viewPolicy: public
       editPolicy: admin
      dateCreated: 1394059106
     dateModified: 1538726436
configurationData: {"name":"phabcontrib","title":"Phabricator Contributor Documentation","short":"Phabricator Contributor Docs","preface":"Information for Phabricator contributors and developers.","root":"\/core\/lib\/phabricator","uri.source":"https:\/\/secure.phabricator.com\/diffusion\/P\/browse\/master\/%f$%l","rules":{"(\\.diviner$)":"DivinerArticleAtomizer"},"exclude":["(^externals\/)","(^resources\/)","(^scripts\/)","(^src\/docs\/flavor\/)","(^src\/docs\/tech\/)","(^src\/docs\/user\/)","(^support\/)","(^webroot\/rsrc\/externals\/)"],"groups":{"contrib":{"name":"Contributor Overview"},"detail":{"name":"Contributing in Detail"},"developer":{"name":"Developer Guides"},"standards":{"name":"Coding Standards"}}}
1 row in set (0.00 sec)

The atoms exist, too:

*************************** 1. row ***************************
            id: 38338
          phid: PHID-ATOM-jsm3rh5uunsriqq6zysn
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: adding_new_classes
     atomIndex: 0
  identityHash: KayRfHCAgbqM
     graphHash: NULL
         title: Adding New Classes
 titleSlugHash: lwERLmOwPQ33
     groupName: developer
       summary: Guide to adding new classes to extend Phabricator.
isDocumentable: 1
      nodeHash: NULL
*************************** 2. row ***************************
            id: 4757
          phid: PHID-ATOM-r5p6rh5qih33hvrxsr2g
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: adding_new_css_and_js
     atomIndex: 0
  identityHash: PtNCmA.nVdKE
     graphHash: NULL
         title: Adding New CSS and JS
 titleSlugHash: cvlBYWWXJnN.
     groupName: developer
       summary: Explains how to add new CSS and JS files to Phabricator.
isDocumentable: 1
      nodeHash: NULL
*************************** 3. row ***************************
            id: 26562
          phid: PHID-ATOM-pmrxaull35ss4fl27vps
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: assistive_technologies
     atomIndex: 0
  identityHash: fHQWEU2X6z6C
     graphHash: NULL
         title: Assistive Technologies
 titleSlugHash: PjJ.5OjjxGa7
     groupName: developer
       summary: Information about making Phabricator accessible to assistive technologies.
isDocumentable: 1
      nodeHash: NULL
*************************** 4. row ***************************
            id: 31928
          phid: PHID-ATOM-p5hwo5vyzpjttqcnh5kx
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: bug_reports
     atomIndex: 0
  identityHash: jh0TM0M2xKRL
     graphHash: NULL
         title: Contributing Bug Reports
 titleSlugHash: 4rgn9RNp4AiU
     groupName: detail
       summary: Describes how to file an effective Phabricator bug report.
isDocumentable: 1
      nodeHash: NULL
*************************** 5. row ***************************
            id: 43832
          phid: PHID-ATOM-ht66qwou5x77ju2suyct
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: cla
     atomIndex: 0
  identityHash: y7ba1fNTnKX2
     graphHash: NULL
         title: Understanding the Phacility CLA
 titleSlugHash: OLDz4wWEP21U
     groupName: detail
       summary: Describes the Contributor License Agreement (CLA).
isDocumentable: 1
      nodeHash: NULL
*************************** 6. row ***************************
            id: 4753
          phid: PHID-ATOM-uiol2fpa5tkuvar2zagt
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: contrib_intro
     atomIndex: 0
  identityHash: An8xcOlLVnE.
     graphHash: NULL
         title: Contributor Introduction
 titleSlugHash: 7PnUbyyXo5Zq
     groupName: contrib
       summary: Introduction to contributing to Phabricator, Arcanist and libphutil.
isDocumentable: 1
      nodeHash: NULL
*************************** 7. row ***************************
            id: 31930
          phid: PHID-ATOM-sreg53wenuulmgrteemv
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: contributing_code
     atomIndex: 0
  identityHash: lIAaFrBOkQv6
     graphHash: NULL
         title: Contributing Code
 titleSlugHash: xEOc_DP4BKCY
     groupName: detail
       summary: Describes how to contribute code to Phabricator.
isDocumentable: 1
      nodeHash: NULL
*************************** 8. row ***************************
            id: 26121
          phid: PHID-ATOM-wlm2mv4rp4ju73xoavjo
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: css_coding_standards
     atomIndex: 0
  identityHash: droWM.tI.NRs
     graphHash: NULL
         title: CSS Coding Standards
 titleSlugHash: O.XUeBOGMS6W
     groupName: standards
       summary: This document describes CSS features and coding standards for Phabricator.
isDocumentable: 1
      nodeHash: NULL
*************************** 9. row ***************************
            id: 4749
          phid: PHID-ATOM-ssculwqmqscbq66o54r6
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: darkconsole
     atomIndex: 0
  identityHash: .zPsYM__vQnW
     graphHash: NULL
         title: Using DarkConsole
 titleSlugHash: ljfHscF_HY9T
     groupName: developer
       summary: Enabling and using the built-in debugging console.
isDocumentable: 1
      nodeHash: NULL
*************************** 10. row ***************************
            id: 4761
          phid: PHID-ATOM-qdf4ifqw4vpgerzclh6y
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: database
     atomIndex: 0
  identityHash: AVXWtW4q1qKq
     graphHash: NULL
         title: Database Schema
 titleSlugHash: FToeyMo9t_ee
     groupName: developer
       summary: This document describes key components of the database schema and should answer
questions like how to store new types of data.
isDocumentable: 1
      nodeHash: NULL
*************************** 11. row ***************************
            id: 44287
          phid: PHID-ATOM-f5uniosu254o7utqbyk5
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: describing_problems
     atomIndex: 0
  identityHash: Wo0hoo3.Aq3U
     graphHash: NULL
         title: Describing Root Problems
 titleSlugHash: Y1Jua8bDkNC9
     groupName: detail
       summary: Explains how to describe a root problem effectively.
isDocumentable: 1
      nodeHash: NULL
*************************** 12. row ***************************
            id: 56143
          phid: PHID-ATOM-fqdnlix7lzz6vg3xbtdn
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: developer_setup
     atomIndex: 0
  identityHash: mluxgdu4CJgX
     graphHash: NULL
         title: Developer Setup
 titleSlugHash: js8RkmrE6TL7
     groupName: developer
       summary: How to configure a Phabricator development environment.
isDocumentable: 1
      nodeHash: NULL
*************************** 13. row ***************************
            id: 31926
          phid: PHID-ATOM-6zd3vxwjqahkaxu5smai
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: feature_requests
     atomIndex: 0
  identityHash: .g6KOuIHxgK9
     graphHash: NULL
         title: Contributing Feature Requests
 titleSlugHash: 2vSvitpr9yks
     groupName: detail
       summary: Describes how to file an effective Phabricator feature request.
isDocumentable: 1
      nodeHash: NULL
*************************** 14. row ***************************
            id: 4765
          phid: PHID-ATOM-5v5qm4zb7s5whxitman7
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: general_coding_standards
     atomIndex: 0
  identityHash: fjocyvniU0Te
     graphHash: NULL
         title: General Coding Standards
 titleSlugHash: ng0X5zaXKC5L
     groupName: standards
       summary: This document is a general coding standard for contributing to Phabricator,
Arcanist, libphutil and Diviner.
isDocumentable: 1
      nodeHash: NULL
*************************** 15. row ***************************
            id: 32102
          phid: PHID-ATOM-l3xa3pq7hcq5lbgo6p4o
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: installing_xhprof
     atomIndex: 0
  identityHash: bdwyjYmfX4sY
     graphHash: NULL
         title: Installing XHProf
 titleSlugHash: 9YY5V9Z8Yr6s
     groupName: developer
       summary: Describes how to install XHProf, a PHP profiling tool.
isDocumentable: 1
      nodeHash: NULL
*************************** 16. row ***************************
            id: 4751
          phid: PHID-ATOM-6bedxl6fx4cmrltbbduf
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: internationalization
     atomIndex: 0
  identityHash: Jk.x4oqmPuFm
     graphHash: NULL
         title: Internationalization
 titleSlugHash: d5EKw_a3dZ02
     groupName: developer
       summary: Describes Phabricator translation and localization.
isDocumentable: 1
      nodeHash: NULL
*************************** 17. row ***************************
            id: 4747
          phid: PHID-ATOM-qlyfr753ktx2f52jrtut
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: javascript_coding_standards
     atomIndex: 0
  identityHash: bETHwHu33gGY
     graphHash: NULL
         title: Javascript Coding Standards
 titleSlugHash: QWLFTAt1BlDY
     groupName: standards
       summary: This document describes Javascript coding standards for Phabricator and Javelin.
isDocumentable: 1
      nodeHash: NULL
*************************** 18. row ***************************
            id: 4759
          phid: PHID-ATOM-du6bv7vldckbxmonodau
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: n_plus_one
     atomIndex: 0
  identityHash: vVjewtr8B3oC
     graphHash: NULL
         title: Performance: N+1 Query Problem
 titleSlugHash: Wi2QzZ3XWLAz
     groupName: developer
       summary: How to avoid a common performance pitfall.
isDocumentable: 1
      nodeHash: NULL
*************************** 19. row ***************************
            id: 4741
          phid: PHID-ATOM-rq4nwzv264xx5gqxxcvn
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: phabricator_code_layout
     atomIndex: 0
  identityHash: pXPgRpVz9.pb
     graphHash: NULL
         title: Phabricator Code Layout
 titleSlugHash: joIEI0Xe_DX0
     groupName: developer
       summary: Guide to Phabricator code layout, including how URI mapping works through
application class and subdirectory organization best practices.
isDocumentable: 1
      nodeHash: NULL
*************************** 20. row ***************************
            id: 4743
          phid: PHID-ATOM-b4xavicdpkkdzjev3elv
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: php_coding_standards
     atomIndex: 0
  identityHash: cgYSEXXDQF__
     graphHash: NULL
         title: PHP Coding Standards
 titleSlugHash: XkcmSVZAdrp2
     groupName: standards
       summary: This document describes PHP coding standards for Phabricator and related
projects (like Arcanist and libphutil).
isDocumentable: 1
      nodeHash: NULL
*************************** 21. row ***************************
            id: 4767
          phid: PHID-ATOM-yh7y6dneh22mz3jq4kh2
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: rendering_html
     atomIndex: 0
  identityHash: iXXKLEatMho7
     graphHash: NULL
         title: Rendering HTML
 titleSlugHash: HZByYx6igNsT
     groupName: developer
       summary: Rendering HTML in the Phabricator environment.
isDocumentable: 1
      nodeHash: NULL
*************************** 22. row ***************************
            id: 48152
          phid: PHID-ATOM-b4e6cgmcx5k76w3cazse
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: reproduction_steps
     atomIndex: 0
  identityHash: wg7elB63Gayo
     graphHash: NULL
         title: Providing Reproduction Steps
 titleSlugHash: YBZD3h70PZ5T
     groupName: detail
       summary: Describes how to provide reproduction steps.
isDocumentable: 1
      nodeHash: NULL
*************************** 23. row ***************************
            id: 4755
          phid: PHID-ATOM-ndgsecavxvi3rvst3vhk
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: running_builtin_php_webserver
     atomIndex: 0
  identityHash: WnAX8YnZZJBg
     graphHash: NULL
         title: Running built-in PHP webserver
 titleSlugHash: 3Z2mqW7wVct5
     groupName: developer
       summary: As of version 5.4.0, the PHP command line interface provides a built-in web
server. This web server is designed for developmental purposes only, and should
not be used in production. Phabricator can be executed under it with the
command:
isDocumentable: 1
      nodeHash: NULL
*************************** 24. row ***************************
            id: 4745
          phid: PHID-ATOM-ydu6a5pka6t4u4bufoqp
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: unit_tests
     atomIndex: 0
  identityHash: J6sgMvg5R5de
     graphHash: NULL
         title: Writing Unit Tests
 titleSlugHash: K77REyR9mJsx
     groupName: developer
       summary: Simple guide to libphutil, Arcanist and Phabricator unit tests.
isDocumentable: 1
      nodeHash: NULL
*************************** 25. row ***************************
            id: 4769
          phid: PHID-ATOM-7ihpynw7ry75rflmjro4
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: using_edges
     atomIndex: 0
  identityHash: Z58y4LGaXUWA
     graphHash: NULL
         title: Using Edges
 titleSlugHash: xzl0h88j6MLq
     groupName: developer
       summary: Guide to the Edges infrastructure.
isDocumentable: 1
      nodeHash: NULL
*************************** 26. row ***************************
            id: 4763
          phid: PHID-ATOM-x2tp4ecp5a4utwojchms
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: using_oauthserver
     atomIndex: 0
  identityHash: LsZePw6DbaDr
     graphHash: NULL
         title: Using the Phabricator OAuth Server
 titleSlugHash: wUJtbM_KToRI
     groupName: developer
       summary: How to use the Phabricator OAuth Server.
isDocumentable: 1
      nodeHash: NULL
*************************** 27. row ***************************
            id: 48154
          phid: PHID-ATOM-cfnlw6hutjjvbbwuwdo7
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: version
     atomIndex: 0
  identityHash: dBzzSRd.Rj.J
     graphHash: NULL
         title: Providing Version Information
 titleSlugHash: .QQLc.8QAvjX
     groupName: detail
       summary: How to provide version information with reports made to the upstream.
isDocumentable: 1
      nodeHash: NULL
*************************** 28. row ***************************
            id: 38337
          phid: PHID-ATOM-3vtxkm33ihzu746sjcnd
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/adding_new_classes.diviner
     atomIndex: 0
  identityHash: Va_7IgNL_VCH
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 29. row ***************************
            id: 4756
          phid: PHID-ATOM-4zn2wfbwikhr244msgn5
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/adding_new_css_and_js.diviner
     atomIndex: 0
  identityHash: XmsxG8x8Xp_6
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 30. row ***************************
            id: 26561
          phid: PHID-ATOM-3xpkvhgrfue3z2unh452
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/assistive_technologies.diviner
     atomIndex: 0
  identityHash: cOfPZDtH2CjY
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 31. row ***************************
            id: 31927
          phid: PHID-ATOM-xb7re767wx6jqmm3crh4
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/bug_reports.diviner
     atomIndex: 0
  identityHash: Cbx2_D.gT.ha
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 32. row ***************************
            id: 43831
          phid: PHID-ATOM-sormtuky4g4z6gqs32ec
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/cla.diviner
     atomIndex: 0
  identityHash: QvigniZuRJXT
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 33. row ***************************
            id: 4752
          phid: PHID-ATOM-lz2tu22lzsta4wdnrmzj
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/contrib_intro.diviner
     atomIndex: 0
  identityHash: QuncvVJuVMHX
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 34. row ***************************
            id: 31929
          phid: PHID-ATOM-hqpbz4giwluvtjnjh6zg
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/contributing_code.diviner
     atomIndex: 0
  identityHash: DM6dcsLMXw9E
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 35. row ***************************
            id: 26120
          phid: PHID-ATOM-asjjpq6dhelkwwdy2ehz
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/css_coding_standards.diviner
     atomIndex: 0
  identityHash: qfElwo117oJw
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 36. row ***************************
            id: 4748
          phid: PHID-ATOM-itb7qkxz7pqnlg5qtue3
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/darkconsole.diviner
     atomIndex: 0
  identityHash: sBz0oc1PSwOe
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 37. row ***************************
            id: 4760
          phid: PHID-ATOM-hkvltvwrhw65bc7x3kad
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/database.diviner
     atomIndex: 0
  identityHash: QbUewAItiapX
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 38. row ***************************
            id: 44286
          phid: PHID-ATOM-ddwspbszqvt4sdgokuoa
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/describing_problems.diviner
     atomIndex: 0
  identityHash: cE3wgNOoPk8y
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 39. row ***************************
            id: 56142
          phid: PHID-ATOM-stfex5kqniwtcdtnbh4f
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/developer_setup.diviner
     atomIndex: 0
  identityHash: gd0CWvn8dLFh
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 40. row ***************************
            id: 31925
          phid: PHID-ATOM-4xwy2o3tly3lvmgbtknn
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/feature_requests.diviner
     atomIndex: 0
  identityHash: iCB38aAGtYWL
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 41. row ***************************
            id: 4764
          phid: PHID-ATOM-j3rzv7wivzzhbe5sound
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/general_coding_standards.diviner
     atomIndex: 0
  identityHash: qoFEfK9QNNwb
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 42. row ***************************
            id: 32101
          phid: PHID-ATOM-tcmy4iuum7fmg2yn6bau
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/installing_xhprof.diviner
     atomIndex: 0
  identityHash: cY6SOF9B86e0
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 43. row ***************************
            id: 4750
          phid: PHID-ATOM-ntlgwgik7fszphi47l4g
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/internationalization.diviner
     atomIndex: 0
  identityHash: 6DLiHZYXMhVz
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 44. row ***************************
            id: 4746
          phid: PHID-ATOM-t377jz6rgexi4dddkogz
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/javascript_coding_standards.diviner
     atomIndex: 0
  identityHash: _jIlOB0lTqyM
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 45. row ***************************
            id: 4758
          phid: PHID-ATOM-db6e57uc3t4olsuh4x34
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/n_plus_one.diviner
     atomIndex: 0
  identityHash: wmq0v7vboWYH
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 46. row ***************************
            id: 4740
          phid: PHID-ATOM-4ouqdjw736aa4ey46ai6
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/phabricator_code_layout.diviner
     atomIndex: 0
  identityHash: EhVrm3yg6in9
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 47. row ***************************
            id: 4742
          phid: PHID-ATOM-4pxzmybctjoka7kwjwcu
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/php_coding_standards.diviner
     atomIndex: 0
  identityHash: abU8f19aR8hC
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 48. row ***************************
            id: 4766
          phid: PHID-ATOM-r4amsvmsgrdfprdjcd7y
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/rendering_html.diviner
     atomIndex: 0
  identityHash: rRTwCKjSzU8X
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 49. row ***************************
            id: 48151
          phid: PHID-ATOM-oarfbgqflakcspzlmf5u
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/reproduction_steps.diviner
     atomIndex: 0
  identityHash: LXxy12KGcOjc
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 50. row ***************************
            id: 4754
          phid: PHID-ATOM-5e6fica5dfilywepxipa
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/running_builtin_php_webserver.diviner
     atomIndex: 0
  identityHash: teNAbBJsy75W
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 51. row ***************************
            id: 4744
          phid: PHID-ATOM-xjx2hf3yh2mv7ps7zprr
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/unit_tests.diviner
     atomIndex: 0
  identityHash: sMBZNGmyLvz5
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 52. row ***************************
            id: 4768
          phid: PHID-ATOM-iqepzfhsnygnjn3slyk2
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/using_edges.diviner
     atomIndex: 0
  identityHash: Zm99jazcznxJ
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 53. row ***************************
            id: 4762
          phid: PHID-ATOM-p2eldvd55nhemailcxc3
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/using_oauthserver.diviner
     atomIndex: 0
  identityHash: PwqyoA0uZgZc
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL
*************************** 54. row ***************************
            id: 48153
          phid: PHID-ATOM-6f5uvnuxnp36dt7p5t6j
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/contributor/version.diviner
     atomIndex: 0
  identityHash: 0TLPGp5YJKc_
     graphHash: NULL
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: NULL

But note that the graphHash and nodeHash columns are 0, which I believe is a signal that these nodes have been deleted (I haven't touched this code in a while). Compare to non-deleted documents:

mysql> select * from diviner_livesymbol where type = 'article' and name like '%contrib%' OR name like '%upgrading%'\G
*************************** 1. row ***************************
            id: 4753
          phid: PHID-ATOM-uiol2fpa5tkuvar2zagt
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: contrib_intro
     atomIndex: 0
  identityHash: An8xcOlLVnE.
     graphHash: NULL
         title: Contributor Introduction
 titleSlugHash: 7PnUbyyXo5Zq
     groupName: contrib
       summary: Introduction to contributing to Phabricator, Arcanist and libphutil.
isDocumentable: 1
      nodeHash: NULL
*************************** 2. row ***************************
            id: 31930
          phid: PHID-ATOM-sreg53wenuulmgrteemv
      bookPHID: PHID-BOOK-ux3r2a54misiq6swn7ze
repositoryPHID: 
       context: NULL
          type: article
          name: contributing_code
     atomIndex: 0
  identityHash: lIAaFrBOkQv6
     graphHash: NULL
         title: Contributing Code
 titleSlugHash: xEOc_DP4BKCY
     groupName: detail
       summary: Describes how to contribute code to Phabricator.
isDocumentable: 1
      nodeHash: NULL
*************************** 3. row ***************************
            id: 38780
          phid: PHID-ATOM-a3o3mgcctpgbdk2xdslp
      bookPHID: PHID-BOOK-pptahmjhs6sq6vl4ybyu
repositoryPHID: 
       context: NULL
          type: file
          name: src/docs/user/upgrading.diviner
     atomIndex: 0
  identityHash: JwBTOrHX9jVl
     graphHash: 6e60d328189a0bf765b214771188ee0dG
         title: NULL
 titleSlugHash: NULL
     groupName: NULL
       summary: 
isDocumentable: 0
      nodeHash: c6da64b795341fe3e3b9743ce19ff7eaN
*************************** 4. row ***************************
            id: 38781
          phid: PHID-ATOM-dgaw6rcpi5hh2elorcgj
      bookPHID: PHID-BOOK-pptahmjhs6sq6vl4ybyu
repositoryPHID: 
       context: NULL
          type: article
          name: upgrading
     atomIndex: 0
  identityHash: wsOtX.FXDyu3
     graphHash: 5d7774509e25f3e152b976e80f03e999G
         title: Upgrading Phabricator
 titleSlugHash: jfaTT.kwrESg
     groupName: intro
       summary: This document contains instructions for keeping Phabricator up to date.
isDocumentable: 1
      nodeHash: 8061759a1cbe847ee9e0e120e74e75f8N
4 rows in set (0.00 sec)
epriestley added a comment.Oct 5 2018, 3:21 PM

But note that the graphHash and nodeHash columns are 0, which I believe is a signal that these nodes have been deleted (I haven't touched this code in a while).

This looks right from the code.

foreach (PhabricatorLiskDAO::chunkSQL($strings, ', ') as $chunk) {
  queryfx(
    $conn_w,
    'UPDATE %T SET graphHash = NULL, nodeHash = NULL
      WHERE graphHash IN (%Q)',
    $symbol_table->getTableName(),
    $chunk);
}
epriestley added a comment.Oct 5 2018, 3:26 PM

So, so far it looks like the database is in a "good" state but the publish workflow incorrectly deleted all the articles. From previous efforts, I believe dropping the cache fixes this issue.

I stopped DivinerPublisher from doing any meaningful work (maybe) by making it die before delete/create, and ran $ ./bin/diviner generate --book src/docs/book/contributor.book . I think it still does "atomize" work in this state, but it doesn't look like that's destroying my reproduction case quite yet. With a couple of added var_dump() calls:

ubuntu@secure001:/core/lib/phabricator$ ./bin/diviner generate --book src/docs/book/contributor.book 
Generating book "contributor.book"...
BUILDING ATOM CACHE
Found 8,262 files in project.
Found 8,237 unatomized, uncached files.
Found 0 files to atomize.
Atomizing 0 files.
Atom cache is up to date, no files to atomize.
Writing atom cache.
Done.

BUILDING GRAPH CACHE
Found 0 obsolete atoms in graph.
Found 0 new atoms in graph.
Propagating changes through the graph.
Found 0 affected atoms.
Writing graph cache.
Done.

PUBLISHING DOCUMENTATION
array(0) {
}
array(0) {
}
Okay, publish.

I'll dig into what atomization is doing in more detail.

epriestley added a comment.Oct 5 2018, 3:36 PM
  • DivinerPublisher->publishAtoms() is being passed nothing, so it's correctly deleting everything.
  • The graph cache is empty.
  • The actual atom.cache file on disk is empty.
  • The input to the cache is empty.
  • The files aren't being passed to the atomizer rules (although the rules appear to be applying correctly).
epriestley added a comment.EditedOct 5 2018, 4:03 PM

The file cache says that these files have already been parsed and contain no atoms, so the rest of the program behavior is expected after that. I'm not sure how this state came to exist.

I tried a general regen like this:

./bin/diviner generate --clean

...and hit this:

[2018-10-05 16:00:23] ERROR 2: unlink(/core/lib/phabricator/.divinercache/phabdev/atom/00/0b/dd8219a3.cache): Permission denied at [/core/lib/libphutil/src/filesystem/Filesystem.php:349]
arcanist(head=master, ref.master=2650e8627a20), libcore(), phabricator(head=master, ref.master=cfd9fa7f559f), phutil(head=master, ref.master=a0b90a12cee9), secure(head=master, ref.master=988cf9bd7958), services(head=master, ref.master=019a12a6556d)
  #0 unlink(string) called at [<phutil>/src/filesystem/Filesystem.php:349]
  #1 Filesystem::executeRemovePath(string) called at [<phutil>/src/filesystem/Filesystem.php:340]
  #2 Filesystem::executeRemovePath(string) called at [<phutil>/src/filesystem/Filesystem.php:340]
  #3 Filesystem::executeRemovePath(string) called at [<phutil>/src/filesystem/Filesystem.php:340]
  #4 Filesystem::executeRemovePath(string) called at [<phutil>/src/filesystem/Filesystem.php:340]
  #5 Filesystem::executeRemovePath(string) called at [<phutil>/src/filesystem/Filesystem.php:302]
  #6 Filesystem::remove(string) called at [<phutil>/src/cache/PhutilDirectoryKeyValueCache.php:150]
  #7 PhutilDirectoryKeyValueCache::destroyCache() called at [<phutil>/src/cache/PhutilKeyValueCacheProxy.php:36]
  #8 PhutilKeyValueCacheProxy::destroyCache() called at [<phabricator>/src/applications/diviner/cache/DivinerDiskCache.php:21]
  #9 DivinerDiskCache::delete() called at [<phabricator>/src/applications/diviner/cache/DivinerAtomCache.php:20]
  #10 DivinerAtomCache::delete() called at [<phabricator>/src/applications/diviner/workflow/DivinerGenerateWorkflow.php:92]
  #11 DivinerGenerateWorkflow::generateBook(string, PhutilArgumentParser) called at [<phabricator>/src/applications/diviner/workflow/DivinerGenerateWorkflow.php:80]
  #12 DivinerGenerateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:457]
  #13 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:349]
  #14 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/diviner/diviner.php:21]

The file was owned by root:root. This is in a different book, but maybe there's a race or sequencing issue with the cache writes? Feels like a long shot.

It looks like some of this may be entagled with root having generated documentation at some point (this unlink is failing because the file is owned by root + root), plus poor recovery from ownership issues.

epriestley added a comment.Oct 5 2018, 4:15 PM

For now, I just hard-wiped the cache and regenerated the documentation again. ¯\_(ツ)_/¯

Pretty sure this still isn't actually fixed.

joshuaspence added a subscriber: joshuaspence.Oct 16 2018, 8:56 AM
avivey mentioned this in Z1336: General Chat.Oct 31 2018, 3:39 AM
epriestley added a comment.Oct 31 2018, 3:21 PM

This cropped up again and the permissions error also resurfaced, even though I previously nuked the directory.

I think the root cause of this is actually just very mundane: we have the crontab installed for both users root and ubuntu. We must then have some poor error recovery behavior in the presence of root-owned cache files.

I removed the root copy of the crontab and am rebuilding the documentation now.

epriestley added a comment.Oct 31 2018, 3:42 PM

The immediate issue now appears fixed and it seems likely that the root cause is also fixed. We should still hunt down the bad cache behavior in the presence of wonky permissions.

epriestley renamed this task from User documentation on "secure" is occasionally unpublishing to Diviner publishing can fail resoundingly with unusual file permissions [was: User documentation on "secure" is occasionally unpublishing].Dec 17 2018, 3:45 PM
epriestley lowered the priority of this task from Normal to Low.
Phabricator · Built by Phacility