Below is my cookbook on setting up phabricator and elasticsearch; task data has over 77k tasks entries. It is my assumption and limited understanding that the data from phabricator doesn't get pushed over unless I run daemons (specifically PhabricatorTaskmasterDaemon) even though I manually ran `bin/search index` but the problem is the daemons consume all 16G of memory and of course this causes very slow or failed webserver response and eventually insert fails as well.
So, what is the correct procedure so can I stop the daemons initially, store/load the data; then force phabricator to trigger data into the queue or whatever magical to pass the indexing over to elasticsearch quickly? My goal is then while in production, stop daemons from running and later possible have a secondary server to handle indexing/inserting batch and another for reading. It seems silly I need a 32G+ Ram Memory or more to handle just less than 100,000 index tasks for searching and retrieval.
The setup will work very slowly. It seems to index but it's super super slow. Meaning it takes close to 1 hour to index just 70 tasks. I feel blind too, because I don't know really how to debug the slowness problem.
So how do I do the following:
1. clear out the 'index queue'; it seems to build up somewhere and if I recreate ubuntu and copy over db; it doesn't start at T-1., currently working at T31396 and not sure why this is the case.
2. determine what task(s) it is on for queuing
3. What logs (commands) allows me to have sight in what is being processed, right now i'm blind with `phd start` or `phd debug <Daemon>`
3. configure index to work faster under the 16G limitation
-----
Below is the suggested configuration values being used; however, I changed the follow
= Hardware Configuration =
16G Ram Memory
1G SSD
Ubuntu 10.04 x86_64
= Software Version =
elasticsearch-1.5.2.deb
**arcanist**
commit 9b7c6786cdc9694b45907ecb46154befbd605a0d
Date: Sun May 24 07:45:01 2015 +1000
**libphutil**
commit 4e0febf56d83d917558606c06ca72cdc42524566
Date: Sun May 24 06:37:12 2015 -0700
**phabricator**
commit 18fe6d58aeba41c6ac17e41c6f374ea429668e6a
Date: Sun May 24 11:22:33 2015 -0700
=top output before phd=
```
KiB Mem: 16120708 total, 3012396 used, 13108312 free, 42992 buffers
KiB Swap: 16457724 total, 1037176 used, 15420548 free. 801372 cached Mem
```
=max task count=
```./bin/search init```
```
Index exists but is incorrect, fixing...done.
```
NOTE: @mapping.json directly copied from {T6552#86240}
```
bin/phd stop --force
curl -XDELETE http://localhost:9200/phabricator?pretty=1
curl -XPOST 'http://localhost:9200/_optimize?only_expunge_deletes=true'
curl -XDELETE 'http://localhost:9200/_template/template_phabricator'
curl -XPUT 'http://localhost:9200/_template/template_phabricator' -d @mapping.json
curl -XGET http://localhost:9200/phabricator
curl -XGET http://localhost:9200/_aliases?pretty=1
curl http://localhost:9200/_cluster/health?pretty=true
cd /var/www/phabricator
bin/search init
bin/search index --all --background
curl http://localhost:9200/phabricator/_stats?pretty=1
curl http://localhost:9200/_cluster/health?pretty=true
```
```
Run this workflow with "--background" to queue tasks for the daemon workers.
Indexing 77281 object of type TASK.
Indexing 94 object of type PROJ.
Indexing 2 object of type USER.
Indexing 1 object of type MOCK.
```
After completion:
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 575,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
```
=unresolved issues=
* All other issues are based on phabricator configuration suggestions
{F428959}
```
bin/phd start
Freeing active task leases...
Freed 11 task lease(s).
Launching daemons:
(Logs will appear in "/var/tmp/phd/log/daemons.log".)
PhabricatorRepositoryPullLocalDaemon (Static)
PhabricatorTriggerDaemon (Static)
PhabricatorTaskmasterDaemon (Autoscaling: group=task, pool=4, reserve=0)
Done.
```
= top after phd =
```
KiB Mem: 16120708 total, 3293332 used, 12827376 free, 263752 buffers
KiB Swap: 16457724 total, 1775912 used, 14681812 free. 878932 cached Mem
```
== elastic search ==
After 30 minutes....
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 18,
"deleted" : 9
},
"store" : {
"size_in_bytes" : 207056,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 43,
```
after 1 hour
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 30,
"deleted" : 16
},
"store" : {
"size_in_bytes" : 323143,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 71,
```
after 2 hour
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 30,
"deleted" : 16
},
"store" : {
"size_in_bytes" : 323143,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 147,
```