Below is my cookbook on setting up phabricator and task data which has over 77k tasks. The problem is the data from phabricator doesn't get pushed over unless I run daemons while initially inserting but the problem is the daemons consume all 16G of memory that I usually get slow or failed webserver response.
So, how can I stop the daemons initially, store the data; then trigger phabricator to queue or whatever magical to pass the indexing over to elasticsearch; then while in production, stop daemons from running and later possible have a secondary server to handle indexing/inserting batch and another for reading. It seems silly I need a 32G Ram Memory or more to handle just less than 100,000 index tasks for searching and retrieval. It does index but it's super super slow. Meaning it takes close to 1 hour to index just 70 tasks.
Please help. Stumped....
= Hardware Configuration =
16G Ram Memory
1G SSD
Ubuntu 10.04 x86_64
= Software Version =
elasticsearch-1.5.2.deb
**arcanist**
commit 9b7c6786cdc9694b45907ecb46154befbd605a0d
Date: Sun May 24 07:45:01 2015 +1000
**libphutil**
commit 4e0febf56d83d917558606c06ca72cdc42524566
Date: Sun May 24 06:37:12 2015 -0700
**phabricator**
commit 18fe6d58aeba41c6ac17e41c6f374ea429668e6a
Date: Sun May 24 11:22:33 2015 -0700
=top output before phd=
```
KiB Mem: 16120708 total, 3012396 used, 13108312 free, 42992 buffers
KiB Swap: 16457724 total, 1037176 used, 15420548 free. 801372 cached Mem
```
=max task count=
```./bin/search init```
```
Index exists but is incorrect, fixing...done.
```
```
bin/phd stop --force
curl -XDELETE http://localhost:9200/phabricator?pretty=1
curl -XPOST 'http://localhost:9200/_optimize?only_expunge_deletes=true'
curl -XDELETE 'http://localhost:9200/_template/template_phabricator'
curl -XPUT 'http://localhost:9200/_template/template_phabricator' -d @mapping.json
curl -XGET http://localhost:9200/phabricator
curl -XGET http://localhost:9200/_aliases?pretty=1
curl http://localhost:9200/_cluster/health?pretty=true
cd /var/www/phabricator
bin/search init
bin/search index --all --background
curl http://localhost:9200/phabricator/_stats?pretty=1
curl http://localhost:9200/_cluster/health?pretty=true
```
```
Run this workflow with "--background" to queue tasks for the daemon workers.
Indexing 77281 object of type TASK.
Indexing 94 object of type PROJ.
Indexing 2 object of type USER.
Indexing 1 object of type MOCK.
```
After completion:
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 0,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 575,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 0,
```
=unresolved issues=
* All other issues are based on phabricator configuration suggestions
{F428959}
```
bin/phd start
Freeing active task leases...
Freed 11 task lease(s).
Launching daemons:
(Logs will appear in "/var/tmp/phd/log/daemons.log".)
PhabricatorRepositoryPullLocalDaemon (Static)
PhabricatorTriggerDaemon (Static)
PhabricatorTaskmasterDaemon (Autoscaling: group=task, pool=4, reserve=0)
Done.
```
= top after phd =
```
KiB Mem: 16120708 total, 3293332 used, 12827376 free, 263752 buffers
KiB Swap: 16457724 total, 1775912 used, 14681812 free. 878932 cached Mem
```
== elastic search ==
After 30 minutes....
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 18,
"deleted" : 9
},
"store" : {
"size_in_bytes" : 207056,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 43,
```
after 1 hour
```
"indices" : {
"phabricator" : {
"primaries" : {
"docs" : {
"count" : 30,
"deleted" : 16
},
"store" : {
"size_in_bytes" : 323143,
"throttle_time_in_millis" : 0
},
"indexing" : {
"index_total" : 71,
```