Change Details

Hello, we started sending phabricator events to elasticsearch. We noticed that some timestamp fileds contain unix epoch values in seconds. Additionally, <_timestamp> and <dateCreated> fields are mapped as "string" type and <when> fields mapped as "long" type. Let me suggest the following changes: 1. fields that contain timestamp fields to be mapped as "date" type fields 2. unix epoch timestamps to be encoded in milliseconds and not in seconds. That way, an elasticsearch user who wants to visualize these events in Kibana/Grafana will not need to modify the date format of the field. Also, epoch_second format is only supported in Elasticsearch 2 and above. That way, theoretically, there will be no need for changing the mapping of phabricator index types (TASK, DREV, CMIT) Best regards, Kostis

Hello, our goal is to visualize phabricator events using Kibana and/or Grafana using elasticsearch as a data source. Phabricator provides the functionality to feed elasticsearch with events, so we have configured search.elastic.host and search.elastic.namespace, and events started populating the phabricator index. However, we are not capable of visualizing the events in the aforementioned tools. After thorough investigation, we noticed that the timestamp fields that we selected as timepickers (dateCreated and _timestamp to be precise) contain unix epoch values in seconds. Additionally, <_timestamp> and <dateCreated> fields are mapped as "string" type after indexing and <when> fields mapped as "long" type. These conditions make elasticsearch ignore these fields as timestamps and the aforementioned tools unable to parse datetime information out of them. Let me suggest the following changes: 1. ensure that fields that contain timestamp fields to be mapped as "date" type fields in Elasticsearch. Especially for dateCreated this seems to be hardcoded in src/applications/search/engine/PhabricatorElasticSearchEngine.php 2. unix epoch timestamps to be converted in a built-in ISO date format that is recognized by current stable Elasticsearch releases. That way, an elasticsearch user who wants to visualize these events in Kibana/Grafana will not need to modify the date format of the field. Notice that some built-in date formats like epoch_second and epoch_millis formats are only supported in Elasticsearch 2 and above. That way, theoretically, there will be no need for changing the mapping of phabricator index types (TASK, DREV, CMIT) and elasticsearch will index automatically the events with the correct date types. [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html Best regards, Kostis

Hello, we started sending phabricator events to elasticsearch.our goal is to visualize phabricator events using Kibana and/or Grafana using elasticsearch as a data source. Phabricator provides the functionality to feed elasticsearch with events, so we have configured search.elastic.host and search.elastic.namespace, and events started populating the phabricator index. However, we are not capable of visualizing the events in the aforementioned tools. After thorough investigation, Wwe noticed that somthe timestamp filedselds that we selected as timepickers (dateCreated and _timestamp to be precise) contain unix epoch values in seconds. Additionally, <_timestamp> and <dateCreated> fields are mapped as "string" type after indexing and <when> fields mapped as "long" type. <_timestamp> and <dateCreated> fields are mapped as "string" type and <when> fields mapped as "long" typeThese conditions make elasticsearch ignore these fields as timestamps and the aforementioned tools unable to parse datetime information out of them. Let me suggest the following changes: 1. ensure that fields that contain timestamp fields to be mapped as "date" type fields in Elasticsearch. Especially for dateCreated this seems to be hardcoded in src/applications/search/engine/PhabricatorElasticSearchEngine.php 2. unix epoch timestamps to be encodconverted in milliseconds and not in secondsa built-in ISO date format that is recognized by current stable Elasticsearch releases. That way, an elasticsearch user who wants to visualize these events in Kibana/Grafana will not need to modify the date format of the field. Notice that some built-in date formats like epoch_second and epoch_millis formats are only supported in Elasticsearch 2 and above. That way, theoretically, there will be no need for changing the mapping of phabricator index types (TASK, AlsoDREV, epoch_second format is only supported in Elasticsearch 2 and above.CMIT) and elasticsearch will index automatically the events with the correct date types. [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html That way, theoretically, there will be no need for changing the mapping of phabricator index types (TASK, DREV, CMIT) Best regards, Kostis