groovy - Why are my Elasticsearch shards sporadically failing when running queries against my alias filter? -
i running elasticsearch v.1.6 setup 2 nodes, , index 5 shards. index, have added alias contains reference index, groovy script filter helps avoiding documents invalid start , end dates. alias looks this:
"myindex" : { "aliases": { "myindex_alias": { "filter" : { "script":{ "script": "\r\n entrydate = doc['availability.start'].date.getmillis(); \r\n exitdate = doc['availability.end'].date.getmillis(); \r\n rightnow = datemidnight.now().getmillis(); \r\n\r\n if (entrydate < rightnow && exitdate > rightnow)\r\n { \r\n return true; \r\n } \r\n return false;", "lang":"groovy"}}}}}
when performing simple query, such term * (/myindex_alias/my_type/_search?request.q=*), number of shards fail query returning following message:
"failed": 1, "failures": [ { "index": "myindex", "shard": 2, "status": 500, "reason": "remotetransportexception[[myescluster][inet[/10.131.43.61:9301]][indices:data/read/search[phase/query]]]; nested: queryphaseexecutionexception[[myindex][2]: query[constantscore(scriptfilter( entrydate = doc['availability.start'].date.getmillis(); exitdate = doc['availability.end'].date.getmillis(); rightnow = datemidnight.now().getmillis(); if (entrydate < rightnow && exitdate > rightnow) { return true; } return false;))],from[0],size[10]: query failed [failed execute main query]]; nested: groovyscriptexecutionexception[arrayindexoutofboundsexception[null]]; " } ]
i have verified following things:
- all documents in index have valid start , end values, , none of them have null value.
- the filter does work on shards perform query successfully
- the cluster has green health status
- the error occurs randomly on shards both in master , worker nodes
- the errors not occur if queries directed index instead of alias
- adding more nodes cluster seems mitigate problem. running 2 nodes leaves 2-3 shards failing, while 3 nodes limits failure 1-2 shards. adding final 4th node leaves query failing on single shard
- even when query succeeds, fewer results expected, again sporadically
any on matter appreciated!
Comments
Post a Comment