span8
span4
span8
span4
I am trying to query elasticsearch using fme to find the most recent date in a document.when I run it in kibana console as :
POST officeearthsearch/earthsearch/_search{ "_source": ["CHANGED_DATE"],"query": { "match_all": {} },"size": 1,"sort": [ { "CHANGED_DATE": { "order": "desc" } } ]}
it works and returns a single record containing the most recent changed date.
In the fme elasticsearch reader I input into the JSON query:
{ "query": { "match_all": {} },"size": 1,"sort": [ { "CHANGED_DATE": { "order": "desc" } } ]}
and get the following error:
FME API version of module 'ELASTICSEARCH' matches current internal version (3.8 20170315)
ELASTICSEARCH reader: Directive '_HOST' has value 'search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:80'
ELASTICSEARCH reader: Directive '_RAW_JSON' has value 'No'
ELASTICSEARCH reader: Directive '_NUM_FEATURES_PER_REQUEST' has value '1'
ELASTICSEARCH reader: Directive '_FME_JSON_ATTRIBUTE_FLATTENING_MODE' has value 'ALL_LEVELS'
ELASTICSEARCH reader: Opening index 'search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:80/officeearthsearch'
ELASTICSEARCH reader: Feature Type Parameter 'elasticsearch_query_document' has value '{
"query": {
"match_all": {}
},
"size": 1,
"sort": [
{
"CHANGED_DATE": {
"order": "desc"
}
}
]
}' for feature type 'earthsearch'
ELASTICSEARCH reader: HTTP Error 400-599: HTTP error code : 403
ELASTICSEARCH reader: {"message":"When Content-Type:application/x-www-form-urlencoded,URL cannot include query-string parameters (after '?'): '/officeearthsearch/earthsearch/_search?scroll=1m&size=1'"}
A fatal error has occurred.Check the logfile above for details
FME Session Duration: 2.4 seconds.(CPU: 0.3s user,0.2s system)
END - ProcessID: 10964,peak process memory usage: 67032 kB,current process memory usage: 48728 kB
Can you please suggest what I am doing wrong.
Many Thanks
Oliver
sorry,still no luck even with the parameter changed.workbench is herehttps://we.tl/TEAr5KcqFP
I am facing the same issue with both loading and reading data to and fro from elasticsearch in FME.Could you tell me what is the fix?I appreciate any help.
Hi@vkambham,
We made changes in the reader in build 18236 and you should no longer have the content issue.Please let me know if you are still having issue in the latest version.
Richard
I am having similar issues.I am using FME beta 2018.1 and used the Elasticsearch reader.The Elasticsearch (version 6.3) inside Amazon and is configured not to use any authentication.It works perfectly when I don't insert any JSON Query but when I put a simple query like
{"query":{"term":{"CountryOrTerritory":"Belgium"}}}
I get the following error.2018-06-22 14:40:32| 0.6| 0.0|ERROR |ELASTICSEARCH reader: HTTP Error 400-599: HTTP error code : 406
2018-06-22 14:40:32| 0.6| 0.0|ERROR |ELASTICSEARCH reader: {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
2018-06-22 14:40:32| 0.6| 0.0|ERROR |A fatal error has occurred.Check the logfile above for details
If I use an HTTPCaller and write a GET statement it works as expected.
/ieda/_search?q=CountryOrTerritory=Belgium
Another error I found is in the Elastic search reader and writer is that it doesn't like HTTP:// syntax.The document says you can use it but it doesn't work when you have something like
http://13.54.7.126:9201,It only works when you type13.54.7.126:9201
Jan
It looks like this issue is caused by Amazon Web Services acting as a proxy and changing the content type of the message which causes any query to fail.Were looking to see if there is anything FME can do to workaround this issue.
I am trying with 17.1.1.0 and couldnt get this to work,still the same error.Running elasticsearch 5.1.1.
ELASTICSEARCH reader: HTTP Error 400-599: HTTP error code : 403
ELASTICSEARCH reader: {"message":"When Content-Type:application/x-www-form-urlencoded,URL cannot include query-string parameters (after '?'): '/officeearthsearch/earthsearch/_search?scroll=1m&size=1'"}
A fatal error has occurred.Check the logfile above for details
Will try with FME b17717
I tried a stripped down version of your query and it seemed to work.If you can let me know which version of elasticsearch you are using and which version of FME I can find more details on syntax,but for now could you try the following syntax and use the " Features Per Request" parameter in the reader navigation -->Advanced parameters to limit the search to 1.
{
"query": { "match_all": {} },
"sort": { "CHANGED_DATE": { "order": "desc" } }
}
Write to ElasticSearch JSON directly4 Answers
how to sort files with dates and count them using json query to pass it as a httpcaller parameter1 Answer
Loop calls to an OData REST endpoint to download all JSON features2 Answers
HTTP call JSON parsing problem0 Answers
replace multiple json values1 Answer
© 2019 亚搏在线Safe Software Inc |Legal