I have all of our jobs resubmit themselves when they error out using a topic/subscription on our scheduled tasks. Thanks for keeping us posted Bruce on the status of this. We have still been getting the 50x errors, but with our logic, I think it generally *should* work (although I haven't had much free time to verify).
Also thanks for sharing your workspace. Planning on taking a look at it when I get some time.
@runneals and @veenendaalo there are cases where AGOL responses are HTTP 200 status (OK) but the response body wraps an HTTP 50X error code, so the request retry logic in FME 2019.1 is not triggered. Without making any promises for Safe I know it is being addressed.
Thank you @bruceharold.
See the attached for some retrying readers. The 5 second wait may be a bit extreme.
I have to say that FME 2019.1 has improved the situation. Before I was getting maybe 7-10 out of 50 workspaces failing in my batch run and now its maybe 1 or 2. Still not solved the issue entirely. I did speak with ESRI regarding this and as @runneals mentions they state that a 'retry on failure logic' should be applied to get round this.
@bruceharold - thanks for the suggestions. Do you have an example of your 'use a FeatureReader in a custom transformer with looping from the Rejected port'.?
Is there also a way to increase the failure logic in 2019.1 from 3 to say 5?
I hear you David, and the issue is getting attention, but in the meantime 2019.1 should solve job failures.
@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.
@veenendaalo curious to know if you were able to test with FME 2019.1 or higher and the timeout fixes that @bruceharold mentioned above. FME 2019.1 (build 19610) and higher will now retry three times if HTTP 500/502/503/504 responses are encountered.
There is a bug open on ESRI BUG-000123780 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service intermittently returns 503 errors.
Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.
Last week there was an upgrade of AGOL and we have seen quite a few "random errors" the last couple of days (for example randomly not possible to create new accounts, users can not access their accounts etc). I am tempted to suggest AGOL might be the problem.
Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.