span8
span4
span8
span4
Hi,
I need to read approx.1.4M records within a single workbench process.No getting around it.They must all be read since they are participating in a FeatureMerger later and any one of the features might be a candidate.
The reader starts out fine reading 2500 records a time under 10 seconds a pop.However this slowly degrades so that after 800K records it is now 30 sec.a pop and near the end close to 50.Soup to nuts,the entire read takes about 4 hrs and 45 min.RAM/Network speed really isn't a problem.So the gradual increase of record processing time seems odd.
Is there any tip/trick for reading a large number of records (>300K ) from PostGIS (or any) reader so that the amount of time is significantly reduced?I was thinking of amping up the "Number of Records To Fetch At A Time" but I don't want to make matters worse.
Thanks very much,
Pete
For more heavy jobs I prefer not to load all the data to workbench.Alternatives I use are:
- Only load the data you need.FeatureReaders with where statements and / or spatial intersects will help you dynamically load subsets of data.
- Let the database do the job,it will be quicker than anything else.Write the to be merged data to a temp table,dynamically write a join script and run it with the SQL executor to pick it up again.
Probably not the fastest to create next-next-finish workspace but problems often have more than one workable solution.As I have not too much patience I tend to rebuild the workspaces when data grows.When data volumes get too big you need to consider an alternative anyway.Try loading terrestrial laserscan data :-)
I'd also suggest trying the latest FME 2018 and try the FeatureJoiner instead of FeatureMerger.I'd suspect it would be much faster in all ways.
Are you running this process on 32bit or 64bit FME?On 32bit you would probably quickly run out of usable physical RAM which would significantly slow down your process.
Hi@pvaziri,have you considered using theDatabaseJoiner?Depending on if you need to do any manipulation of the PostGIS features prior to merging,this transformer may help.
I tried it@takashi.It is faster.However,in this case,the database table does not have geometry and the FGDB source does.So if I want to keep the geometry provided by the FGDB feature class,don't I have to always define it as the Requestor?
Hi@pvaziri,I suspect that your 亚搏在线workflow could consume a large memory space and it could cause decreasing the performance gradually.
If you send the features read from the PostGIS database to the Requestor port of the FeatureMerger,consider if you could apply the Supplier First option.If you could guarantee that all Supplier features would arrive in the FeatureMerger before the first Requestor feature would arrive,the option could be applied,then it could reduce memory usage and you could expect the performance would be increased.
加载一个大CityGML数据集2 Answers
"Join" SQL statement in PostGIS reader2 Answers
Holding an entire workbench4 Answers
How to replicate the summary / preview shown in a new Reader3 Answers
Reader Max Features to Read Per Feature Type not working4 Answers
© 2019 亚搏在线Safe Software Inc |Legal