Bug #44
closed
Importing of datasets is too slow
Added by Anonymous over 12 years ago.
Updated over 12 years ago.
Description
Importing of datasets takes way too much time. For importing a Site which has high resolution sky emission templates it takes around 10 minutes.
- Status changed from In Progress to Resolved
The problem was that via JPA the creation of the objects was too slow. This was due to constraints in the database and the fact that JPA was not optimized for batch. Because only the second can be fixed from JPA, I changed the population of the data of the dataset to happen using directly JDBC by temporarly droping the constraints and using SYSCS_IMPORT_DATA to import the data from a file. This speed up the process around 10 times and now it is acceptable.
- Status changed from Resolved to In Progress
Reopened this issue, as the fix of dropping and recreating the constraints turned to be very slow in databases with big number of datasets. More investigation is needed.
After some tweaking on the database and the JPA classes the performance is now bearable, but still quite long. These tweaks will be available from version 0.3.1. I leave the bug open because a better solution must be implemented. A good solution could be to store the data as a BLOB.
- Status changed from In Progress to Resolved
Spend the time and implement the solution of storing the data in the database as a BLOB. This speeds up things around 60 times from the initial performance, in any size of database. Also, this modification guarantees that there will be no bottleneck if in a database there are thousands of datasets. This fix will be in version 0.3.2.
- Status changed from Resolved to Closed
Closing the bug, as the speed of the dataset importing is confirmed to be fast enough.
Also available in: Atom
PDF
