Project

General

Profile

Actions

Bug #44

closed

Importing of datasets is too slow

Added by Anonymous over 12 years ago. Updated about 12 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Start date:
2011-10-27
Due date:
% Done:

0%

Estimated time:

Description

Importing of datasets takes way too much time. For importing a Site which has high resolution sky emission templates it takes around 10 minutes.

Actions #1

Updated by Anonymous over 12 years ago

  • Status changed from In Progress to Resolved

The problem was that via JPA the creation of the objects was too slow. This was due to constraints in the database and the fact that JPA was not optimized for batch. Because only the second can be fixed from JPA, I changed the population of the data of the dataset to happen using directly JDBC by temporarly droping the constraints and using SYSCS_IMPORT_DATA to import the data from a file. This speed up the process around 10 times and now it is acceptable.

Actions #2

Updated by Anonymous over 12 years ago

  • Status changed from Resolved to In Progress

Reopened this issue, as the fix of dropping and recreating the constraints turned to be very slow in databases with big number of datasets. More investigation is needed.

Actions #3

Updated by Anonymous over 12 years ago

After some tweaking on the database and the JPA classes the performance is now bearable, but still quite long. These tweaks will be available from version 0.3.1. I leave the bug open because a better solution must be implemented. A good solution could be to store the data as a BLOB.

Actions #4

Updated by Anonymous over 12 years ago

  • Status changed from In Progress to Resolved

Spend the time and implement the solution of storing the data in the database as a BLOB. This speeds up things around 60 times from the initial performance, in any size of database. Also, this modification guarantees that there will be no bottleneck if in a database there are thousands of datasets. This fix will be in version 0.3.2.

Actions #5

Updated by Anonymous about 12 years ago

  • Status changed from Resolved to Closed

Closing the bug, as the speed of the dataset importing is confirmed to be fast enough.

Actions

Also available in: Atom PDF