History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: DOTCMS-1937
Type: Bug Bug
Status: Released Released
Resolution: Released
Priority: Critical Critical
Assignee: Testing User
Reporter: Maria Bouza
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
dotCMS

Import / Export not working with large files

Created: September 12, 2008 3:07 PM   Updated: July 23, 2009 11:53 AM
Component/s: a. Unknown
Affects Version/s: 1.6.0.10
Fix Version/s: 1.6.5

Time Tracking:
Original Estimate: Not Specified
Remaining Estimate: 0 minutes
Time Spent - 1 day, 5 hours
Time Spent: 1 day, 5 hours
Time Spent - 1 day, 5 hours


 Description  « Hide
We've been trying to export / import the Edu edition and it's not working with less than 4GB of RAM.
We need to come up with a better solution because Edu is not a large site.


 All   Comments   Work Log   Change History   Subversion Commits   FishEye      Sort Order: Ascending order - Click to sort in descending order
Jason Tesser - September 16, 2008 10:51 AM
OK So the problem is this. When you try to import a dotCMS backup if the backup is large over a couple hundred megs it fails. The reason it is failing is because during the unzip process we run out of memory.

Some notes about backup/restore process in dotCMS
1. When you restore you are blowing away all data and restore from the zip file.
2. The zip file is made up of assets and data serialized to XML
3. The first time dotCMS starts on a new dotCMS db it loads from a file named starter.zip which is the same as going to CMS Admin->Maintenance->restore
4. You can create the backup from the Maintenance screen mentioned above

OK so the problem is ImportExportUtil.validateZipFile(zipFile)

deleteTempFiles();
byte[] currentData = new byte[0];
FileInputStream is = new FileInputStream(zipFile);
int size = is.available();
currentData = new byte[size];
is.read(currentData);

File ftempDir = new File(tempdir);
ftempDir.mkdirs();
File tempZip = new File(tempdir + File.separator + zipFile.getName());
tempZip.createNewFile();
FileChannel ic = new FileInputStream(zipFile).getChannel();
FileChannel oc = new FileOutputStream(tempZip).getChannel();
ic.transferTo(0, ic.size(), oc);
ic.close();
oc.close();

We run out of memory. We need to find a way of unzipping the file so that even a 3 gig file can be unzipped with only a 1.5 gig jvm or so. So it needs to all be done in a stream. So that the zip file and the unzipped contents do not live in memory.

If needed we can look into a seperate zip library to handle this like http://www.jcraft.com/jzlib

Jason Tesser - September 16, 2008 11:09 AM
Ravi
On thing to mention is that we do all our development in the trunk. https://svn.dotmarketing.net/wwwroot/dotcms.org

VERY IMPORTANT!!!
WHENEVER YOU COMMIT YOU MUST PUT THE JIRAID ON THE SVN COMMENT LIKE SO
http://jira.dotmarketing.net/browse/DOTCMS-1937 or just DOTCMS-1937

So you only commit to trunk even though you can check out from other places and you always put the jira id which the code you are committing fixes.

I will get you an svn user for your team to share

Jason Tesser - September 16, 2008 11:09 AM
Also when the issue is done assign the issue back to me always. Thanks.