Large CSV Files

dBase, Calc, CSV, MS ACCESS, MySQL, PostgrSQL, OTHER

Large CSV Files

Postby tomisurf » Thu Oct 25, 2012 6:11 pm

I have a set of extremely large CSV files, 4million plus lines each. Essentially they contain info on a vast number of sites, items that have been ordered and quantities etc. Each file represents a months worth of data.

I have been manipulating the data using a text editor, Ultraedit, which has been fine however I would like to query the data more effectively.

I have managed to import one CSV file into a Base DB by starting a new DB and connecting to an existing DB, a spreadsheet. I am not sure however if it is picking up all the data as if i run a query that should pick out a site id near to the bottom of the file i get nothing returned.

Im not sure if I am doing this correctly, copying and pasting data doesnt work, there is just too much of it. Am i just better off using the text editor or should i be able to import all the data in all the files into one database to query really effectively.

Thanks

Tom
OpenOffice 3.4.1 on Windows 7
tomisurf
 
Posts: 1
Joined: Thu Oct 25, 2012 6:04 pm

Re: Large CSV Files

Postby Villeroy » Thu Oct 25, 2012 6:31 pm

Use a true database such as MySQL or HSQLDB. Let it index the important fields. Then you may connect Base to that database if you really need those data in your office suite.

You may even try the built in HSQLDB as described in viewtopic.php?f=83&t=23260
Code: Select all   Expand viewCollapse view
CREATE TEXT TABLE "Import"(
  column definition 1,
  column definition 2,
  column definition 3,
and so on
);
SET "Import" SOURCE "blah.csv;with csv options"
Please, edit this topic's initial post and add "[Solved]" to the subject line if your problem has been solved.
Ubuntu 18.04, no OpenOffice, LibreOffice 6.4
User avatar
Villeroy
Volunteer
 
Posts: 28668
Joined: Mon Oct 08, 2007 1:35 am
Location: Germany


Return to External Data Sources

Who is online

Users browsing this forum: No registered users and 2 guests