We have an outside job working in our production on a daily basis. It calls the shell script, which in turn calls the Java servlet. This servlet reads these files and inserts data into two different tables, and then does some processing. The Java version is 1.6 and the application server is WAS7 and the database is oracel-11g.
We get a few problems with this process, as it takes time, goes out of memory, etc. Below are details of how we encoded this process. Please let me know if this can be improved.
When we read a file using BufferedReader, do we really get many lines created in memory that are returned by the readLine () method of BufferedReader? These files contain 4-5Lac lines. All entries are separated by a newline. Is there a better way to read files in java for performance? I could not find any of the facts that all the lines of the record in the file are of variable length.
When we insert data, we do a batch process with the statement / ready statement. We make one batch containing all the entries in the file. Does it matter to break the batch size for better performance?
If there are no pointers or any other restrictions in the tables, and all the columns are of the VARCHAR type, then what operation will be faster: - insert a new row or update an existing row based on some matching condition?