java - Apache Spark and Spring Transaction Management -

i using apache spark in clustered environment 3 workers. spark's 'foreachpartition', sending data in batches spring jdbctemplate.batchupdate in transaction mode.

i dont have idea spring transactions. spring transaction using below code :

defaulttransactiondefinition paramtransactiondefinition = new      defaulttransactiondefinition();         transactionstatus status =  transactionmanager.gettransaction(paramtransactiondefinition );  //sql , jdbcargs preparation... //.... jdbctemplate.batchupdate(finalsql, jdbcargs);             transactionmanager.commit(status); 

above code getting called :

.foreachpartition(...params..) {    call(..params..) {       if(basicdatasource == null || transactionmanager == null) {          basicdatasource bds = getbasicdatasource(basicdatasrc,                                                istransactionrequired);//for transaction           transactionmanager2.setdatasource(bds);//datasourcetransactionmanager           jdbctemplate.setbasicdatasource(bds);          // initializing datasource transactionmanager , jdbctemplate both because read somewhere connection object instance should same transactionmanager , jdbctemplate.       }//end of if                // here call above transaction , jdbctemplate.batchupdate code goes    }//end of call    private basicdatasource getdatasource() {         basicdatasource basicds = new basicdatasource();         basicds.setusername(sparkds.getusername());         basicds.setpassword(sparkds.getpassword());         basicds.seturl(sparkds.geturl());         basicds.setdriverclassname(sparkds.getdriverclassname());         basicds.setmaxactive(sparkds.getmaxactive());         basicds.setdefaultautocommit(false);    }//end of getdatasource   } 

i not closing connections here, saw in jdbctemplate code connections released spring itself. now, issue : above code of transaction doesn't write complete data db. when don't use transaction, there isn't problem.

my question is, above right way of creating datasource , transaction ? because above code gets executed on 3 workers 40 partitions. though basicdatasource doesn't gets created 40 times getting created 5-8 times.

can please me here.