scala - spark job cassandra error -


i getting error everytime running scala program in spark cassandra connector

exception during preparation of select count(*) "eventtest"."simpletbl" token("a") > ? , token("a") <= ? allow filtering: class org.joda.time.datetime in javamirror org.apache.spark.util.mutableurlclassloader@23041911 of type class org.apache.spark.util.mutableurlclassloader classpath [file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./spark-cassandra-connector_2.10-1.4.0-m1.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./cassandra-driver-core-2.1.5.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./cassandra-spark-job_2.10-1.0.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./guava-18.0.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./joda-convert-1.2.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./cassandra-clientutil-2.1.5.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/work/app-20150711142923-0023/0/./google-collections-1.0.jar] , parent being sun.misc.launcher$appclassloader@6132b73b of type class sun.misc.launcher$appclassloader classpath [file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/conf/,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-hadoop2.4.0.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,file:/home/sysadmin/apachespark/spark-1.4.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar] , parent being sun.misc.launcher$extclassloader@489bb457 of type class sun.misc.launcher$extclassloader classpath [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar] , parent being primordial classloader boot classpath [/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes] not found. @ com.datastax.spark.connector.rdd.cassandratablescanrdd.createstatement(cassandratablescanrdd.scala:163)

here program

/** cassandrajob.scala **/  import com.datastax.spark.connector._ import  org.apache.spark._   object cassandrajob {         def main(args: array[string]) {                 val conf = new sparkconf(true)                                 .set("spark.cassandra.connection.host", "172.28.0.164")                                 .set("soark.cassandra.connection.rpc.port", "9160")                  val sc = new sparkcontext(conf)                 val rdd = sc.cassandratable("eventtest", "simpletbl");                 println("cassandra row count : " + rdd.count + ", cassanra row : " + rdd.first)          } } 

i have build file using sbt compile, sbt package

here how submitting spark job ./bin/spark-submit --jars $(echo /home/sysadmin/apachespark/jar/*.jar | tr ' ' ',') --class "cassandrajob" --master spark://noi-cs-01:7077 /home/sysadmin/apachesparkproj/cassandrajob/target/scala-2.10/cassandra-spark-job_2.10-1.0.jar

i guess using org.joda.time.datetime missing in submitted jar. add jar dependencies: ... --jars $(echo /home/sysadmin/apachespark/jar/*.jar | tr ' ' ','),/path/to/downloaded/jodatime/jar --class "cassandrajob..."

the other way include org.joda.time.datetime in library dependencies in sbt , assembly fat jar library using sbt assembly plugin instead of sbt package.


Comments

Popular posts from this blog

android - Gradle sync Error:Configuration with name 'default' not found -

java - Andrioid studio start fail: Fatal error initializing 'null' -

html - jQuery UI Sortable - Remove placeholder after item is dropped -