maven - Spark 2.2.0 Streaming Failed to find data source: kafka -
i use maven manage project. , i add
<groupid>org.apache.spark</groupid> <artifactid>spark-sql-kafka-0-10_2.11</artifactid> <version>2.2.0</version> to maven dependencies
below pom.xml
<project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>***</groupid> <artifactid>***</artifactid> <version>1.0</version> <packaging>jar</packaging> <properties> <project.build.sourceencoding>utf-8</project.build.sourceencoding> </properties> <build> <sourcedirectory>src/main/scala</sourcedirectory> <plugins> <plugin> <groupid>org.scala-tools</groupid> <artifactid>maven-scala-plugin</artifactid> <version>2.11</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testcompile</goal> </goals> </execution> </executions> <configuration> <scalaversion>2.11.8</scalaversion> </configuration> </plugin> <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-compiler-plugin</artifactid> <version>2.0.2</version> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> <plugin> <artifactid>maven-assembly-plugin</artifactid> <version>2.6</version> <groupid>org.apache.maven.plugins</groupid> <configuration> <appendassemblyid>false</appendassemblyid> <descriptorrefs> <descriptorref>jar-with-dependencies</descriptorref> </descriptorrefs> <archive> <manifest> <mainclass>***</mainclass> </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <dependencies> <dependency> <groupid>org.apache.spark</groupid> <artifactid>spark-sql_2.11</artifactid> <version>2.2.0</version> </dependency> <dependency> <groupid>org.apache.spark</groupid> <artifactid>spark-sql-kafka-0-10_2.11</artifactid> <version>2.2.0</version> </dependency> <dependency> <groupid>org.apache.spark</groupid> <artifactid>spark-streaming-kafka-0-10_2.11</artifactid> <version>2.2.0</version> </dependency> <dependency> <groupid>org.scalaj</groupid> <artifactid>scalaj-http_2.11</artifactid> <version>2.2.0</version> </dependency> </dependencies> </project> i package using:
mvn clean package i submit job locally typing:
spark-submit --class ... <path jar file> <arguments run main class>
but error saying:exception in thread "main" java.lang.classnotfoundexception: failed find data source: kafka. please find packages @ http://spark.apache.org/third-party-projects.html
i know can fix problem adding --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0 after spark-submit.
but how can modify pom advoid doing that? the thing in maven repo, can see spark-sql-kafka-0-10_2.11-2.2.0.jar has been downloaded. why need add dependency mannually during spark submit? feel there might error in pom.xml though use assembly build jar.
hope can me out!
finally sloved problem. changed pom.xml follows:
<project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>***</groupid> <artifactid>***</artifactid> <version>1.0</version> <packaging>jar</packaging> <properties> <project.build.sourceencoding>utf-8</project.build.sourceencoding> <spark.version>2.2.0</spark.version> </properties> <dependencies> <dependency> <groupid>org.apache.spark</groupid> <artifactid>spark-core_2.11</artifactid> <version>${spark.version}</version> <scope>${spark.scope}</scope> </dependency> <dependency> <groupid>org.apache.spark</groupid> <artifactid>spark-sql_2.11</artifactid> <version>${spark.version}</version> <scope>${spark.scope}</scope> </dependency> <dependency> <groupid>org.apache.spark</groupid> <artifactid>spark-sql-kafka-0-10_2.11</artifactid> <version>${spark.version}</version> </dependency> </dependencies> <profiles> <profile> <id>default</id> <properties> <profile.id>dev</profile.id> <spark.scope>compile</spark.scope> </properties> <activation> <activebydefault>true</activebydefault> </activation> </profile> <profile> <id>test</id> <properties> <profile.id>test</profile.id> <spark.scope>provided</spark.scope> </properties> </profile> <profile> <id>online</id> <properties> <profile.id>online</profile.id> <spark.scope>provided</spark.scope> </properties> </profile> </profiles> <build> <plugins> <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-compiler-plugin</artifactid> <version>2.0.2</version> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> <plugin> <groupid>org.scala-tools</groupid> <artifactid>maven-scala-plugin</artifactid> <version>2.11</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testcompile</goal> </goals> </execution> </executions> <configuration> <scalaversion>2.11.8</scalaversion> </configuration> </plugin> <plugin> <artifactid>maven-assembly-plugin</artifactid> <configuration> <appendassemblyid>false</appendassemblyid> <descriptorrefs> <descriptorref>jar-with-dependencies</descriptorref> </descriptorrefs> <archive> <manifest> <mainclass>***</mainclass> </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> basically added profiles section , add scope each dependency. instead of using mvn clean package used mvn clean install -ponline -dskiptests. , suprisingly, works perfect.
i not quite clear details why method work, jar file can see jar created mvn clean package include lots of folders while other method includes few. maybe there conflict between folders in first method. don't know, hope experienced people can explain this.
Comments
Post a Comment