Flink Windows环境配置

Flink项目初始化

  • 通过mvn来构建项目，依次输入groupdid, artifactid
    1
    2
    3
    4
    $ mvn archetype:generate                               \
          -DarchetypeGroupId=org.apache.flink              \
          -DarchetypeArtifactId=flink-quickstart-java      \
          -DarchetypeVersion=1.9.0
  • 通过官方脚本快速构建
    1
    curl https://flink.apache.org/q/quickstart.sh | bash -s 1.9.0

项目结构

项目结构

  • 其中 BatchJob 为批处理的样例代码
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    package meekou;

    import org.apache.flink.api.java.ExecutionEnvironment;

    /**
     * Skeleton for a Flink Batch Job.
     *
     * <p>For a tutorial how to write a Flink batch application, check the
     * tutorials and examples on the <a href="http://flink.apache.org/docs/stable/">Flink Website</a>.
     *
     * <p>To package your application into a JAR file for execution,
     * change the main class in the POM.xml file to this class (simply search for 'mainClass')
     * and run 'mvn clean package' on the command line.
     */
    public class BatchJob {

    	public static void main(String[] args) throws Exception {
    		// set up the batch execution environment
    		final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

    		/*
    		 * Here, you can start creating your execution plan for Flink.
    		 *
    		 * Start with getting some data from the environment, like
    		 * 	env.readTextFile(textPath);
    		 *
    		 * then, transform the resulting DataSet<String> using operations
    		 * like
    		 * 	.filter()
    		 * 	.flatMap()
    		 * 	.join()
    		 * 	.coGroup()
    		 *
    		 * and many more.
    		 * Have a look at the programming guide for the Java API:
    		 *
    		 * http://flink.apache.org/docs/latest/apis/batch/index.html
    		 *
    		 * and the examples
    		 *
    		 * http://flink.apache.org/docs/latest/apis/batch/examples.html
    		 *
    		 */

    		// execute program
    		env.execute("Flink Batch Java API Skeleton");
    	}
    }
  • 其中 StreamingJob 为流处理的样例代码
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    package meekou;

    import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

    /**
     * Skeleton for a Flink Streaming Job.
     *
     * <p>For a tutorial how to write a Flink streaming application, check the
     * tutorials and examples on the <a href="http://flink.apache.org/docs/stable/">Flink Website</a>.
     *
     * <p>To package your application into a JAR file for execution, run
     * 'mvn clean package' on the command line.
     *
     * <p>If you change the name of the main class (with the public static void main(String[] args))
     * method, change the respective entry in the POM.xml file (simply search for 'mainClass').
     */
    public class StreamingJob {

    	public static void main(String[] args) throws Exception {
    		// set up the streaming execution environment
    		final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    		/*
    		 * Here, you can start creating your execution plan for Flink.
    		 *
    		 * Start with getting some data from the environment, like
    		 * 	env.readTextFile(textPath);
    		 *
    		 * then, transform the resulting DataStream<String> using operations
    		 * like
    		 * 	.filter()
    		 * 	.flatMap()
    		 * 	.join()
    		 * 	.coGroup()
    		 *
    		 * and many more.
    		 * Have a look at the programming guide for the Java API:
    		 *
    		 * http://flink.apache.org/docs/latest/apis/streaming/index.html
    		 *
    		 */

    		// execute program
    		env.execute("Flink Streaming Java API Skeleton");
    	}
    }

    getExecutionEnvironment 代表获取批处理或流处理的执行环境，如果是本地运行则获取到的就是本地的执行环境；如果在集群上运行，得到的就是集群的执行环境。

FAQ

  • Error “The goal you specified requires a project to execute but there is no POM in this directory” after executing maven command
    • 方案一 对于Windows平台，通过cmd.exe来执行，通过PowerShell来执行这些命令会导致错误
    • 方案二 对参数加上""
      1
      $ mvn archetype:generate "-DarchetypeGroupId=org.apache.flink" "-DarchetypeArtifactId=flink-quickstart-java" "-DarchetypeVersion=1.9.0"
  • Error “Cannot find a class with the main method”
    • 关闭vscode 和 eclipse
    • 删除根目录下的.project 和.classpath文件
    • 在根目录执行mvn eclipse:eclipse
    • 通过VS Code打开根目录
    • F5执行Debug项目
  • Error “var cannot be resolved to a type”
    • 方案一 Updatepom.xml
      1
      2
      3
      4
      5
      6
      7
      <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
          <configuration>
              <release>11</release>
          </configuration>
      </plugin>
    • 方案二
    • 关闭vscode 和 eclipse
    • 删除根目录下的.project 和.classpath文件
    • 在根目录执行mvn eclipse:eclipse
    • 通过VS Code打开根目录
    • F5执行Debug项目

      引用

  • Flink
  • Flink official demo debug will get error :Cannot find a class with the main method #478
  • Error “The goal you specified requires a project to execute but there is no POM in this directory” after executing maven command
  • Flink 开发环境搭建