SparkMLib 数据类型 - Data sources

spark2.4.8

介绍怎样在ML中使用数据源来加载数据. 除此之外一些常用的数据源像Parquet, CSV, JSON 和JDBC,提供了一些专用的数据源.

图数据源

图数据源被用来加载目录中图文件,它可以加载被压缩的图片(jpeg,png,等)为原始图片,通过Java 类库中ImageIO. 加载的DataFrame有一个StructType列:“image”,包含图片数据存储图片模式. 图片列模式:

• origin: StringType(代表图文件路径)
• height: IntegerType (图片高度)
• width: IntegerType (图片宽度)
• nChannels: IntegerType (number of image channels)
• mode: IntegerType (OpenCV-compatible type)
• data: BinaryType (Image bytes in OpenCV-compatible order: row-wise BGR in most cases)

为加载图数据ImageDataSource 实现Spark SQL数据源 API 来加载图片数据.

scala> val df = spark.read.format("image").option("dropInvalid", true).load("data/mllib/images/origin/kittens")
df: org.apache.spark.sql.DataFrame = [image: struct<origin: string, height: int ... 4 more fields>]scala> df.select("image.origin", "image.width", "image.height").show(truncate=false)
+-----------------------------------------------------------------------+-----+------+
|origin                                                                 |width|height|
+-----------------------------------------------------------------------+-----+------+
|file:///spark/data/mllib/images/origin/kittens/54893.jpg |300 |311 |
|file:///spark/data/mllib/images/origin/kittens/DP802813.jpg |199 |313 |
|file:///spark/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg |300 |200 |
|file:///spark/data/mllib/images/origin/kittens/DP153539.jpg |300 |296 |
+-----------------------------------------------------------------------+-----+------+

https://spark.apache.org/docs/2.4.8/ml-datasource