Scala集合函数笔记

sorted、sortBy

sorted：按照元素自身进行排序；
sortBy: 按照应用函数 f 之后产生的元素进行排序；

scala> val list = List(1,4,2,3,5)
list: List[Int] = List(1, 4, 2, 3, 5)
// 按照自身元素排序
scala> list.sorted
res0: List[Int] = List(1, 2, 3, 4, 5)
// 按照指定元素排序
scala> val list = List(("a",3), ("c",1), ("b",2))
list: List[(String, Int)] = List((a,3), (c,1), (b,2))
// 按照元组里第一个元素排序
scala> list.sortBy(_._1)
res1: List[(String, Int)] = List((a,3), (b,2), (c,1))
// 按照元组里第二个元素排序
scala> list.sortBy(_._2)
res2: List[(String, Int)] = List((c,1), (b,2), (a,3))
// 按照元组里第二个元素降序排序  
scala> list.sortBy(_._2).reverse
res3: List[(String, Int)] = List((a,3), (b,2), (c,1))

flatten

当有一个集合的集合，然后你想对这些集合的所有元素进行操作时，就会用到 flatten；
flatten 不支持元组

scala> val list = List(List(1,2), List(3,4))
list: List[List[Int]] = List(List(1, 2), List(3, 4))

scala> list.flatten
res4: List[Int] = List(1, 2, 3, 4)

scala> list.flatten.sum
res5: Int = 10

scala> val list = List((1,2), (3,4))
list: List[(Int, Int)] = List((1,2), (3,4))
// flatten 不支持元组
scala> list.flatten
<console>:13: error: No implicit view available from (Int, Int) => scala.collection.GenTraversableOnce[B].
       list.flatten
            ^

scala>

map, flatMap

map 操作
map操作是针对集合的典型变换操作，它将某个函数应用到集合中的每个元素，并产生一个结果集合；
map方法返回一个与原集合类型大小都相同的新集合，只不过元素的类型可能不同。

scala> val list = List(1,2,3,4)
list: List[Int] = List(1, 2, 3, 4)
// 对list 里面的每个元素加1，并返回新的集合
scala> list.map(_ + 1)
res7: List[Int] = List(2, 3, 4, 5)

scala> val list1 = List("a b c", "d e f")
list1: List[String] = List(a b c, d e f)
// 新集合和原集合的类型不同
scala> list1.map(_.split(" "))
res8: List[Array[String]] = List(Array(a, b, c), Array(d, e, f))

flatMap 操作

flatMap的执行过程： map --> flatten
注意：同flatten一样，不支持元组

scala> val list = List("a b c", "d e f")
list: List[String] = List(a b c, d e f)

scala> list.flatMap(_.split(" "))
res9: List[String] = List(a, b, c, d, e, f)

scala> list.map(_.split(" "))
res10: List[Array[String]] = List(Array(a, b, c), Array(d, e, f))

scala> list.map(_.split(" ")).flatten
res11: List[String] = List(a, b, c, d, e, f)

filter

遍历一个集合并从中获取满足指定条件的元素组成一个新的集合；

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> list.filter(_ % 2 == 0)
res12: List[Int] = List(2, 4)

scala> list.filter(f => if(f % 2 == 0 && f < 4) true else false)
res15: List[Int] = List(2)

并行集合

通过list.par 会将集合变成并行集合，可以利用多线程来进行运算。

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println("-----------------list-----------------")
    val s1 = list.foreach(f =>{
      println(s"${f}, threadName:${Thread.currentThread().getName}")
    })
    println("-------------list.par-------------")
    val s2 = list.par.foreach(f => {
      println(s"${f}, threadName:${Thread.currentThread().getName}")
    })
  }
}

RUN
单线程
-----------------list-----------------
1, threadName:main
2, threadName:main
3, threadName:main
4, threadName:main
5, threadName:main
多线程
-------------list.par-------------
1, threadName:scala-execution-context-global-20
5, threadName:scala-execution-context-global-24
2, threadName:scala-execution-context-global-22
3, threadName:scala-execution-context-global-21
4, threadName:scala-execution-context-global-23

reduce、reduceLeft、reduceRight

reduce：reduce(op: (A1, A1) => A1): A1 。reduce操作是按照从左到右的顺序进行规约。(((1+2)+3)+4)+5
reduceLeft：reduceLeft[B >: A](f: (B, A) => B): B。是按照从左到右的顺序进行规约。 (((1+2)+3)+4)+5
reduceRight：reduceRight[B >: A](op: (A, B) => B): B。是按照从右到左的顺序进行规约。1+(2+(3+(4+5)))
[B >: A] ：如果A类型固定，那B类型就是A的类型或A的父类

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println("-------------reduce-------------")
    val sum:Int = list.reduce((a:Int,b:Int) => {
      println(s"a:${a},b:${b}")
      a + b
    })
    println(sum)

    println("------------reduceLeft------------")
    val sum1 = list.reduceLeft((a:Int,b:Int) => {
      println(s"a:${a},b:${b}")
      a + b
    })
    println(sum1)

    println("------------reduceRight------------")
    val sum2 = list.reduceRight((a:Int,b:Int) => {
      println(s"a:${a},b:${b}")
      a + b
    })
    println(sum2)

    println("-----------并行集合的reduce-----------")
    val sum3 = list.par.reduce((a:Int,b:Int) => {
      println(s"a:${a},b:${b},ThreadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum3)

    println("-----------并行集合的reduceLeft-----------")
    val sum4 = list.par.reduceLeft((a:Int,b:Int) => {
      println(s"a:${a},b:${b},ThreadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum4)

    println("-----------并行集合的reduceRight-----------")
    val sum5 = list.par.reduceRight((a:Int,b:Int) => {
      println(s"a:${a},b:${b},ThreadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum5)
  }
}


RUN

-------------reduce-------------
a:1,b:2
a:3,b:3
a:6,b:4
a:10,b:5
15
------------reduceLeft------------
a:1,b:2
a:3,b:3
a:6,b:4
a:10,b:5
15
------------reduceRight------------
a:4,b:5
a:3,b:9
a:2,b:12
a:1,b:14
15
-----------并行集合的reduce-----------
a:4,b:5,ThreadName:scala-execution-context-global-21
a:1,b:2,ThreadName:scala-execution-context-global-20
a:3,b:9,ThreadName:scala-execution-context-global-21
a:3,b:12,ThreadName:scala-execution-context-global-20
15
-----------并行集合的reduceLeft-----------
a:1,b:2,ThreadName:main
a:3,b:3,ThreadName:main
a:6,b:4,ThreadName:main
a:10,b:5,ThreadName:main
15
-----------并行集合的reduceRight-----------
a:4,b:5,ThreadName:main
a:3,b:9,ThreadName:main
a:2,b:12,ThreadName:main
a:1,b:14,ThreadName:main
15

Process finished with exit code 0

//简写
object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println(list.reduce(_ + _))
    println(list.reduceLeft(_ + _))
    println(list.reduceRight(_ + _))

    println(list.par.reduce(_ + _))
    println(list.par.reduceLeft(_ + _))
    println(list.par.reduceRight(_ + _))
  }
}

//组合写法
object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(List(1,2),List(3,4), List(5,6), List(7,8,9))
    println(list.flatten.reduce(_ + _))
    println(list.flatten.reduceLeft(_ + _))
    println(list.flatten.reduceRight(_ + _))
  }
}

fold, foldLeft, foldRight

fold：fold[A1 >: A](z: A1)(op: (A1, A1) => A1): A1 。带有初始值的reduce，从一个初始值开始，从左向右将两个元素合并成一个，最终把列表合并成单一元素。((((10+1)+2)+3)+4)+5
foldLeft：foldLeft[B](z: B)(f: (B, A) => B): B 。带有初始值的reduceLeft。((((10+1)+2)+3)+4)+5
foldRight：foldRight[B](z: B)(op: (A, B) => B): B 。带有初始值的reduceRight。1+(2+(3+(4+(5+10))))

object FoldDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println("-------------fold-------------")
    val sum = list.fold(10)((a:Int, b:Int)=> {
      println(s"a:${a}, b:${b}")
      a+b
    })
    println(sum)
    println("------foldLeft------------")
    val sum1: Int = list.foldLeft(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b}")
      a + b
    })
    println(sum1)
    println("------foldRight------------")
    val sum2: Int = list.foldRight(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b}")
      a + b
    })
    println(sum2)
    println("-------并行集合的fold-----------")
    // 利用并行集合多线程运算，没有顺序
    val sum3: Int = list.par.fold(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b},threadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum3)
    println("-------并行集合的foldLeft-----------")
    // foldLeft 将并行集合多线程的运算变成了单线程，有顺序
    val sum4: Int = list.par.foldLeft(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b},threadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum4)
    println("-------并行集合的foldRight-----------")
    // foldRight 将并行集合多线程的运算变成了单线程，有顺序
    val sum5: Int = list.par.foldRight(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b},threadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum5)
  }
}

RUN

-------------fold-------------
a:10, b:1
a:11, b:2
a:13, b:3
a:16, b:4
a:20, b:5
25
------foldLeft------------
a:10, b:1
a:11, b:2
a:13, b:3
a:16, b:4
a:20, b:5
25
------foldRight------------
a:5, b:10
a:4, b:15
a:3, b:19
a:2, b:22
a:1, b:24
25
// 并行集合下 根据CPU核数运算，如果初始值设置不当，会造成运算结果失真
-------并行集合的fold-----------
a:10, b:5,threadName:scala-execution-context-global-24
a:10, b:4,threadName:scala-execution-context-global-23
a:10, b:2,threadName:scala-execution-context-global-22
a:10, b:3,threadName:scala-execution-context-global-21
a:14, b:15,threadName:scala-execution-context-global-23
a:10, b:1,threadName:scala-execution-context-global-20
a:13, b:29,threadName:scala-execution-context-global-21
a:11, b:12,threadName:scala-execution-context-global-20
a:23, b:42,threadName:scala-execution-context-global-20
65
-------并行集合的foldLeft-----------
a:10, b:1,threadName:main
a:11, b:2,threadName:main
a:13, b:3,threadName:main
a:16, b:4,threadName:main
a:20, b:5,threadName:main
25
-------并行集合的foldRight-----------
a:5, b:10,threadName:main
a:4, b:15,threadName:main
a:3, b:19,threadName:main
a:2, b:22,threadName:main
a:1, b:24,threadName:main
25

Process finished with exit code 0

组合写法
scala> val list = List(List(1,2),List(3,4), List(5,6), List(7,8,9))
list: List[List[Int]] = List(List(1, 2), List(3, 4), List(5, 6), List(7, 8, 9))
// 初始值和元素类型一致
scala> list.fold(List())(_ ++ _)
res16: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9)

scala> list.fold(List())(_ ++ _).reduce(_ + _)
res17: Int = 45
// 初始值和元素类型不一致
scala> list.foldLeft(0)(_ + _.sum)
res19: Int = 45

scala> list.par.foldLeft(0)(_ + _.sum)
res20: Int = 45

groupBy、grouped

groupBy：将list 按照某个元素内的字段分组，返回map。 List((k,v),(k,v)) --> Map(k, List(k,v))
grouped：按列表按照固定的大小进行分组，返回迭代器。List(1,2,3,4,5) --> Iterator[List[A]]

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(("a",1),("a",2), ("b",3))
    // 将list里内容的按照元素内的第一个进行分组
    val map = list.groupBy(_._1)
    println(map)
    val list1 = List("a", 1, "a", 2, "b", 3, 4)
    val iterator = list1.grouped(2)
    println(iterator.isEmpty)
    // 调用迭代器toBuffer，会把迭代器里的数据都迭代出来，有且只能迭代一次
    println(iterator.toBuffer)
    println(iterator.isEmpty)
  }
}

RUN

Map(a -> List((a,1), (a,2)), b -> List((b,3)))
false
ArrayBuffer(List(a, 1), List(a, 2), List(b, 3), List(4))
true

mapValues

对map映射里每个key的value 进行操作。

object MapDemo {
  def main(args: Array[String]): Unit = {
    val map = Map("b" -> List(1,2,3), "a" -> List(4,5,6))
    val n1 = map.mapValues(_.sum)
    println(n1)
  }
}

RUN

Map(b -> 6, a -> 15)

scala> val list = List(("a",1),("a",1),("b",1))
list: List[(String, Int)] = List((a,1), (a,1), (b,1))

scala> list.groupBy(_._1)
res0: scala.collection.immutable.Map[String,List[(String, Int)]] = Map(a -> List((a,1), (a,1)), b -> List((b,1)))

scala> list.groupBy(_._1).mapValues(_.size)
res1: scala.collection.immutable.Map[String,Int] = Map(a -> 2, b -> 1)

scala> val list = List(("a",1),("a",2),("b",3))
list: List[(String, Int)] = List((a,1), (a,2), (b,3))

scala> list.groupBy(_._1)
res2: scala.collection.immutable.Map[String,List[(String, Int)]] = Map(a -> List((a,1), (a,2)), b -> List((b,3)))

scala> list.groupBy(_._1).mapValues(_.map(_._2))
res3: scala.collection.immutable.Map[String,List[Int]] = Map(a -> List(1, 2), b -> List(3))

scala> list.groupBy(_._1).mapValues(_.map(_._2).sum)
res4: scala.collection.immutable.Map[String,Int] = Map(a -> 3, b -> 3)

aggregate

将每个分区里面的元素进行聚合，然后用combine函数将每个分区的结果和初始值进行combine操作；

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    // 当集合不是并行集合时，combop函数不执行
    val sum = list.aggregate(0)((a:Int,b:Int) => {
      println(s"step1:a:${a},b:${b}")
      a + b
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum)
    println("------------par-------------")
    // 当集合是并行集合时，combop函数执行
    // step1:做基础聚合， step2:在step1 基础上做聚合，相当于combiner
    val sum1 = list.par.aggregate(0)((a:Int,b:Int) => {
      println(s"step1:a:${a},b:${b}")
      a + b
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum1)
  }
}

RUN

step1:a:0,b:1
step1:a:1,b:2
step1:a:3,b:3
step1:a:6,b:4
step1:a:10,b:5
15
------------par-------------
step1:a:0,b:2
step1:a:0,b:4
step1:a:0,b:1
step1:a:0,b:5
step1:a:0,b:3
step2:a:1,b:2
step2:a:4,b:5
step2:a:3,b:9
step2:a:3,b:12
15

Process finished with exit code 0

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(List(1,2), List(1,2,3),List(1,2,3,4), List(1))
    // 当集合不是并行集合时，combop函数不执行
    // 等价于list.aggregate(0)(_ + _.sum,_ + _)
    val sum = list.aggregate(0)((a:Int,b:List[Int]) => {
      println(s"step1:a:${a},b:${b}")
      a + b.sum
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum)
    // 当集合是并行集合时，combop函数执行
    // step1:做基础聚合， step2:在step1 基础上做聚合，相当于combiner
    // 等价于list.par.aggregate(0)(_ + _.sum,_ + _)
    val sum1 = list.par.aggregate(0)((a:Int,b:List[Int]) => {
      println(s"step1:a:${a},b:${b}")
      a + b.sum
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum1)
  }
}

RUN

step1:a:0,b:List(1, 2)
step1:a:3,b:List(1, 2, 3)
step1:a:9,b:List(1, 2, 3, 4)
step1:a:19,b:List(1)
20
step1:a:0,b:List(1)
step1:a:0,b:List(1, 2, 3)
step1:a:0,b:List(1, 2, 3, 4)
step1:a:0,b:List(1, 2)
step2:a:3,b:6
step2:a:10,b:1
step2:a:9,b:11
20

diff, union, intersect

diff : 两个集合的差集；
union : 两个集合的并集；
intersect: 两个集合的交集；

object ListDemo {
  def main(args: Array[String]): Unit = {
    val nums1 = List(1,2,3)
    val nums2 = List(2,3,4)
    println("-----------diff-----------")
    println(nums1 diff nums2)
    println(nums2 diff nums1)
    println("-----------union-----------")
    println(nums1 union nums2)
    println(nums1 ++ nums2)
    println("------------intersect-----------")
    println(nums1 intersect( nums2))
  }
}

RUN

-----------diff-----------
List(1)
List(4)
-----------union-----------
List(1, 2, 3, 2, 3, 4)
List(1, 2, 3, 2, 3, 4)
------------intersect-----------
List(2, 3)

实现workcount

object WordCount {
  def main(args: Array[String]): Unit = {
    val list = List("a b c d","a d e s","a b d e")
    println(list.map(_.split(" ")).flatten.map((_,1)).groupBy(_._1).mapValues(_.size).toList.sortBy(_._2))
    println(list.flatMap(_.split(" ")).map((_,1)).groupBy(_._1).mapValues(_.size).toList.sortBy(_._2))
  }
}

RUN

List((s,1), (c,1), (e,2), (b,2), (a,3), (d,3))
List((s,1), (c,1), (e,2), (b,2), (a,3), (d,3))

Q.E.D.

៚忘ོ尘ོོ

忘尘的个人空间

sorted、sortBy

flatten

map, flatMap

flatMap 操作

filter

并行集合

reduce、reduceLeft、reduceRight

fold, foldLeft, foldRight

groupBy、grouped

mapValues

aggregate

diff, union, intersect

实现workcount

Spark通过sc.textFile读取文件报数组越界异常 java.lang.ArrayIndexOutOfBoundsException: 17418

剑指 Offer 38. 字符串的排列 Scala解法

忘尘

忘尘的个人空间