sorted、sortBy

sorted:按照元素自身进行排序;
sortBy: 按照应用函数 f 之后产生的元素进行排序;

scala> val list = List(1,4,2,3,5)
list: List[Int] = List(1, 4, 2, 3, 5)
// 按照自身元素排序
scala> list.sorted
res0: List[Int] = List(1, 2, 3, 4, 5)
// 按照指定元素排序
scala> val list = List(("a",3), ("c",1), ("b",2))
list: List[(String, Int)] = List((a,3), (c,1), (b,2))
// 按照元组里第一个元素排序
scala> list.sortBy(_._1)
res1: List[(String, Int)] = List((a,3), (b,2), (c,1))
// 按照元组里第二个元素排序
scala> list.sortBy(_._2)
res2: List[(String, Int)] = List((c,1), (b,2), (a,3))
// 按照元组里第二个元素降序排序  
scala> list.sortBy(_._2).reverse
res3: List[(String, Int)] = List((a,3), (b,2), (c,1))

flatten

当有一个集合的集合,然后你想对这些集合的所有元素进行操作时,就会用到 flatten;
flatten 不支持元组

scala> val list = List(List(1,2), List(3,4))
list: List[List[Int]] = List(List(1, 2), List(3, 4))

scala> list.flatten
res4: List[Int] = List(1, 2, 3, 4)

scala> list.flatten.sum
res5: Int = 10

scala> val list = List((1,2), (3,4))
list: List[(Int, Int)] = List((1,2), (3,4))
// flatten 不支持元组
scala> list.flatten
<console>:13: error: No implicit view available from (Int, Int) => scala.collection.GenTraversableOnce[B].
       list.flatten
            ^

scala>

map, flatMap

map 操作
map操作是针对集合的典型变换操作,它将某个函数应用到集合中的每个元素,并产生一个结果集合;
map方法返回一个与原集合类型大小都相同的新集合,只不过元素的类型可能不同。

scala> val list = List(1,2,3,4)
list: List[Int] = List(1, 2, 3, 4)
// 对list 里面的每个元素加1,并返回新的集合
scala> list.map(_ + 1)
res7: List[Int] = List(2, 3, 4, 5)

scala> val list1 = List("a b c", "d e f")
list1: List[String] = List(a b c, d e f)
// 新集合和原集合的类型不同
scala> list1.map(_.split(" "))
res8: List[Array[String]] = List(Array(a, b, c), Array(d, e, f))

flatMap 操作

flatMap的执行过程: map --> flatten
注意:同flatten一样,不支持元组

scala> val list = List("a b c", "d e f")
list: List[String] = List(a b c, d e f)

scala> list.flatMap(_.split(" "))
res9: List[String] = List(a, b, c, d, e, f)

scala> list.map(_.split(" "))
res10: List[Array[String]] = List(Array(a, b, c), Array(d, e, f))

scala> list.map(_.split(" ")).flatten
res11: List[String] = List(a, b, c, d, e, f)

filter

遍历一个集合并从中获取满足指定条件的元素组成一个新的集合;

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> list.filter(_ % 2 == 0)
res12: List[Int] = List(2, 4)

scala> list.filter(f => if(f % 2 == 0 && f < 4) true else false)
res15: List[Int] = List(2)

并行集合

通过list.par 会将集合变成并行集合,可以利用多线程来进行运算。

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println("-----------------list-----------------")
    val s1 = list.foreach(f =>{
      println(s"${f}, threadName:${Thread.currentThread().getName}")
    })
    println("-------------list.par-------------")
    val s2 = list.par.foreach(f => {
      println(s"${f}, threadName:${Thread.currentThread().getName}")
    })
  }
}

RUN
单线程
-----------------list-----------------
1, threadName:main
2, threadName:main
3, threadName:main
4, threadName:main
5, threadName:main
多线程
-------------list.par-------------
1, threadName:scala-execution-context-global-20
5, threadName:scala-execution-context-global-24
2, threadName:scala-execution-context-global-22
3, threadName:scala-execution-context-global-21
4, threadName:scala-execution-context-global-23

reduce、reduceLeft、reduceRight

reduce:reduce(op: (A1, A1) => A1): A1 。reduce操作是按照从左到右的顺序进行规约。(((1+2)+3)+4)+5
reduceLeft:reduceLeft[B >: A](f: (B, A) => B): B。是按照从左到右的顺序进行规约。 (((1+2)+3)+4)+5
reduceRight:reduceRight[B >: A](op: (A, B) => B): B。是按照从右到左的顺序进行规约。1+(2+(3+(4+5)))
[B >: A] : 如果A类型固定,那B类型就是A的类型或A的父类

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println("-------------reduce-------------")
    val sum:Int = list.reduce((a:Int,b:Int) => {
      println(s"a:${a},b:${b}")
      a + b
    })
    println(sum)

    println("------------reduceLeft------------")
    val sum1 = list.reduceLeft((a:Int,b:Int) => {
      println(s"a:${a},b:${b}")
      a + b
    })
    println(sum1)

    println("------------reduceRight------------")
    val sum2 = list.reduceRight((a:Int,b:Int) => {
      println(s"a:${a},b:${b}")
      a + b
    })
    println(sum2)

    println("-----------并行集合的reduce-----------")
    val sum3 = list.par.reduce((a:Int,b:Int) => {
      println(s"a:${a},b:${b},ThreadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum3)

    println("-----------并行集合的reduceLeft-----------")
    val sum4 = list.par.reduceLeft((a:Int,b:Int) => {
      println(s"a:${a},b:${b},ThreadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum4)

    println("-----------并行集合的reduceRight-----------")
    val sum5 = list.par.reduceRight((a:Int,b:Int) => {
      println(s"a:${a},b:${b},ThreadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum5)
  }
}


RUN

-------------reduce-------------
a:1,b:2
a:3,b:3
a:6,b:4
a:10,b:5
15
------------reduceLeft------------
a:1,b:2
a:3,b:3
a:6,b:4
a:10,b:5
15
------------reduceRight------------
a:4,b:5
a:3,b:9
a:2,b:12
a:1,b:14
15
-----------并行集合的reduce-----------
a:4,b:5,ThreadName:scala-execution-context-global-21
a:1,b:2,ThreadName:scala-execution-context-global-20
a:3,b:9,ThreadName:scala-execution-context-global-21
a:3,b:12,ThreadName:scala-execution-context-global-20
15
-----------并行集合的reduceLeft-----------
a:1,b:2,ThreadName:main
a:3,b:3,ThreadName:main
a:6,b:4,ThreadName:main
a:10,b:5,ThreadName:main
15
-----------并行集合的reduceRight-----------
a:4,b:5,ThreadName:main
a:3,b:9,ThreadName:main
a:2,b:12,ThreadName:main
a:1,b:14,ThreadName:main
15

Process finished with exit code 0

//简写
object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println(list.reduce(_ + _))
    println(list.reduceLeft(_ + _))
    println(list.reduceRight(_ + _))

    println(list.par.reduce(_ + _))
    println(list.par.reduceLeft(_ + _))
    println(list.par.reduceRight(_ + _))
  }
}

//组合写法
object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(List(1,2),List(3,4), List(5,6), List(7,8,9))
    println(list.flatten.reduce(_ + _))
    println(list.flatten.reduceLeft(_ + _))
    println(list.flatten.reduceRight(_ + _))
  }
}

fold, foldLeft, foldRight

fold:fold[A1 >: A](z: A1)(op: (A1, A1) => A1): A1 。带有初始值的reduce,从一个初始值开始,从左向右将两个元素合并成一个,最终把列表合并成单一元素。((((10+1)+2)+3)+4)+5
foldLeft:foldLeft[B](z: B)(f: (B, A) => B): B 。带有初始值的reduceLeft。((((10+1)+2)+3)+4)+5
foldRight:foldRight[B](z: B)(op: (A, B) => B): B 。带有初始值的reduceRight。1+(2+(3+(4+(5+10))))

object FoldDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    println("-------------fold-------------")
    val sum = list.fold(10)((a:Int, b:Int)=> {
      println(s"a:${a}, b:${b}")
      a+b
    })
    println(sum)
    println("------foldLeft------------")
    val sum1: Int = list.foldLeft(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b}")
      a + b
    })
    println(sum1)
    println("------foldRight------------")
    val sum2: Int = list.foldRight(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b}")
      a + b
    })
    println(sum2)
    println("-------并行集合的fold-----------")
    // 利用并行集合多线程运算,没有顺序
    val sum3: Int = list.par.fold(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b},threadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum3)
    println("-------并行集合的foldLeft-----------")
    // foldLeft 将并行集合多线程的运算变成了单线程,有顺序
    val sum4: Int = list.par.foldLeft(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b},threadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum4)
    println("-------并行集合的foldRight-----------")
    // foldRight 将并行集合多线程的运算变成了单线程,有顺序
    val sum5: Int = list.par.foldRight(10)((a: Int, b: Int) => {
      println(s"a:${a}, b:${b},threadName:${Thread.currentThread().getName}")
      a + b
    })
    println(sum5)
  }
}

RUN

-------------fold-------------
a:10, b:1
a:11, b:2
a:13, b:3
a:16, b:4
a:20, b:5
25
------foldLeft------------
a:10, b:1
a:11, b:2
a:13, b:3
a:16, b:4
a:20, b:5
25
------foldRight------------
a:5, b:10
a:4, b:15
a:3, b:19
a:2, b:22
a:1, b:24
25
// 并行集合下 根据CPU核数运算,如果初始值设置不当,会造成运算结果失真
-------并行集合的fold-----------
a:10, b:5,threadName:scala-execution-context-global-24
a:10, b:4,threadName:scala-execution-context-global-23
a:10, b:2,threadName:scala-execution-context-global-22
a:10, b:3,threadName:scala-execution-context-global-21
a:14, b:15,threadName:scala-execution-context-global-23
a:10, b:1,threadName:scala-execution-context-global-20
a:13, b:29,threadName:scala-execution-context-global-21
a:11, b:12,threadName:scala-execution-context-global-20
a:23, b:42,threadName:scala-execution-context-global-20
65
-------并行集合的foldLeft-----------
a:10, b:1,threadName:main
a:11, b:2,threadName:main
a:13, b:3,threadName:main
a:16, b:4,threadName:main
a:20, b:5,threadName:main
25
-------并行集合的foldRight-----------
a:5, b:10,threadName:main
a:4, b:15,threadName:main
a:3, b:19,threadName:main
a:2, b:22,threadName:main
a:1, b:24,threadName:main
25

Process finished with exit code 0

组合写法
scala> val list = List(List(1,2),List(3,4), List(5,6), List(7,8,9))
list: List[List[Int]] = List(List(1, 2), List(3, 4), List(5, 6), List(7, 8, 9))
// 初始值和元素类型一致
scala> list.fold(List())(_ ++ _)
res16: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9)

scala> list.fold(List())(_ ++ _).reduce(_ + _)
res17: Int = 45
// 初始值和元素类型不一致
scala> list.foldLeft(0)(_ + _.sum)
res19: Int = 45

scala> list.par.foldLeft(0)(_ + _.sum)
res20: Int = 45

groupBy、grouped

groupBy:将list 按照某个元素内的字段分组,返回map。 List((k,v),(k,v)) --> Map(k, List(k,v))
grouped:按列表按照固定的大小进行分组,返回迭代器。List(1,2,3,4,5) --> Iterator[List[A]]

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(("a",1),("a",2), ("b",3))
    // 将list里内容的按照元素内的第一个进行分组
    val map = list.groupBy(_._1)
    println(map)
    val list1 = List("a", 1, "a", 2, "b", 3, 4)
    val iterator = list1.grouped(2)
    println(iterator.isEmpty)
    // 调用迭代器toBuffer,会把迭代器里的数据都迭代出来,有且只能迭代一次
    println(iterator.toBuffer)
    println(iterator.isEmpty)
  }
}

RUN

Map(a -> List((a,1), (a,2)), b -> List((b,3)))
false
ArrayBuffer(List(a, 1), List(a, 2), List(b, 3), List(4))
true

mapValues

对map映射里每个key的value 进行操作。

object MapDemo {
  def main(args: Array[String]): Unit = {
    val map = Map("b" -> List(1,2,3), "a" -> List(4,5,6))
    val n1 = map.mapValues(_.sum)
    println(n1)
  }
}

RUN

Map(b -> 6, a -> 15)
scala> val list = List(("a",1),("a",1),("b",1))
list: List[(String, Int)] = List((a,1), (a,1), (b,1))

scala> list.groupBy(_._1)
res0: scala.collection.immutable.Map[String,List[(String, Int)]] = Map(a -> List((a,1), (a,1)), b -> List((b,1)))

scala> list.groupBy(_._1).mapValues(_.size)
res1: scala.collection.immutable.Map[String,Int] = Map(a -> 2, b -> 1)

scala> val list = List(("a",1),("a",2),("b",3))
list: List[(String, Int)] = List((a,1), (a,2), (b,3))

scala> list.groupBy(_._1)
res2: scala.collection.immutable.Map[String,List[(String, Int)]] = Map(a -> List((a,1), (a,2)), b -> List((b,3)))

scala> list.groupBy(_._1).mapValues(_.map(_._2))
res3: scala.collection.immutable.Map[String,List[Int]] = Map(a -> List(1, 2), b -> List(3))

scala> list.groupBy(_._1).mapValues(_.map(_._2).sum)
res4: scala.collection.immutable.Map[String,Int] = Map(a -> 3, b -> 3)

aggregate

将每个分区里面的元素进行聚合,然后用combine函数将每个分区的结果和初始值进行combine操作;
image.png

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(1,2,3,4,5)
    // 当集合不是并行集合时,combop函数不执行
    val sum = list.aggregate(0)((a:Int,b:Int) => {
      println(s"step1:a:${a},b:${b}")
      a + b
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum)
    println("------------par-------------")
    // 当集合是并行集合时,combop函数执行
    // step1:做基础聚合, step2:在step1 基础上做聚合,相当于combiner
    val sum1 = list.par.aggregate(0)((a:Int,b:Int) => {
      println(s"step1:a:${a},b:${b}")
      a + b
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum1)
  }
}

RUN

step1:a:0,b:1
step1:a:1,b:2
step1:a:3,b:3
step1:a:6,b:4
step1:a:10,b:5
15
------------par-------------
step1:a:0,b:2
step1:a:0,b:4
step1:a:0,b:1
step1:a:0,b:5
step1:a:0,b:3
step2:a:1,b:2
step2:a:4,b:5
step2:a:3,b:9
step2:a:3,b:12
15

Process finished with exit code 0

object ListDemo {
  def main(args: Array[String]): Unit = {
    val list = List(List(1,2), List(1,2,3),List(1,2,3,4), List(1))
    // 当集合不是并行集合时,combop函数不执行
    // 等价于list.aggregate(0)(_ + _.sum,_ + _)
    val sum = list.aggregate(0)((a:Int,b:List[Int]) => {
      println(s"step1:a:${a},b:${b}")
      a + b.sum
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum)
    // 当集合是并行集合时,combop函数执行
    // step1:做基础聚合, step2:在step1 基础上做聚合,相当于combiner
    // 等价于list.par.aggregate(0)(_ + _.sum,_ + _)
    val sum1 = list.par.aggregate(0)((a:Int,b:List[Int]) => {
      println(s"step1:a:${a},b:${b}")
      a + b.sum
    },
      (a:Int,b:Int) => {
        println(s"step2:a:${a},b:${b}")
        a + b
      })
    println(sum1)
  }
}

RUN

step1:a:0,b:List(1, 2)
step1:a:3,b:List(1, 2, 3)
step1:a:9,b:List(1, 2, 3, 4)
step1:a:19,b:List(1)
20
step1:a:0,b:List(1)
step1:a:0,b:List(1, 2, 3)
step1:a:0,b:List(1, 2, 3, 4)
step1:a:0,b:List(1, 2)
step2:a:3,b:6
step2:a:10,b:1
step2:a:9,b:11
20

diff, union, intersect

diff : 两个集合的差集;
union : 两个集合的并集;
intersect: 两个集合的交集;

object ListDemo {
  def main(args: Array[String]): Unit = {
    val nums1 = List(1,2,3)
    val nums2 = List(2,3,4)
    println("-----------diff-----------")
    println(nums1 diff nums2)
    println(nums2 diff nums1)
    println("-----------union-----------")
    println(nums1 union nums2)
    println(nums1 ++ nums2)
    println("------------intersect-----------")
    println(nums1 intersect( nums2))
  }
}

RUN

-----------diff-----------
List(1)
List(4)
-----------union-----------
List(1, 2, 3, 2, 3, 4)
List(1, 2, 3, 2, 3, 4)
------------intersect-----------
List(2, 3)

实现workcount

object WordCount {
  def main(args: Array[String]): Unit = {
    val list = List("a b c d","a d e s","a b d e")
    println(list.map(_.split(" ")).flatten.map((_,1)).groupBy(_._1).mapValues(_.size).toList.sortBy(_._2))
    println(list.flatMap(_.split(" ")).map((_,1)).groupBy(_._1).mapValues(_.size).toList.sortBy(_._2))
  }
}

RUN

List((s,1), (c,1), (e,2), (b,2), (a,3), (d,3))
List((s,1), (c,1), (e,2), (b,2), (a,3), (d,3))

Q.E.D.