Q:
Here is a sample Spark code, which converts the Seq
to the Dataset
:
import spark.implicits._
val s = Seq(1, 2, 3, 4)
val ds = s.toDS()
The Scala Seq
does not have the toDS
method, it comes from Spark implicits. How is the Dataset created here?
A:
Scala has a way to add methods to existing classes, like extension method in Kotlin (and C# as I remember), but does it in a different way, through implicits.
To add the method to existing class, you first create implicit class:
object StringImplicits {
implicit class StringUtils(s: String) {
def someCoolMethod = println("Yooo")
}
}
object Application extends App {
import StringImplicits._
val s = "Hello"
s.someCoolMethod
}
You import this StringUtils
and can call someCoolMethod
on instance of String
Notice that StringUtils
class takes String
as a constuctor param.
When calling some method on String
, scala compiler first looks this method in String
class.
If it does not find it, it will look imported implicit classes which take String
param.
If found, it calls the method from that class.
If no such class found, it will raise the error.