pyspark.sql.functions.concat#

pyspark.sql.functions.concat(*cols)[source]#

Collection function: Concatenates multiple input columns together into a single column. The function works with strings, numeric, binary and compatible array columns.

New in version 1.5.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colsColumn or str: target column or columns to work on.

Returns

Column: concatenated values. Type of the Column depends on input columns’ type.

See also

pyspark.sql.functions.concat_ws()
pyspark.sql.functions.array_join(): to concatenate string columns with delimiter

Examples

Example 1: Concatenating string columns

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([('abcd','123')], ['s', 'd'])
>>> df.select(sf.concat(df.s, df.d)).show()
+------------+
|concat(s, d)|
+------------+
|     abcd123|
+------------+

Example 2: Concatenating array columns

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([([1, 2], [3, 4], [5]), ([1, 2], None, [3])], ['a', 'b', 'c'])
>>> df.select(sf.concat(df.a, df.b, df.c)).show()
+---------------+
|concat(a, b, c)|
+---------------+
|[1, 2, 3, 4, 5]|
|           NULL|
+---------------+

Example 3: Concatenating numeric columns

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1, 2, 3)], ['a', 'b', 'c'])
>>> df.select(sf.concat(df.a, df.b, df.c)).show()
+---------------+
|concat(a, b, c)|
+---------------+
|            123|
+---------------+

Example 4: Concatenating binary columns

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(bytearray(b'abc'), bytearray(b'def'))], ['a', 'b'])
>>> df.select(sf.concat(df.a, df.b)).show()
+-------------------+
|       concat(a, b)|
+-------------------+
|[61 62 63 64 65 66]|
+-------------------+

Example 5: Concatenating mixed types of columns

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1,"abc",3,"def")], ['a','b','c','d'])
>>> df.select(sf.concat(df.a, df.b, df.c, df.d)).show()
+------------------+
|concat(a, b, c, d)|
+------------------+
|          1abc3def|
+------------------+