pyspark.sql.functions.variant_insert#

pyspark.sql.functions.variant_insert(v, path, value)[source]#

Inserts a value into a variant at the given JSONPath location. An object path adds a new field (error if it already exists); an array path inserts at the index, shifting later elements right. Missing intermediate keys are created. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.

Added in version 4.3.0.

Parameters:

vColumn or str: a variant column or column name A column that evaluates to a variant.
pathColumn or str: the JSONPath insertion target. A str is a literal path; a Column supplies the path at runtime. A valid path should start with $ and is followed by one or more segments like [123], .name, [‘name’], or A column that evaluates to a string. [“name”]. The root path $ is not allowed.
valueColumn or str: the value to insert. Any expression castable to variant.

Returns:

Column: a variant column with value inserted at path Returns a column that evaluates to a variant.

Examples

>>> from pyspark.sql.functions import lit, parse_json, to_json, variant_insert
>>> df = spark.createDataFrame([{
...     'json': '''{ "a": 1, "arr": ["x", "y"] }''',
...     'path': '$.d'
... }])
>>> v = parse_json(df.json)
>>> df.select(to_json(variant_insert(v, "$.b", lit(2))).alias("r")).collect()
[Row(r='{"a":1,"arr":["x","y"],"b":2}')]
>>> df.select(to_json(variant_insert(v, "$.c.d", lit(3))).alias("r")).collect()
[Row(r='{"a":1,"arr":["x","y"],"c":{"d":3}}')]
>>> df.select(to_json(variant_insert(v, "$.arr[1]", lit("z"))).alias("r")).collect()
[Row(r='{"a":1,"arr":["x","z","y"]}')]
>>> df.select(to_json(variant_insert(v, "$.arr[5]", lit("z"))).alias("r")).collect()
[Row(r='{"a":1,"arr":["x","y",null,null,null,"z"]}')]
>>> df.select(to_json(variant_insert(v, df.path, lit(9))).alias("r")).collect()
[Row(r='{"a":1,"arr":["x","y"],"d":9}')]
>>> df.select(to_json(variant_insert(v, "$.b", parse_json(lit('null')))).alias("r")).collect()
[Row(r='{"a":1,"arr":["x","y"],"b":null}')]
>>> df.select(variant_insert(v, "$.b", lit(None)).alias("r")).collect()
[Row(r=None)]