Answers for "spark to pandas"

0

convert pandas dataframe to spark dataframe

import pandas as pd
from pyspark.sql import SparkSession

filename = <'path to file'>
spark = SparkSession.build.appName('pandasToSpark').getOrCreate()
# Assuming file is csv
pandas_df = pd.read_csv(filename)
spark_df = spark.CreateDataFrame(pandas_df)
Posted by: Guest on March-14-2021
0

create spark dataframe from pandas

import numpy as np
import pandas as pd

# Enable Arrow-based columnar data transfers
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

# Generate a pandas DataFrame
pdf = pd.DataFrame(np.random.rand(100, 3))

# Create a Spark DataFrame from a pandas DataFrame using Arrow
df = spark.createDataFrame(pdf)
Posted by: Guest on March-27-2021
-1

spark to pandas

pandas_df = some_df.toPandas()
Posted by: Guest on March-11-2021

Python Answers by Framework

Browse Popular Code Answers by Language