Now that you have downloaded and extracted the Spark file, follow these steps:

1. Move and rename Spark folder:

   1. Move the extracted Spark folder to your C: drive.

   2. Rename the folder to spark (C:\spark).

2. Add Spark to path (SPARK_HOME):
   1. Navigate to Windows > System Properties.
   2. Click Environment Variables.
   3. Under System variables, click New...
   4. In the Variable name field, enter SPARK_HOME.
   5. In the Variable value field, enter C:\spark.
   6. Click OK.
   7. Select the Path variable from System variables and click Edit.
   8. Click New and add %SPARK_HOME%\bin.
   9. Click OK.
3. Create Hadoop folder and add Winutils:

   1. Create a new folder named hadoop in your C: drive (C:\hadoop).

   2. Inside the hadoop folder, create a bin folder.

   3. Move the winutils.exe file (which you have downloaded earlier) into this bin folder (C:\hadoop\bin\winutils.exe).

4. Add Hadoop to Path (HADOOP_HOME): Add this path to your system environment variables, similar to Spark:

   1. Navigate to Windows > System Properties > Environment Variables.

   2. Under "System variables," click New....
   3. In the "Variable name" field, enter HADOOP_HOME.

   4. In the "Variable value" field, enter C:\hadoop.

   5. Click OK.
   6. Select the "Path" variable from "System variables" and click Edit.
   7. Click New and add %HADOOP_HOME%\bin.
   8. Click OK.

To interact with Spark, you can open either a Spark shell or a PySpark shell in your terminal and execute commands.

• $ spark-shell: This command opens a Spark shell directly in your terminal, allowing you to run Spark scripts.

• $ pyspark:This command functions similar to spark-shell, opening a PySpark shell for Python-based Spark interactions.

Here are some basic commands you can try in the PySpark shell:

# Simple data
data = [
    ("Alice", 25, "Engineer"),
    ("Bob", 30, "Data Scientist"),
    ("Charlie", 35, "Manager"),
    ("Diana", 28, "Analyst")
]
columns = ["name", "age", "role"]
# Create DataFrame
df = spark.createDataFrame(data, columns)
# Show the data
df.show()
# Show with specific number of rows
df.show(2)  # Show only 2 rows