some spark commands

some spark commands, 28 Feb 2017

Suppose we have tiny dataframe as following

df = spark.createDataFrame([[1, 'Anna'], [2, 'Bryan']], schema=['index', 'name'])

We can find row with name Anna by using condition or filter

df[df['name'].like('%Anna%')].show()
df.filter(df['name'] == 'Anna').show()