如何遍历 Pandas 中 DataFrame 中的行

I have a DataFrame from Pandas:

import pandas as pd
inp = [{'c1':10, 'c2':100}, {'c1':11,'c2':110}, {'c1':12,'c2':120}]
df = pd.DataFrame(inp)
print df

Output:

   c1   c2
0  10  100
1  11  110
2  12  120

现在我想遍历这个框架的行。 对于每一行,我希望能够通过列名访问其元素(单元格中的值)。 例如:

for row in df.rows:
   print row['c1'], row['c2']

在 Pandas 中可以做到这一点吗?

我发现了这个类似的问题 。 但它没有给我我需要的答案。 例如,建议在那里使用:

for date, row in df.T.iteritems():

or

for row in df.iterrows():

但我不明白 row 对象是什么以及我如何使用它。

While iterrows() is a good option, sometimes itertuples() can be much faster:

df = pd.DataFrame({'a': randn(1000), 'b': randn(1000),'N': randint(100, 1000, (1000)), 'x': 'x'})

%timeit [row.a * 2 for idx, row in df.iterrows()]

=> 10 loops, best of 3: 50.3 ms per loop

%timeit [row[1] * 2 for row in df.itertuples()]

=> 1000 loops, best of 3: 541 µs per loop

</div>

How to iterate efficiently

If you really have to iterate a Pandas dataframe, you will probably want to avoid using iterrows(). There are different methods and the usual iterrows() is far from being the best. itertuples() can be 100 times faster.

In short:

  • As a general rule, use df.itertuples(name=None). In particular, when you have a fixed number columns and less than 255 columns. See point (3)
  • Otherwise, use df.itertuples() except if your columns have special characters such as spaces or '-'. See point (2)
  • It is possible to use itertuples() even if your dataframe has strange columns by using the last example. See point (4)
  • Only use iterrows() if you cannot the previous solutions. See point (1)

Different methods to iterate over rows in a Pandas dataframe:

Generate a random dataframe with a million rows and 4 columns:

    df = pd.DataFrame(np.random.randint(0, 100, size=(1000000, 4)), columns=list('ABCD'))
    print(df)

1) The usual iterrows() is convenient, but damn slow:

start_time = time.clock()
result = 0
for _, row in df.iterrows():
    result += max(row['B'], row['C'])

total_elapsed_time = round(time.clock() - start_time, 2)
print(“1. Iterrows done in {} seconds, result = {}”.format(total_elapsed_time, result))

2) The default itertuples() is already much faster, but it doesn't work with column names such as My Col-Name is very Strange (you should avoid this method if your columns are repeated or if a column name cannot be simply converted to a Python variable name).:

start_time = time.clock()
result = 0
for row in df.itertuples(index=False):
    result += max(row.B, row.C)

total_elapsed_time = round(time.clock() - start_time, 2)
print(“2. Named Itertuples done in {} seconds, result = {}”.format(total_elapsed_time, result))

3) The default itertuples() using name=None is even faster but not really convenient as you have to define a variable per column.

start_time = time.clock()
result = 0
for(_, col1, col2, col3, col4) in df.itertuples(name=None):
    result += max(col2, col3)

total_elapsed_time = round(time.clock() - start_time, 2)
print(“3. Itertuples done in {} seconds, result = {}”.format(total_elapsed_time, result))

4) Finally, the named itertuples() is slower than the previous point, but you do not have to define a variable per column and it works with column names such as My Col-Name is very Strange.

start_time = time.clock()
result = 0
for row in df.itertuples(index=False):
    result += max(row[df.columns.get_loc('B')], row[df.columns.get_loc('C')])

total_elapsed_time = round(time.clock() - start_time, 2)
print(“4. Polyvalent Itertuples working even with special characters in the column name done in {} seconds, result = {}”.format(total_elapsed_time, result))

Output:

         A   B   C   D
0       41  63  42  23
1       54   9  24  65
2       15  34  10   9
3       39  94  82  97
4        4  88  79  54
...     ..  ..  ..  ..
999995  48  27   4  25
999996  16  51  34  28
999997   1  39  61  14
999998  66  51  27  70
999999  51  53  47  99

[1000000 rows x 4 columns]

  1. Iterrows done in 104.96 seconds, result = 66151519
  2. Named Itertuples done in 1.26 seconds, result = 66151519
  3. Itertuples done in 0.94 seconds, result = 66151519
  4. Polyvalent Itertuples working even with special characters in the column name done in 2.94 seconds, result = 66151519

This article is a very interesting comparison between iterrows and itertuples

You can write your own iterator that implements namedtuple

from collections import namedtuple

def myiter(d, cols=None):
if cols is None:
v = d.values.tolist()
cols = d.columns.values.tolist()
else:
j = [d.columns.get_loc(c) for c in cols]
v = d.values[:, j].tolist()

n = namedtuple('MyTuple', cols)

for line in iter(v):
    yield n(*line)

This is directly comparable to pd.DataFrame.itertuples. I'm aiming at performing the same task with more efficiency.


For the given dataframe with my function:

list(myiter(df))

[MyTuple(c1=10, c2=100), MyTuple(c1=11, c2=110), MyTuple(c1=12, c2=120)]

Or with pd.DataFrame.itertuples:

list(df.itertuples(index=False))

[Pandas(c1=10, c2=100), Pandas(c1=11, c2=110), Pandas(c1=12, c2=120)]


A comprehensive test
We test making all columns available and subsetting the columns.

def iterfullA(d):
    return list(myiter(d))

def iterfullB(d):
return list(d.itertuples(index=False))

def itersubA(d):
return list(myiter(d, [‘col3’, ‘col4’, ‘col5’, ‘col6’, ‘col7’]))

def itersubB(d):
return list(d[[‘col3’, ‘col4’, ‘col5’, ‘col6’, ‘col7’]].itertuples(index=False))

res = pd.DataFrame(
index=[10, 30, 100, 300, 1000, 3000, 10000, 30000],
columns=‘iterfullA iterfullB itersubA itersubB’.split(),
dtype=float
)

for i in res.index:
d = pd.DataFrame(np.random.randint(10, size=(i, 10))).add_prefix(‘col’)
for j in res.columns:
stmt = ‘{}(d)’.format(j)
setp = ‘from main import d, {}’.format(j)
res.at[i, j] = timeit(stmt, setp, number=100)

res.groupby(res.columns.str[4:-1], axis=1).plot(loglog=True);

In short

  • Use vectorization if possible
  • If an operation can't be vectorized - use list comprehensions
  • If you need a single object representing the entire row - use itertuples
  • If the above is too slow - try swifter.apply
  • If it's still too slow - try a Cython routine

Benchmark