numpy中文文档(updating…)
numpy,scipy,matplotlib,pandas,keras,scikit-learn简明实例教程
- 基础部分
numpy的主要对象是一个同类元素的多维数组. 这是一个所有元素均为同种类型，并通过正整数元组来进行索引的元素(一般为数字)表. 在numpy中维度(dimensions)称之为轴(axes). 数目称之为秩(rank).
就比如，在3D空间中一个点的坐标[1, 2, 1]就是一个秩为1的数组，因为它仅有一个轴, 并且其长度为3. 又比如在下面的例子中，数组的秩为2(有两个维度)，第一个维度(轴)的长度为2，第二个维度(轴)的长度为3.
[[1., 0., 0.],[0., 1., 2.]]numpy的数组类被称之为ndarray, 我们也将它叫做array. 需要注意的是，numpy.array与python标准库中的array.array是有区别的，后者仅处理一维数组并且只提供了少量的功能. 对于ndarray对象而言，比较重要的属性有：
ndarray.ndim 数组中轴(维度)的个数，在python的世界里，维度的个数是指秩
ndarray.shape 数组的维度. 这是一个表示数组在每一个维度上的大小的一个整数元组. 对于一个n行m列的矩阵而言，它的shape属性就为(n, m). 那么，这个元组的长度就必然为秩，或者为维度的个数，或为ndim属性
ndarray.size 数组中元素的总个数. 也就等于shape属性元组中各个元素的乘积.
ndarray.dtype 一个用来描述数组中元素类型的对象. 你能通过标准的python类型来创建或者直接指定dtype属性. 另外numpy也提供了它自己的数据类型. 例如，numpy.int32, numpy.int16, 以及numpy.float64, 等等.
ndarray.itemsize 数组中元素的字节大小(bytes). 例如，一个类型为float64的数组元素的itemsize为8(=64/8), 而一个类型为complex32的数组元素的itemsize为4(=32/8). 这个属性等价于ndarray.dtype.itemsize
ndarray.data 包含了实际数组元素的缓冲区. 通常我们不需要用这个属性，原因是，我们会用索引(功能)访问数组中元素.
……
Pandas
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import warningswarnings.filterwarnings('ignore')
plt.rcParams['font.sans-serif'] = ['SimHei'] # 用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = Falsedef test1():# for each Series, it includes `index`, so merging them into `DataFrame`, as corresponding index-value into a row# print(pd.Index([3]*4))# print(pd.Index(range(4)))# print(pd.date_range('20180201', periods=4)) # DatetimeIndex, default 'D' (calendar daily), `stride` as daily# print(pd.period_range('20180101', '2018-01-04')) # PeriodIndex# print(pd.Index(data=[i for i in 'ABCDEF']))# print(list(pd.RangeIndex(10)))# s = pd.Series(10) # scalar# s = pd.Series(data=[1, 2, 3], index=[10, 20, 20]) # array-like, and non-unique index values are alloweds = pd.Series({
'a': 10, 10: 'AA'}, index=['aa', 10]) # dictprint(s) # print(s[:])# df = pd.DataFrame(data=np.random.randn(4, 3), index=pd.RangeIndex(1, 5), columns=['A', 'B', 'C']) # ndarray# df = pd.DataFrame(data={'A': np.array(range(1, 4))**2, 'B': pd.Timestamp('20180206'),# 'C': pd.Series(data=['MLee', 'python', 'Pearson']), 'D': 126,# 'E': pd.Categorical(values=['Upper', 'Middle', 'Lower'], categories=['Middle', 'Lower']),# 'F': 'Laplace'}, index=pd.RangeIndex(3), columns=['A', 'B']) # dictdf = pd.DataFrame(data={
'A': np.array(range(1, 4))**2, 'B': pd.Timestamp('20180206'),'C': pd.Series(data=['MLee', 'python', 'Pearson']), 'D': [126, 10, 66],'E': pd.Categorical(values=['Upper', 'Middle', 'Lower'], categories=['Middle', 'Lower']),'F': 'Laplace'}, index=pd.RangeIndex(3), columns=pd.Index([i for i in 'FEDCBA'])) # dictprint(df)# print(df.dtypes)# print(df.index)# print(df.columns)# print(df.values) # numpy.ndarray# print('*'*126)# print(df.info())# print('*'*126)# print(df.describe())# print(df.transpose())# print(df.sort_index(axis=0, ascending=False))# print(df.sort_index(axis=1))# print(df.sort_values(by='D'))print('*'*126)df = pd.DataFrame(data=np.arange(24).reshape(6, 4), index=pd.date_range('20180201', periods=6),columns=pd.Index([i for i in 'ABCD']))# df = pd.DataFrame(data=np.arange(24).reshape(6, 4))# print(df[0])# print(df[0:1]) # select rows require to use the sliceprint(df[:])print(df['A']) # print(df.A)print(df[0:2][['A', 'B']])print(df['20180201':'20180202'][['A', 'B']])# select rows require to use the `slice`, while select columns require to use the `list`# print(df[['A', 'B']])# print(df[0:2]) # exclude 3th row# print(df['20180201':'20180203']) # include index `20180203` row# print(df[0:1])# print(df.loc['20180201']) # enable to get only one# For row & column, df.loc requires to use `index` and `column`, while df.iloc requires to use `slice` and `slice`,# in particular, df.ix supports mixed-selectionprint(df.loc['20180201':'20180202'][['A', 'B']])print(df.loc['20180201':'20180202', ['A', 'B']]) # print(df['20180201':'20180202', ['A', 'B']]) # error# equivalent to df.iloc[0, 0]print(df.iloc[0:1, 0:1]) # use index(both row and column, necessarily all like 0, 1, 2, ...)print(df.iloc[[0, 2, 4], 0:2])print(df.ix[0:2, 0:2])# print(df.ix['20180201':'20180202', 0:2])# print(df.ix[0:2, ['A', 'B']])# print(df.ix['20180201':'20180202', ['A', 'B']])print('**')print(df['B'][df.A>4]) # print(df.B[df.A>4])df.B[df.A > 4] = np.nanprint(df)df['E'] = 0print(df)df['F'] = pd.Series(data=range(6), index=pd.date_range('20180201', periods=6))print(df)print('*'*12)df = pd.DataFrame(data=np.arange(24).reshape(6, 4), index=pd.date_range('20180201', periods=6), columns=pd.Index([i for i in 'ABCD']))# print(df)# df.dropna()# df.fillna()def test2():dataset_training = pd.read_csv('C:/users/myPC/Desktop/ml/Titanic/train.csv')# print(dataset_training)print(dataset_training.Survived.value_counts())deceased = dataset_training.Pclass[dataset_training.Survived == 0].value_counts(sort=True)survived = dataset_training.Pclass[dataset_training.Survived == 1].value_counts(sort=True)# print(deceased, survived, sep='\n')df = pd.DataFrame({
'Survived': survived, 'Deceased': deceased})print(df)df.plot(kind='bar', stacked=True)plt.title('Distribution of SES')plt.xlabel('Class')plt.ylabel('Numbers')plt.show()def test3():id = ['1001', '1008', '1102', '1001', '1003', '1101', '1126', '1007']name = ['Shannon', 'Gauss', 'Newton', 'Leibniz', 'Taylor', 'Lagrange', 'Laplace', 'Fourier']country = ['America', 'Germany', 'Britain', 'Germany', 'Britain', 'France', 'France', 'France']iq = [168, 180, 172, 228, 182, 172, 160, 186]sq = [180, 194, 160, 274, 150, 200, 158, 180]eq = [144, 152, 134, 166, 118, 144, 156, 128]dataset = list(zip(id, name, country, iq, sq, eq))df = pd.DataFrame(data=dataset, columns=['Id', 'Name', 'Country', 'IQ', 'SQ', 'EQ'])df.to_csv('persons.csv', index=True, header=True)df = pd.read_csv('persons.csv', usecols=range(1, 7))print(df)# print(df.info())# print(df[df.IQ == df.IQ.max()])print(df.sort_values(by='IQ', axis=0, ascending=False)) # df.head(1)plt.subplot2grid((1, 3), (0, 0))df.IQ.plot()df.SQ.plot()df.EQ.plot()for i in range(df.shape[0]):plt.annotate(s=df.ix[i, 'Name'], xy=(i, df.ix[i, 'IQ']), xytext=(1, 1), xycoords='data', textcoords='offset points')plt.subplot2grid((1, 3), (0, 1), colspan=2)df[['IQ', 'SQ', 'EQ']].plot(kind='bar')# df['IQ'].plot(kind='bar')# df['SQ'].plot(kind='bar')# df['EQ'].plot(kind='bar')for i in range(df.shape[0]):plt.annotate(s=df.ix[i, 'Name'], xy=(i, df.ix[i, 'IQ']), xytext=(1, 1), xycoords='data', textcoords='offset points')plt.show()def test4():data_train = pd.read_csv(r"C:\Users\myPC\Desktop\ml\Titanic\train.csv")# plt.subplot2grid((2, 3), (0, 0)) # 在一张大图里分列几个小图survived = data_train.Pclass[data_train.Survived == 1].value_counts()deceased = data_train.Pclass[data_train.Survived == 0].value_counts()pd.DataFrame({
'Survived': survived, 'deceased': deceased}).plot(kind='bar', stacked=True)# print(data_train.Sex[data_train.Survived == 1].value_counts())print(data_train.groupby(by='Survived').count())# data_train.Survived.value_counts().plot(kind='bar') # 柱状图# plt.title("获救情况 (1为获救)")# plt.ylabel("人数")# plt.subplot2grid((2, 3), (0, 1))# data_train.Pclass.value_counts().plot(kind="bar")# plt.ylabel("人数")# plt.title("乘客等级分布")## plt.subplot2grid((2, 3), (0, 2))# plt.scatter(data_train.Survived, data_train.Age)# plt.ylabel("年龄") # 设定纵坐标名称# plt.grid(b=True, which='major', axis='y')# plt.title("按年龄看获救分布 (1为获救)")x## plt.subplot2grid((2, 3), (1, 0), colspan=2)# data_train.Age[data_train.Pclass == 1].plot(kind='kde')# data_train.Age[data_train.Pclass == 2].plot(kind='kde')# data_train.Age[data_train.Pclass == 3].plot(kind='kde')# plt.xlabel("年龄") # plots an axis lable# plt.ylabel("密度")# plt.title("各等级的乘客年龄分布")# plt.legend(('头等舱', '2等舱', '3等舱'), loc='best') # sets our legend for our graph.## plt.subplot2grid((2, 3), (1, 2))# data_train.Embarked.value_counts().plot(kind='bar')# plt.title("各登船口岸上船人数")# plt.ylabel("人数")plt.show()def test5():url = r'http://s3.amazonaws.com/assets.datacamp.com/course/dasi/present.txt'present = pd.read_table(url, sep=' ')# print(present)# present.set_index(keys=['year'], inplace=True)# print(present)print(present.columns)print(present.index)print(present.dtypes)# present.boys.plot(kind='kde')# present.girls.plot(kind='kde')present.set_index(keys=['year'], inplace=True)kinds = ['line', 'bar', 'barh', 'hist', 'box', 'kde', 'density', 'area', 'pie', 'scatter', 'hexbin']# plt.figure()# for i in range(len(kinds)):# plt.subplot2grid(shape=(2, 3), loc=(i//3, i % 3))# present[:10].plot(kind=kinds[i], subplots=True)present[:].plot(x='boys', y='girls', kind=kinds[-1])plt.legend(loc='upper right')plt.show()def test6():s = pd.Series(data=np.random.randn(1000), index=pd.date_range('20180101', periods=1000))print(s)s = np.exp(s.cumsum())s.plot(style='m*', logy=True)plt.show()def test7():df = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data',names=['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'name'])# print(df)# df.boxplot(by='name')# df.plot(kind='kde')# df.ix[:, :-1].plot(kind='hist')setosa = df[df.name == 'Iris-setosa']versicolor = df[df.name == 'Iris-versicolor']virginica = df[df.name == 'Iris-virginica']# plt.subplot2grid(shape=(1, 3), loc=(0, 0))plt.subplot(131)pd.DataFrame.plot(setosa)# setosa.plot(title='setosa', subplots=True)# plt.subplot2grid(shape=(1, 3), loc=(0, 1))# versicolor.plot(title='versicolor', subplots=True)# pd.DataFrame.plot(versicolor)# plt.subplot2grid(shape=(1, 3), loc=(0, 2))# virginica.plot(title='virginica', subplots=True)# pd.DataFrame.plot(data=virginica)plt.show()# df.sepal_length.plot(kind='hist')# plt.show()def test8():import numpy as npimport pandas as pdfrom sklearn.ensemble import RandomForestRegressorfrom sklearn import preprocessingfrom sklearn import linear_modeldataset_training = pd.read_csv('C:/users/myPC/Desktop/ml/Titanic/train.csv')dataset_test = pd.read_csv('C:/users/myPC/Desktop/ml/Titanic/test.csv')passenger_id = dataset_test['PassengerId']# for the feature `Fare` in the `test_data`, only one misseddataset_test.loc[dataset_test.Fare.isnull(), 'Fare'] = 0.0# drop the irrelevant featuresdataset_training.drop(labels=['PassengerId', 'Name', 'Ticket'], axis=1, inplace=True)dataset_test.drop(columns=['PassengerId', 'Name', 'Ticket'], inplace=True)# predict `age` which is missed by others' featuresdataset_training_age = dataset_training[['Pclass', 'SibSp', 'Parch', 'Fare', 'Age']]dataset_test_age = dataset_test[['Pclass', 'SibSp', 'Parch', 'Fare', 'Age']]age_known0 = dataset_training_age[dataset_training_age.Age.notnull()].as_matrix() # get the `ndarray`age_unknown0 = np.array(dataset_training_age[dataset_training_age.Age.isnull()])age_unknown1 = dataset_test_age[dataset_test_age.Age.isnull()].as_matrix()training_data_age = age_known0[:, :-1]training_target_age = age_known0[:, -1]rfr = RandomForestRegressor(n_estimators=1000, n_jobs=-1, random_state=0) # enable to fit them by the 1000 treesrfr.fit(training_data_age, training_target_age)predicts = rfr.predict(age_unknown0[:, :-1])dataset_training.ix[dataset_training.Age.isnull(), 'Age'] = predicts # fill the `age` which is missed# fit model(RandomForestRegressor) by the `training data`dataset_test.loc[dataset_test.Age.isnull(), 'Age'] = rfr.predict(age_unknown1[:, :-1])dataset_training.ix[dataset_training.Cabin.notnull(), 'Cabin'] = 'Yes' # fill the `Cabin` as `Yes` which `notnull`dataset_training.ix[dataset_training.Cabin.isnull(), 'Cabin'] = 'No' # else, `No`dataset_test.ix[dataset_test.Cabin.notnull(), 'Cabin'] = 'Yes'dataset_test.ix[dataset_test.Cabin.isnull(), 'Cabin'] = 'No'# dummy some fields whose types of [`object`, `category`] to eliminate relation between categoriesdataset_training_dummies = pd.get_dummies(dataset_training, columns=['Pclass', 'Sex', 'Cabin', 'Embarked'])dataset_test_dummies = pd.get_dummies(dataset_test, columns=['Pclass', 'Sex', 'Cabin', 'Embarked'])ss = preprocessing.StandardScaler() # standardize some features which have some differencesdataset_training_dummies['Age'] = ss.fit_transform(dataset_training_dummies.Age.reshape(-1, 1))dataset_training_dummies['Fare'] = ss.fit_transform(dataset_training_dummies.Fare.reshape(-1, 1))dataset_test_dummies['Age'] = ss.fit_transform(dataset_test_dummies.Age.reshape(-1, 1))dataset_test_dummies['Fare'] = ss.fit_transform(dataset_test_dummies.Fare.reshape(-1, 1))# get all processed samplesprint(dataset_training_dummies)dataset_training_dummies = dataset_training_dummies.filter(regex='Age|SibSp|Parch|Fare|Pclass_*|Sex_*|Cabin_*|Embarked_*|Survived').as_matrix()# print(data_training_dummies.info())training_data = dataset_training_dummies[:, 1:]training_target = dataset_training_dummies[:, 0:1]lr = linear_model.LogisticRegression(C=1.0, penalty='l1', tol=1e-5)from sklearn import model_selectionprint(model_selection.cross_val_score(lr, training_data, training_target, cv=4))lr.fit(training_data, training_target)predicts = lr.predict(dataset_test_dummies)ans = pd.DataFrame({
'PassengerId': passenger_id, 'Survived': predicts.astype(np.int32)})# print(ans)# ans.to_csv('C:/users/myPC/Desktop/ml/Titanic/submission.csv', index=False) # ignore label-index# print(pd.DataFrame({'features': list(dataset_test_dummies[1:]), 'coef': list(lr.coef_.T)}))def test9():import numpy as npimport numpy.linalg as nlaimport scipy.linalg as slaa = np.random.randint(20, size=(3, 4))print(a)print(np.diag(a))U, Sigma, V_H = nla.svd(a) # 其中U, V为酉矩阵，Sigma为一个由奇异值组成的对角矩阵(但返回的是一个由奇异值组成的向量形式)Sigma = np.concatenate((np.diag(Sigma), np.zeros((U.shape[0], V_H.shape[1]-U.shape[1]))), axis=1)print(U)print(Sigma)print(V_H)print(U.dot(Sigma.dot(V_H)))def google():import tensorflow as tfa = tf.constant((1, 1))b = tf.constant((2, 2))ans = a + bsess = tf.Session()# print(type(sess.run(ans)))print(sess.run(ans))if __name__ == '__main__':# test9()# google()test8()# pd.concat()# df.drop()# df = pd.DataFrame({'A': [1, 2, 3], 'B': [np.nan, 1, np.nan], 'C': [10, 111, 1111], 'D': ['good', 'common', 'bad']})# df.loc[df.B.notnull(), 'B'] = 'Yes' # prior to judge `isnull()`, or leading to all values as the `Yes`# df.loc[df.B.isnull(), 'B'] = 'No'# print(df.ix[:, 'B'])# df.ix[df.B.isnull(), 'B'] = [0, 0]# print(pd.get_dummies(df, columns=['D', 'B']))# print(df)# print(df.filter(regex='A|D|B'))# df = pd.get_dummies(df, prefix=['M', 'L']) # loss original data(variable)# print(df)