I wrote Python code to create a transition probability matrix from the data, but I continue to get the wrong values โโfor two specific data points. I spent several days trying to figure out the problem, but without success.
About the code: The entry consists of 4 columns in a csv file. After preparing the data, the first two columns represent the new and old state values. I need to calculate how often each old state value is transferred to a new one (basically, how often each pair (x, y) occurs in the first two columns of data). The values โโin these columns are from 0 to 99. In the trans_pr matrix I want to get a number, how often the pair (x, y) occurs in the data and has this number in the corresponding coordinates (x, y) in the trans_pr matrix. Since the values โโare from 0 to 99, I can simply add 1 to the matrix in these coordinates every time they appear in the data.
Problem: the code works fine, but I always get zeros in the coordinates (:, 29) and (:, 58) and (29, :) and (58; :), despite the fact that there are observations. It also sometimes seems that a number in these coordinates is added to the previous line. Again, this makes no sense to me.
I would really appreciate it if anyone could help. (I'm new to Python, so the code is probably inefficient, but only the error is important.)
The code is as simple as it can be:
from numpy import *
import csv
my_data = genfromtxt('99c_test.csv', delimiter=',')
"""prepares data for further calculations"""
my_data1=zeros((len(my_data),4))
my_data1[1:,0]=100*my_data[1:,0]
my_data1[1:,1]=100*my_data[1:,3]
my_data1[1:,2]=my_data[1:,1]
my_data1[1:,3]=my_data[1:,2]
my_data2=my_data1
trans_pr=zeros((101,101))
print my_data2
"""fills the matrix with frequencies of observations"""
for i in range(len(my_data2)):
trans_pr[my_data2[i,1],my_data2[i,0]]=trans_pr[my_data2[i,1],my_data2[i,0]]+1
c = csv.writer(open("trpr1.csv", "wb"))
c.writerows(trans_pr)
You can test the code with this input (just save it as a csv file):
p_cent,p_euro,p_euro_old,p_cent_old
0.01,1,1,0.28
0.01,1,1,0.29
0.01,1,1,0.3
0.01,1,1,0.28
0.01,1,1,0.29
0.01,1,1,0.3
0.01,1,1,0.57
0.01,1,1,0.58
0.01,1,1,0.59
0.01,1,1,0.6