Data analysis and modification with Python

[burdickjp]

I've seen a number of tools for doing data analysis on here. Most are done with spreadsheets, which aren't conducive to large amounts of data. Others are written in Java, or .NET, which isn't bad, but isn't conducive to the end user wanting to learn not just what the tool does but how it does it.
Python is, by its nature, a human readable language, so I'd like to share with you how to do some data analysis in Python.

If you're running Linux you probably already have Python. We need the libraries called pandas and bokeh. I assume, since you are running Linux, that you know how to get these. I prefer using Python3.
If you're running anything else you can get an all-inclusive package called Anaconda which pulls everything in for you. Anaconda includes several environments for playing with Python. I prefer Spyder, as it has a variable explorer similar to MatLab. Think of it as being a MatLab environment, but better, because it's not MatLab.

Once you've got everything up, fire up Spyder. Throw this into the left pane and hit the green arrow. You'll want to edit line 13 to point to the log you'd like to use. Line 15 is the X axis of the graph. Line 16 is the Y axis.

# -*- coding: utf-8 -*-
 """
 Created on Thu Jul  9 22:06:52 2015
 

 @author: jeff
 """
 

 import pandas as pd
 from bokeh.plotting import figure, HBox, output_file, show, VBox
 

 output_file("graph.html")
 

 logRaw = pd.read_csv('log.csv')
 

 p1x = logRaw['time']
 p1y = logRaw['RPM']
 

 p1 = figure(tools="pan,wheel_zoom,box_zoom,reset,save", plot_width=1024, plot_height=600)
 p1.scatter(p1x, p1y, size=1, color="red", alpha=0.5)
 

 show(VBox(HBox(p1)))

I'll be updating this to include more awesome things like 3d graphs, etc. I'd eventually like to get MAF scaling tools and others ported to Python.

[Kodename47]

If I remove the top line I get this error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 24: invalid start byte

If I put the 1st line in I get:
p, li { white-space: pre-wrap; } p, li { white-space: pre-wrap; } # -*- coding: utf-8 -*-
^
SyntaxError: invalid syntax

[burdickjp]

Originally Posted by Kodename47

If I remove the top line I get this error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 24: invalid start byte

If I put the 1st line in I get:
p, li { white-space: pre-wrap; } p, li { white-space: pre-wrap; } # -*- coding: utf-8 -*-
^
SyntaxError: invalid syntax

Try it now. That "p, li { white-space: pre-wrap; } p, li { white-space: pre-wrap; }" was junk added in the posting process. If you open a blank spyder doc you'll see what the header is supposed to look like.

[Kodename47]

This is what I've put entered:

# -*- coding: utf-8 -*-
"""
 Created on Thu Jul  9 22:06:52 2015
 

 @author: jeff
"""
 

import pandas as pd
from bokeh.plotting import figure, HBox, output_file, show, VBox
 

output_file("graph.html")
 

logRaw = pd.read_csv('C:\EcuTek\MapAccessLogs\Sprintex 70mm-01-03-2015.csv')
 

p1x = logRaw['Time']
p1y = logRaw['FLKC']
 

p1 = figure(tools="pan,wheel_zoom,box_zoom,reset,save", plot_width=1024, plot_height=600)
p1.scatter(p1x, p1y, size=1, color="red", alpha=0.5)
 

show(VBox(HBox(p1)))

I still get the same unicode decode error as above. Is it a log file length/format thing?

[phrosty]

Why do you believe Python is, by its nature, a more human readable language than the other ones you mentioned (Java and C#)?

[burdickjp]

Originally Posted by phrosty

Why do you believe Python is, by its nature, a more human readable language than the other ones you mentioned (Java and C#)?

I'm not a programmer, so the specifics of it are lost on me. I know that python is much easier for ME to read and understand than other languages. The only c-like language I've seen as similarly easy is wiring (arduino programming). It's my understanding wiring handles a lot in the background to make it easier in the foreground.

There's a reason python is experiencing explosive growth in academia, science, and research. It's not necessarily on topic here, but I LOVE python and feel it's the right tool for this job.

[burdickjp]

Originally Posted by Kodename47

This is what I've put entered:

# -*- coding: utf-8 -*-
"""
 Created on Thu Jul  9 22:06:52 2015
 

 @author: jeff
"""
 

import pandas as pd
from bokeh.plotting import figure, HBox, output_file, show, VBox
 

output_file("graph.html")
 

logRaw = pd.read_csv('C:\EcuTek\MapAccessLogs\Sprintex 70mm-01-03-2015.csv')
 

p1x = logRaw['Time']
p1y = logRaw['FLKC']
 

p1 = figure(tools="pan,wheel_zoom,box_zoom,reset,save", plot_width=1024, plot_height=600)
p1.scatter(p1x, p1y, size=1, color="red", alpha=0.5)
 

show(VBox(HBox(p1)))

I still get the same unicode decode error as above. Is it a log file length/format thing?

I'm not sure. Are you using anaconda with python 3 or python 2?

[wbradley]

Python: the programming language for Arduino.
Some cool project ts done with it. In the pinball machine hobby there are some real magicians. A guy named Ben Heck started with hacking game consoles and then controller U its for one off custom pinball machines.
Cool stuff.

[burdickjp]

Originally Posted by wbradley

Python: the programming language for Arduino.
Some cool project ts done with it. In the pinball machine hobby there are some real magicians. A guy named Ben Heck started with hacking game consoles and then controller U its for one off custom pinball machines.
Cool stuff.

I'm sure someone had python being compiled for arduino, but I'm most familiar with wiring and arduino. Again: right tool for the job. Wiring is great for it.

Sent from my XT1045 using Tapatalk

[Kodename47]

Originally Posted by bur****jp

I'm not sure. Are you using anaconda with python 3 or python 2?

I downloaded Python 3. I literally installed and then ran Spyder as you suggested.

[coyote]

Python is great for this kind of stuff. I use it a lot.

Sent from my Nexus 6 using Tapatalk

[burdickjp]

Originally Posted by Kodename47

I downloaded Python 3. I literally installed and then ran Spyder as you suggested.

There might be a bug with the current version of anaconda using Python3 and windows. Let me see what I can find.

Sent from my XT1045 using Tapatalk

[Kodename47]

Originally Posted by bur****jp

There might be a bug with the current version of anaconda using Python3 and windows. Let me see what I can find.

Sent from my XT1045 using Tapatalk

I'm on Windows8 64 bit. Should I revert to python2?

[burdickjp]

Originally Posted by Kodename47

I'm on Windows8 64 bit. Should I revert to python2?

Give it a try. I booted into windows and got a similar error.