Data mining, Stats, etc.

'Just came across a cool link and wanted to file it somewhere.  I thought I already had a blog post to gather up links like this, but I can't find it.  Oh well.

Seeing Theory

Python Installation

It's been a few years since I've needed to use python!

If you open Terminal and type "python", it will open the interpreter and tell you the version.  If you need to know where on your computer Python is installed, type the following:

>>> import sys
>>> sys.executable

If you don't already have python:
Notes:
Follow the MacPorts guide, which instructs you to install Xcode  (need Apple ID).  You'll also need to install the Xcod command line tools  ("xcode-select --install") and  accept the license at the command line (it gives you the command).  Next the guide provides a link to download and install Macports.
From there, install Python, nice guide here.

sudo port install python34

# make this new version of python default
sudo port select --set python python34

Example python package installation: 
sudo port install py34-readline
sudo port install py34-pandas

I also installed pip, which is supposed to be better for managing python packages.
sudo port install py34-pip
To ensure the most recent version of pip is default, I did:
port select --set pip pip34

Install python package, SQLalchemy
sudo pip install SQLAlchemy

sudo pip install pandas
sudo pip install pymysql

Next I needed pymssql but this needed freetds first.  First I tired:

sudo port install freetds  But that didn't work so I did:

brew install freetds

Then as per the message displayed in Homebrew after I installed freetds,
I had to set some environment variables:

Then manually link my homebrew-installed openssl to pip:

export LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib" 

export CPPFLAGS="-I/usr/local/opt/openssl/include"

export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

Then pymysql STILL wouldn't install so I did this:

brew unlink freetds; brew install homebrew/versions/freetds091

reference:  https://github.com/pymssql/pymssql/issues/432  

I had also tried:

sudo port install freetds +mssql +odbc +universal

ref: http://www.cerebralmastication.com/2013/01/installing-debugging-odbc-on-mac-os-x/

ref: https://gist.github.com/tommct/5749453 

ref: https://github.com/lionheart/django-pyodbc/wiki/Mac-setup-to-connect-to-a-MS-SQL-Server 

The following link was useful when I tried, again with difficulty, to do the above on my laptop.  I had trouble with the below because I couldn't get Homebrew to update.  It was giving Error: /usr/local is not writable. You should change the ownership
and permissions of /usr/local back to your user account:  So I did sudo chown -R $(whom) /usr/local and then changed it back with sudo chown root:wheel /usr/local.  On the last step, I had to do "sudo pip install pymssql" not just "pip install pymssql".
http://gree2.github.io/python/setup/2017/04/19/python-instal-pymssql-on-mac

For FreeTDS, you need to use the configuration file, freetds.conf.
Location of freetds.conf:
It will tell you if you type the following command:

tsql -C

Inside freetds.conf I had to use:

[myhost]
    host = myhost.whatever.ca
    port = 1433
    tds version = 8.0
    use ntlmv2 = yes

Then at the command line, you use the "tsql" command to check that FreeTDS is installed correctly.  The -S option links to [myhost] in freetds.conf.

tsql -S myhost -U DOMAIN\\username

Note the two slashes!   Before I figured out you need two slashes, I did lots of debugging . You can create a log file like this:

TDSDUMP=/Users/stephfrost/bin/freetds.log tsql -S DEFSQLDEV -U USASK/slb534

Specify TDS protocol version at command line:

TDSVER=8.0 tsql -S DEFSQLDEV -U USASK/slb534 

pip install pymssql
That failed so I did:
sudo pip install git+https://github.com/pymssql/pymssql.git

I had to do this in Eclipse:
Window -> Preferences -> PyDev -> Editor -> Code Analysis -> Undefined -> Undefined Variable From Import -> Ignore
ref:
https://stackoverflow.com/questions/2112715/how-do-i-fix-pydev-undefined-variable-from-import-errors

Useful link about importing Python libs:
https://stackoverflow.com/questions/4631377/unresolved-import-issues-with-pydev-and-eclipse