Note: a free tool converting Metastock data to text files is available.
The source code is available on GitHub: git://github.com/themech/ms2txt.git.
Last week I decided to check how my trading system performs while playing on different foreign indexes. First I had to download the test data. I found a web page offering the quotations I was interested in – luckily it wasn’t expensive.The problem was (of course it occurred after I had paid) that the data was available only in Metastock format. Of course I use my own software (just as every other programmer 😉 ) that helps me to play stocks and futures and I don’t have Metastock. So I decided I would give it a try and I wrote a tiny program that reads Metastock files and generates text files with quotes. I used my beloved Python. You can find the source here.
Metastock format
The data was in Metastock 6.x format (XMASTER file was not present). Generally Metastock data consist of:
- MASTER/EMASTER file which holds general information about the tickers, stock names. This is an index file
- F{n}.DAT files which hold actual quotation data.
- Every F{n}.DAT file has a corresponding F{n}.DOP file which holds an information about the data columns available in the DAT file
The EMASTER file is essential because it holds the references to F{n}.DAT. So at the beginning we open EMASTER file and read what quotations are available and in which DAT files they are held. The following data describes each symbol:
- file number
- number of fields in one quotation
- stock symbol
- stock name
- first date and last date
After reading the description we open the appropriate DAT file and read the quotations. Each row contain at least the date, open price, highest prices, lowest price and close price. Optionally there data contains volume and open positions data. The problem is that each data entry is a float number. To make is worse it is in Microsoft Binary floating point format :), while most of the programming languages use IEEE floating point format.
Microsoft Binary Format floating point number
In the ’80’s Microsoft had a proprietary binary structure to handle floating point numbers. IEEE format (used nowadays) is more accurate and MBF format is generally not supported nowadays. Unfortunately Metastock still uses MBF so we have to handle it.
We have to convert the 4 bytes of the MS Binary Format to IEEE. After some googling I found a little piece of code in pure C and I’ve rewritten it using Python. The function is quite simple:
def fmsbin2ieee(bytes):
"""Convert an array of 4 bytes containing Microsoft Binary
floating point number to IEEE floating point format
(which is used by Python)"""
r = struct.unpack("i", bytes)
if not r:
return 0.0
man = long(struct.unpack('H', bytes[2:])[0])
if not man:
return 0.0
exp = (man & 0xff00) - 0x0200
man = man & 0x7f | (man < < 8) & 0x8000
man |= exp >> 1
bytes2 = bytes[:2]
bytes2 += chr(man & 255)
bytes2 += chr((man >> 8) & 255)
return struct.unpack("f", bytes2)[0]
This little function returns a valid Python floating point number.
Handling floats
As I said every data entry is a floating point number which represents different data types. Converting it to integer is straightforward. But converting it to a data or time is more tricky. The following functions show how to extract date and time from floating point numbers:
def float2date(date): """Convert a float to a string containig a date""" date = int(date) year = 1900 + (date / 10000) month = (date % 10000) / 100 day = date % 100 return '%04d%02d%02d' % (year, month, day) def float2time(time): """Convert a float to a string containig a time""" time = int(time) hour = time / 10000 min = (time % 10000) / 100 return '%02d%02d' % (hour, min)
About the tool
The tool I included should be run inside a directory containing Metastock quotations. It opens EMASTER file and lets you do the following things:
- list all the symbols
- extract to text files all the symbols
- extract to text files specified symbols
For those who don’t have python installed I included this tool as a Windows executable. Download it here.
P.S. There is another open source Metastock converter written by Rudi: https://github.com/rudimeier/atem. It’s written in C and the source is very well commented, so I recommend checking it out if you’re interested in reading MS files.
Hello,
Great post!
I am wanting to do something similar in C#. Do you know how to convert your python methods to C# so that MASTER/XMASTER and .DAT/.MWD files can be read?
sfx,
Doing the same thing in C# shouldn’t be difficult. While writing my tool I was using the following Metastock datafile format description: http://www.purebytes.com/archives/metastock/2001/msg04890.html. I belive you can find the XMASTER file format description easily. The tricky part is to convert MBF bytes to float numbers. I found an article about it with ready-to-use C# code: http://www.boyet.com/Articles/MBFSinglePrecision.html. The rest of my code is very simple and I think you can rewrite it in C# with just minor modifications.
This is a stunning piece of work. It doesn’t just work for Metastock files, it works for CSIM files too. And he supplies the Python code too. Thank you. You saved me many, many hours.
Many thanks! Great piece of work!
Not sure whether this is still active or not.
Just wanted to update on same topic that recently, I used some of your code to perform read and write operations on latest metastock format files.
developed in python, I used mbs2Ieee, float2date function of yours and developed reverse functions. Implemented writing and merging functionality as well. Its working fine.
I looked at your repository, not sure why the activity stopped. if anybody interested in extending it, i’m ready to give my code.
My code performs read, write, merge oprations on Master, EMaster, XMaster and data files (DAT, MWD).
If anybody interested, mail to narendra.kamma at gmail dot com
narendra,
This blog entry is quite old. I just needed to read the Metastock quotations and convert it to other format. As there was very little information on the web about Metastock file format and MBF, I thought I would share this knowledge.
As you’ve noticed, it’s just a starting point for reading metastock files, there are a lot of topics not covered here.
I’m impressed with the functionality you’ve added and I’m interested to see the details 🙂 I’m also glad you want to share this with others. The best way I think is to put your source code on some public source code hosting server (like github). If you would simply like to update my project on github, I can grant to you the necessary privileges.
Hi, I used it but only could read the sysmbols that are in master data. there are many on XMASTER. how can I extract the ohter symbols ? is it possible ? tks regards
Gabriel,
narendra (you can read his comment above) created a tool that reads XMASTER files and does many other things. I suggest contacting him or checking his website (http://www.kakup.com/)
Hello, mech.
I’ve just downloaded your source code from github. It’s producing an error but no sweat, I’ll go through the python code and see what’s going wrong.
Meanwhile, I do want to thank you, and ask Narendra if he would be kind enough to post his source code for writing operations. I got the .exe off the kakup website but can’t find an address for him there (didn’t want to use the contact-at address in case it goes to the wrong person).
I had to do following changes because python reads one more line from some files:
— a/metastock/files.py
+++ b/metastock/files.py
@@ -43,9 +43,10 @@ class DataFileInfo(object):
file_handle = open(filename, ‘r’)
lines = file_handle.read().split()
file_handle.close()
– assert(len(lines) == self.num_fields)
+ assert(len(lines) > self.num_fields)^M
self.columns = []
– for line in lines:
+ for position in range(self.num_fields):^M
+ line = lines[position]^M
match = self.reg.search(line)
colname = match.groups()[0]
self.columns.append(colname)
hi all,
thank you for your work, it is very useful for many people. I downloaded the compiled utility ms_convert.exe and compared its output with that of Metastock Downloader noting some differences.
It seems ms_convert rounds the numbers to the second decimal, and put 1 instead of 0 for empty fields. Are you awar e of that? Do you have a newer routine or some code that bypass the problem? I can send the compared files if you are interested in… many thanks
Hello,
Did any one was able to get data from master.dat using c#? I am facing many issues and struggling to get the get data. Please let us know if anyone was able to do this?
Thanks,
Any idea how to convert ascii tick information into Metastock 10 format?
I am looking to load real time tick information into Metastock. Any help is appreciated.
There is a great open source tool written in C++ that reads Metastock files: https://github.com/rudimeier/atem. It’s very nicely written and has a lot of comments in the code, you can learn a lot from it.
Hey mech, thanks for the script, it worked really well.
Just a note, if anyone is facing the same situation as me, where you don’t have the DOP files, you can just edit the “files.py” source and try setting the columns configuration manually. In my case I just set “self.columns = [“DATE”, “OPEN”, “HIGH”, “LOW”, “CLOSE”, “VOL”, “OI”]”.
Is it possible to modify the code to a Visual Basic subroutine? Especially one that could handle Excel 64 bit? Thanks very much.