python and synthetic listmode data

Time to blow the dust off my little blog. How y'all been?

today I'm documenting a little exercise I did in python to make synthetic FCS files. Why on earth would I want synthetic ones you ask? because unlike in "real" data, synthetic data has "known" distributions and properties. You can use "synthetic" data to validate your software's summary stats.

For this exercise we'll need:
- a Mac with Python. Python is installed by default on my Leopard system.
- Peter Borg's Smultron - get it here.
- FlowJo V8 for Mac.

First, I like to generate a bunch of lismode values for some distribution. In the attached python code, the function "linvar" will take a mean, sd and number of events, and generate a series of gaussians, each with half the mean. This creates a text file called "test_lin.txt"

Second, I write out a header of FJTEXTDATA format (described here) with the function "header". This calculates the range keyword $P#R to the nearest power of 2 which is bigger than the biggest value in your stream. The FJTEXTDATA format works well up to 32768, or 2^15. Bigger bits/ranges cause complications I won't go into here. The result of this is a text file called "header.txt".

Third, I merge the data stream and the header with the function "merger". This appends "test_lin.txt" to "header.txt" and makes the file "sample.txt".

Fourth, we have to take the sample.txt file into smultron, select the whole text, and click Text menu, Line Endings, Mac (CR). Then save.

Now you can drag "sample.txt" into flowjo. Shown below are some results.. I used the "derived parameter" platform to make the log version shown on the right:

Here's the python source code:

Download test.py

To use it, start terminal and change directory to where your test.py lives. For me that's in the user home dir. Once you're in the right folder, type "python" without quotes and hit enter:

to make python aware of your script file, type "import test" without quotes, and hit enter. It won't say anything back, you'll just get a new prompt line.

Now type each of the three lines below, followed by enter. Each command will make a different text file, the last one being "sample.txt".

test.linvar(30000,250,1000)
test.header()
test.merger()

this will make the following text file:

Download sample.txt

which you can drag into flowjo and do with what you please, including exporting it into an actual FCS file.

For my next trick, I'll try to do this same thing in R. :)

Comments

[Mannie]

Hi, would you be aware if FlowJo or any other software can allow manipulation & transformation of flow data (individual data points that make up the dot plots) as one could do in say, Excel, and then allow conversion back into listmode or other format that can be imported back into flow analysis software so that regions so identified may be gated? Please kindly email Mannie at mannief@hotmail.com if you have any suggestsions at all -- thanks!

[Maciej Simm]

Mannie, you can take the vector graphs out of FlowJo and move each dot around with your mouse after ungrouping. Just kidding, we don't really recommend you do that.

You can mathematically transform your plots, sure. We use the Derived Parameters platform for that. Once you derive a parameter you can then export your current tube into a new FCS or CSV listmode file - this will include your newly derived parameter.

You can do it the other way too - take the FCS file from flowjo, export it into excel as CSV (limitation of 65K rows/columns in excel!) do your transformation in excel, then use my method to make a FJTEXTDATA FCS file. These will only work with FlowJo, though. However, once in FlowJo, you can export the file into FCS binary file, which should work in other programs too.

[Noah]

Have you written (or do you know of) a general script that will make an appropriate FCS header for a synthetic data file with an arbitrary number of parameters?
- Thanks!

[maciej simm]

Hi Noah,

no, it's very high on my priority list as a flowjo developer but we tend to prioritize by customer's feedback first, Maciej-says second.

I know that some of our non-flow users have developed their own proprietary scripts to parse stuff like mass-spec data into flowjo.

One trick that exists for FJ-Mac users is the FJTEXTDATA one -http://flowjo.typepad.com/the_daily_dongle/2006/06/how_to_roll_you.html - it's much easier to write text files than binary ones.