The Daily Dongle

New Class: "FlowJo for Integrators" @ 2011Northwest Regional Cytometry Meeting

No single analysis solution is performant enough to handle all needs.  For many institutions, improving flow cytometry analysis comes from integrating with databases and ever more hardware.
As a contribution to the Northwest Regional meeting, Tree Star is offering a "FlowJo for Integrators" practicum.  Spend a day with FlowJo architects Jay Almarode and Adam Treister to get the straight story on what's inside the software.
This one day workshop covers new or unpublicized features of FlowJo intended for advanced users who wish to:

- work with remote file repositories
- deploy FlowJo in a client/server, grid or cloud architecture
- execute FlowJo templates using command line scripts
- understand the various settings for optimizing the FlowJo calculation engine caches
- integrate results of other applications into FlowJo, or FlowJo results into other applications
- utilize FlowDx capabilities to manage calculate comparative gating statistics

Paid preregistration of $200 is required per institution with a limit of 8.  Up to 3 people per institution.
All proceeds will go to the Friday Night pub crawl.  Send an email to learning@treestar.com to sign up.

For more on NRCM, go here.

 

Cheers,

Adam Treister
President, FlowJo LLC

Keystroke Shortcuts in FlowJo

Find 'em here: http://www.flowjo.com/home/shortcuts.html

FlowJo Java, threaded

Got to spend some quality time with my testing 16 core today: 

FJ hitting all 16 cores, memory utilization ~8GB = instant gate recalcs :)

 

while at it, we made some performance improvements: 

This is the first time I'm showing a timing from FJ v10 (Java.) This version is currently in Alpha but we hope to have public beta soonish (Spring?)

Please note that this drastic change in performance in 7.6.4 and V10 will only be observed on:

- computers with more than 8 CPU's. I saw no change with 2/4 CPU models.  
- .. "distributed engines" setting in performance preferences turned ON 
- .. storage device fast enough to saturate all your CPU's with data 

the main reason for this improvement was the architecture of FJ's distributed engines - they ran in series, now - parallel. Also, the improved timings show FJ as running in 64 bit mode. I am not yet sure if 7.6.4 will ship in 64 bit compatible mode, but 10 definitely will.  64bitness is NOT required for this improvement to work. 

Oceans of Flow

Daniel Vaulot's work over the last decade has been instrumental in application of flow cytometry to oceanography. In particular, flow is well suited for the study of the smallest size class of the plankton (below 2 µm), called picoplankton, which is  composed of heterotrophic prokaryotes, prochlorophytes (Prochlorococcus), cyanobacteria (Synechococcus) and eukaryotes. The light-scattering parameters and fluorescence of natural photosynthetic pigments (chlorophyll, phycoerythrin) allows the identification of different groups that differ in terms of size and pigment content. Fluorescent microspheres (beads) are normally added as internal reference.

The small picoplankton requires lowering of instrument threshold levels in order to detect. One consequence of this is the acquisition of background "noise", as observed in the Figure below. In this plot, the large round population at the origin is not a picoplankton subpopulation, but rather a collection of background noise.  One of the main problems of collected noise is that it contributes to the file size. In this example, the >99% of noise equates to a file size of more than 100MB! Working with 100MB files are not fun. What can FlowJo do to make life easier?

 

Create a gate to include the populations of interest and to exclude most of the noise; as seen in Figure below. Alternatively you can use a "Not Gate". I have chosen to still keep some of the noise for referencing purposes.

 

Select the gate that you just created for one of the samples in your Workspace. For example, in the above Figure, the gate selected would be the "Export Gate". Now go to the main Workspace Tab and select the Export/Concatenate option. As per the Figure below, you will get a dialog box. Here you have to select Output Format as FSC 3.0 and use the "Select.... " option to choose a file name and file saving destination. This step generates a new FCS 3.0 file without a the huge chunk of noise.  This new file can be introduced into a FlowJo Workspace like any other FCS file, as seen in Figure 3. The new files are only about 700kb!

 

Once you have created the gates for populations of interest; in this case of Euk, Pro, Syn and Beads (see below), these can be overlayed in the Layout Editor.

 

In fact the Legend created (which appears as you overlay) allows you to remove the noise altogether by commanding the noise layer to be deleted (menu option appears when you mouse-right click on the Legend). Just remember that frequency statistics are based on the total events available on the plot, as such the percentage values for Euk, Pro, Syn and Beads will change based on inclusion or exclusion of noise events.

 

References:

Simon, N., Barlow, R. G., Marie, D., Partensky, F., Vaulot, D. (1994). Flow cytometry analysis of oceanic photosynthetic picoeucaryotes. J. Phycol. 30: 922-935

Vaulot, D., Marie, D., Olson, R. J., Chisholm, S. W. (1995). Growth of Prochlorococcus, a photosynthetic prokaryote, in the equatorial Pacific Ocean. Science 268: 1480-1482

Vaulot, D., Partensky, F. (1992). Cell cycle distributions of prochlorophytes in the North Western Mediterranean Sea. Deep Sea Res. 39: 727-742

FJML keyword examples

I've compiled this as a primer to FJML keywords for those who prefer to learn by example. To use these examples, first open your Mac FlowJo to a new workspace with some data, then click Workspace menu, Add Keyword. Once you have created this new keyword, you can double-click its cell in the workspace window, just like in Excel, and enter formulas. Here are some to try (using LMD data made by CXP software.) 

This syntax..	..renders as:	notes
FJML=<>	<>	anything to the right of equals that is not valid FJML is rendered as text.
FJML=<key $SYS>	CXP	assuming $SYS = CXP
FJML=File Made By: <key $SYS>	File Made By: CXP	assuming $SYS = CXP
FJML=<stat /gate1:Freq. of parent>	80.1	assuming there is a gate called “gate1” with a stat 80.1 % :)
FJML=<key $P2B> * 2	16 * 2	assuming Parameter2 exists and its B=16. Notice the failed attempt to invoke a math formula gets parsed as a text string.
FJML={{<key $P2B> * 2}}	32	the presence of {{ }} brackets around the expression forces FJ to evaluate the contents as a formula.
FJML={{sub(<key $CYT>, 1, 3)}}	Interpreter Error.	assumes $CYT = Cytomics FC 500. Fails to parse as shown, see below for corrected example.
FJML={{sub(“<key $CYT>”, 1, 3)}}	Cyt	notice the quotes around the keyword!

finally here's a harder one: convert BTIM/ETIM keywords' values (HH:MM:SS) into number of seconds the acquisition took. Since this expression is longer, I'll do it outside the table, split into the components of extracting the hours, days and seconds separately. This code does not take into account the clock resetting/starting over every 24 hrs ;) 

to calculate hours in a day and convert to seconds:

3600* abs(num(sub("<key $BTIM>", 1, 2)) - num(sub("<key $eTIM>", 1, 2)))

The "abs" stands for absolute value, and negates any negative sign. 

"num" converts a string to a number. This is in case we get something like "06" for HH. 

"sub" stands for substring, with arguments of starting/stopping character index. 

Next, minutes -> seconds: 

60* abs(num(sub("<key $BTIM>", 4, 5)) - num(sub("<key $eTIM>", 4, 5)))

and finally, seconds (no conversion :) ): 

abs(num(sub("<key $BTIM>", 7, 8)) - num(sub("<key $eTIM>", 7, 8)))

So the combined expression to paste into the keyword, is:

FJML={{  

 3600* abs(num(sub("<key $BTIM>", 1, 2)) - num(sub("<key $eTIM>", 1, 2))) +

60* abs(num(sub("<key $BTIM>", 4, 5)) - num(sub("<key $eTIM>", 4, 5))) +

abs(num(sub("<key $BTIM>", 7, 8)) - num(sub("<key $eTIM>", 7, 8)))   

 }} 

About the Daily Dongle

Welcome to the Daily Dongle. This blog started in June 2007 to provide content in three main areas: FJ support tutorials, insiders' perspectives and finally news/humor. 

We try to categorize posts on this blog to help you find them later. You will see categories tagged to most posts, so that if you later want to see other posts in those categories, just click on that category to get the full listing. For example, here is the search of all posts tagged as "tips and tricks". You can also browse old contents by date, or use the search field to jump to relevant posts. Please note the new author tags - they will help you find all posts by Aaron (France), John (Philly), Sach(Oz), Don Gull (Orygun) and other contributors. 

Comments posted on the DD do not get automatically sent to us - for urgent comments/questions, rather than clicking the "comment" link at the bottom of each post, email us - either the author directly, or the general support email flowjo@treestar.com. 

Thanks for visiting, and enjoy the site! On behalf of the  support & development crews,

Maciej 'm-diddy' Simm
(simm at treestar dot com )

FlowJo and Lab Archives

you might have noticed that starting with Friday's release of 7.6.3 there is a new field in Preferences' License tab: 

Should you click it and enter your Lab Archives credentials & restart FJ...

..you would add the option "Add Remote Samples" to your workspace menu. If you click on that menu item,  you will get a window to browse your Lab Archives root: 

anyone with Lab Archives accounts - please let us know how you find this new feature. Anyone without a Lab Archives acct, you can contact them about it. 

maciej simm

 

Why do My Platform Stats Differ from my Workspace Stats?

Users of the Proliferation and Cell Cycle Platforms have undoubtedly noticed that the statistics the model produces do not transfer exactly to the workspace when Gates are created from the model. 

For example, the figure below shows a Watson-Pragmatic cell cycle model for one of our demo files.  The associated statistics are noted on the model.  The stats pertinent to this discussion are the G₀/G₁, S, G₂/M frequencies which in the model are roughly 57% / 24% / 18%.

If you were to press the Create Gates button at the top of the model window, and accept the “Stage” prefix you would get the workspace shown below the figure. Notice that the frequencies are roughly 64% / 14% / 21%.  Why have 7% of the cells “shifted” from the S phase into the G₀/G₁ and another 3% shifted from S into the G₂/M?

The statistics are actually calculated in two different ways.  When using the models the frequencies are calculated using probabilities of membership, based on the model fit.  For cells with intensity measured within a given range, the model can be used to determine the prob

ability of the cell belonging to each of the identified populations plus sub-G₀/G₁ and super G₂/M.  The probabilities of all cells are then added up to computethe phase probabilities for the entire sample.  The summation of these probabilities will be another probability.  To create fractions that add up to ~100% a normalization step is required.  To preserve the relative relationship between phases we do not simply normalize to 100% but rather map the probabilities to those that sum to 100%.  This process can sometimes lead to small negative numbers, particularly in the sub-G0/G1 or super G2/M populations.  In these cases it is reasonable to think of a population with a negative frequency as simply a very small number of cells. 

When you press the Create Gates button FlowJo places hard line gates on the model at the points where it becomes less probable that a cell belongs to one population and more probable that it belongs to another.  Cells that are in the gate are treated as having a probability of membership of 100% for the “winning” phase and 0% for all other. 

You’ll notice in the above example that the difference in the statistics is that the gated frequencies do not take into account that there are probably a decent number of S phase cells that spill into the other two distributions, so if you have the opportunity to use the model statistics, do so.  The stochastic approach models the data better.  However, if you use the model created subpopulations, continue to add layers of gates, and in the end need to come up with fractions that add up to 100%, you will need to use the gate frequencies. 

- Bad Mo'Flow

Send comments and questions to john@treestar.com

 

 

FJRealData

In addition to fjTextData, Mac version of FlowJo has another port for opening text files, called fjRealData. the primary difference between the two is header construction syntax and the fact that "real" format uses doubles, while 'text' uses integers. Data formatted in this way can be read directly into FlowJo V9 for analysis and/or exported as FCS files for future uses

please note - FJ PC users should instead convert their CSV files into FCS files using text-to-FCS utility provided on this blog a few posts back. 

 

 

Format of this data file:

Line1 = "FJREALDATA"           //(allows floating point valus)
M <tab> N                              
Key1 <tab> value1
...
KeyM <tab> valueM
P1Value1 <tab> P2Value1 <tab> ... PjValue1 <tab> Parm#Key1Name
P1Value2 <tab> P2Value2 <tab> ... PjValue2 <tab> Parm#Key2Name
...
P1ValueN <tab> P2ValueN <tab> ... PjValueN <tab> Parm#KeyNName
{values...}

M: Number of keyword name / value pairs specified one at a time

N: Number of parameter keywords that are specified in a matrix format

Key1, Value1:  The name and value for the first of M keywords specified one at a time (Use this for things like $FIL or other attributes you might want to display in the workspace, like custom keyword values, e.g., treatment, time, etc.)

PjValuek:  The value of parameter keyword number "k" for parameter number "j"

Parm#KeykName: The name of parameter keyword number "k", using # to specify where the number is to be inserted

possible values for P#DISPLAY:

LINEAR  ... linear axis
MANUAL = (x, y)   .... log axis

x = number of decades
y = offset of lowest displayed value on log scale

Look at the file for an example.  Download 4Channel

 

FCS Files Care About Their Privacy Too!

Data privacy, being an obvious concern to some of our users, got us thinking about how we can give users the ability to share data without compromising confidential information. What grew out of that is a utility that Seth, one of our engineers, developed and we've been using in house for a little while. Another utility for creating FCS files has also been helpful. Both are free :)

-The de-identification utility allows you to quickly remove, edit or replace keywords that contain patient, proprietary or otherwise top-secret information. It writes an entirely new set of FCS files, i.e. your originals stay untouched, so now you can visualize, analyze or share data without compromising confidentiality.

-The text-to-FCS utility allows you to convert CSV (comma separated value) files into FCS files. "Synthetic" FSC files have been useful for data-mining, visualization, and batching operations in FlowJo, and we see the bridge for "generic data" and FlowJo's user interface as a great way to expand FlowJo's capabilities. 

To grab the utilities, go here. Enjoy :)