The Daily Dongle

Clone Gates

Please note: this functionality requires FlowJo for Mac 9.3b5 or later.

Introduction

The use case for this feature came from our clinically inclined users who wanted one version of a gate propagated to different points in a hierarchy. This used to be done by one of two ways, neither of which is optimal in terms of workflow:

FJML keyword examples

I've compiled this as a primer to FJML keywords for those who prefer to learn by example. To use these examples, first open your Mac FlowJo to a new workspace with some data, then click Workspace menu, Add Keyword. Once you have created this new keyword, you can double-click its cell in the workspace window, just like in Excel, and enter formulas. Here are some to try (using LMD data made by CXP software.) 

This syntax..	..renders as:	notes
FJML=<>	<>	anything to the right of equals that is not valid FJML is rendered as text.
FJML=<key $SYS>	CXP	assuming $SYS = CXP
FJML=File Made By: <key $SYS>	File Made By: CXP	assuming $SYS = CXP
FJML=<stat /gate1:Freq. of parent>	80.1	assuming there is a gate called “gate1” with a stat 80.1 % :)
FJML=<key $P2B> * 2	16 * 2	assuming Parameter2 exists and its B=16. Notice the failed attempt to invoke a math formula gets parsed as a text string.
FJML={{<key $P2B> * 2}}	32	the presence of {{ }} brackets around the expression forces FJ to evaluate the contents as a formula.
FJML={{sub(<key $CYT>, 1, 3)}}	Interpreter Error.	assumes $CYT = Cytomics FC 500. Fails to parse as shown, see below for corrected example.
FJML={{sub(“<key $CYT>”, 1, 3)}}	Cyt	notice the quotes around the keyword!

finally here's a harder one: convert BTIM/ETIM keywords' values (HH:MM:SS) into number of seconds the acquisition took. Since this expression is longer, I'll do it outside the table, split into the components of extracting the hours, days and seconds separately. This code does not take into account the clock resetting/starting over every 24 hrs ;) 

to calculate hours in a day and convert to seconds:

3600* abs(num(sub("<key $BTIM>", 1, 2)) - num(sub("<key $eTIM>", 1, 2)))

The "abs" stands for absolute value, and negates any negative sign. 

"num" converts a string to a number. This is in case we get something like "06" for HH. 

"sub" stands for substring, with arguments of starting/stopping character index. 

Next, minutes -> seconds: 

60* abs(num(sub("<key $BTIM>", 4, 5)) - num(sub("<key $eTIM>", 4, 5)))

and finally, seconds (no conversion :) ): 

abs(num(sub("<key $BTIM>", 7, 8)) - num(sub("<key $eTIM>", 7, 8)))

So the combined expression to paste into the keyword, is:

FJML={{  

 3600* abs(num(sub("<key $BTIM>", 1, 2)) - num(sub("<key $eTIM>", 1, 2))) +

60* abs(num(sub("<key $BTIM>", 4, 5)) - num(sub("<key $eTIM>", 4, 5))) +

abs(num(sub("<key $BTIM>", 7, 8)) - num(sub("<key $eTIM>", 7, 8)))   

 }} 

FJRealData

In addition to fjTextData, Mac version of FlowJo has another port for opening text files, called fjRealData. the primary difference between the two is header construction syntax and the fact that "real" format uses doubles, while 'text' uses integers. Data formatted in this way can be read directly into FlowJo V9 for analysis and/or exported as FCS files for future uses

please note - FJ PC users should instead convert their CSV files into FCS files using text-to-FCS utility provided on this blog a few posts back. 

 

 

Format of this data file:

Line1 = "FJREALDATA"           //(allows floating point valus)
M <tab> N                              
Key1 <tab> value1
...
KeyM <tab> valueM
P1Value1 <tab> P2Value1 <tab> ... PjValue1 <tab> Parm#Key1Name
P1Value2 <tab> P2Value2 <tab> ... PjValue2 <tab> Parm#Key2Name
...
P1ValueN <tab> P2ValueN <tab> ... PjValueN <tab> Parm#KeyNName
{values...}

M: Number of keyword name / value pairs specified one at a time

N: Number of parameter keywords that are specified in a matrix format

Key1, Value1:  The name and value for the first of M keywords specified one at a time (Use this for things like $FIL or other attributes you might want to display in the workspace, like custom keyword values, e.g., treatment, time, etc.)

PjValuek:  The value of parameter keyword number "k" for parameter number "j"

Parm#KeykName: The name of parameter keyword number "k", using # to specify where the number is to be inserted

possible values for P#DISPLAY:

LINEAR  ... linear axis
MANUAL = (x, y)   .... log axis

x = number of decades
y = offset of lowest displayed value on log scale

Look at the file for an example.  Download 4Channel

 

Quad Gates Using FJML

This is an interesting application of FJLM/Vertex gating in FlowJo V9.2 - driving the center position of a quad gate based on another gate's position or summary stat. 

Let's say you like using quad gates - and you want to tie the center position to more less where the negative cells end on x/y axes. Here's how I assembled that. 

vertex editing rant

semi-tutorial, semi-rant - this is to address removal of vertices from a polygon gate, and resulting performance differences, in FlowJo mac/java. Spoiler alert: fewer vertices = faster. 

Download Vertex tutorial [MOV, 6.4MB]

Autobatching & Command Line tutorial

Download autobatch tutorial [MOV, 18.2MB]

Summary: 

Auto-Batching is a feature which can be utilized when running flowjo in Command-Line Mode (CLI). CLI was developed to give the set of features necessitated by large volumes of data set that may need automated pre-processing prior to 'attended' analysis - pre-gating, compensation calculations,  exporting gates to smaller FCS files, are some examples of tasks that fall into that category. 

AutoBatch is best initiated from an existing template which has things like: 

- dynamic live groups with group-owned-gates
- tables defined in the Table Editor
- layouts defined in the Layout Editor

AutoBatch in Table Editor: 

- open an existing Table in your Table Editor. Click Edit menu >> Include in Batches. 
- define an output for your table that gets written to disk. Click Batch button, then set the option for "File", pick a flavor & output destination, and hit "Save" and "Apply" buttons. This commits the autobatch settings to this particular table.
- notice the tables which were autobatch enabled, are listed in BOLD typeface.

AutoBatch in Layout Editor: 

- open an existing layout, click Layout menu >> Include in Batches. 
- Click Batch button and define some sort of File output (PDF, PPT etc.) and destination. Remember to hit "Save" and "Apply". 
- notice autobatched layouts are listed in BOLD typeface. 

AutoBatching file export/concatenation:

- this is initiated slightly differently from the above two, since there is no "batch" button. 
- to autobatch the export or concatenation of UNGATED FCS files to csv/fcs data on disk, right-click the group definition at the top level, click Export/Concat. In the Export/Concat dialog select your destination (currently ignored, defaults to template location) and file flavor, as well as whether or not to concatenate the data. Lastly, check the "save in template" checkbox - this commits your actions to the template's memory. 
- to autobatch the export or concatenation of GATED data (ex. "merge all my T cell gates into one CSV/FCS file"), right-click the group definition at that gate's level, then click Export/Concat. After this point the directions match the "ungated" export/concat. 

Executing "AutoBatch" templates: 

this requires either access to the command prompt or ability to generate and execute .bat (batch) files. 
the command syntax follows: 

c:\path\flowjo.exe c:\path\template.wspt c:\path\experiment1 -save -quit

where the flowjo exectuable (path + exe name)  is called with four arguments: the input template (path + document name), the input data (path), and two optional arguments to save the executed work into a workspace file. The "-quit" argument is optional but recommended to free up any runaway RAM consumed by each batch :) 

recovering from broken layouts

If you are trying to open a FlowJo for Mac workspace and experience the "size == lirSize" error in "FLayoutStream.cp": 

then the file you are trying to open did not save properly and part of the layout editor information was truncated. There is a way to recover partial information from such workspaces with use of the "recovery" command: click File menu in FlowJo, hold down "alt" key so that File-> Open becomes File->Recover, then point it to your document. 

If this is a problem you experience more than once, there is likely some indication of a disk problem or a permissions issue, so you may wish to exercise more caution when handling documents into which hours of work were invested. 

One such strategy is to manually version your workspaces (file-> save as -> give it a new version in file name). THere is also a preference to do this automatically (Workspace tab, Save area) but I prefer manual :)

In case one of your documents becomes corrupted in this way, you can "recover" it, and open an older version of the same document. Then open the layout editor windows from both documents side by side, and drag the layouts from the list of layouts of the "backup" to your "recovered" workspace. 

  

green dot by file name?

Someone recently asked what the green dot represents, as seen in the Workspace window, adjacent to the sample names. My answer is usually "if you open it in a graph window, you get the green dot", however recently one of our senior engineers corrected me,   - 

it's an indicator of which data files currently reside in memory.  When FJ dumps the file out of memory the dot goes away.  In large scale batch analyses, it kind of indicates where FJ is in computation.  It was more of a debugging of performance issue than any thing beneficial for the user.  It's been there for .... 15 years....

so there you go. :)

new FJ-mac beta 9.1b12

addresses several format issues for high-range and non-power-of-two range data.

http://flowjo.com/v9/html/macbeta.html

(may not yet have been updated with the latest info, but the zip it points to is b12.)

converting keyword $Vol into absolute cell counts in gates (Accuri C6 data)

This is a tutorial which revolves around the keyword $Vol found in Accuri data. Someone asked, how they can use it in FlowJo to convert gate stats into absolute counts of cells.

The background on this is that $Vol is a keyword found in some Accuri data which contains the volume of the sample acquired in that file, in the units of nL. However, the accuracy handling of the C6 is to the uL, so you may want to adjust your accuracy expectations by three places :P

The best and easiest way to get cell concentration from a gate is in the table editor. The "starting blocks" for this are:

- $Vol (total volume acquired in sample)
- $Tot (total amount of events acquired)
- your gate frequency (relative to $Tot, not parent)

And the formula for this would be..

$Tot (cells) / $Vol (nL) * 1000 (nL/uL) * freq  = cells / uL 

or as it would appear in table editor..

<Column "$TOT">/<Column "$VOL"> * 1000 * <Column "Freq. of my gate">

To implement this in FJ's table editor, drag the keywords $Vol, $TOT in first & label them (give them custom column names). Then drag in your gate to get the frequency of it.

Now when batched, you'll have a new column in your table that calculates the cells/uL values for all tubes :)