The Daily Dongle

FlowJo Java, threaded

Got to spend some quality time with my testing 16 core today: 

FJ hitting all 16 cores, memory utilization ~8GB = instant gate recalcs :)

 

while at it, we made some performance improvements: 

This is the first time I'm showing a timing from FJ v10 (Java.) This version is currently in Alpha but we hope to have public beta soonish (Spring?)

Please note that this drastic change in performance in 7.6.4 and V10 will only be observed on:

- computers with more than 8 CPU's. I saw no change with 2/4 CPU models.  
- .. "distributed engines" setting in performance preferences turned ON 
- .. storage device fast enough to saturate all your CPU's with data 

the main reason for this improvement was the architecture of FJ's distributed engines - they ran in series, now - parallel. Also, the improved timings show FJ as running in 64 bit mode. I am not yet sure if 7.6.4 will ship in 64 bit compatible mode, but 10 definitely will.  64bitness is NOT required for this improvement to work. 

FlowJo and Lab Archives

you might have noticed that starting with Friday's release of 7.6.3 there is a new field in Preferences' License tab: 

Should you click it and enter your Lab Archives credentials & restart FJ...

..you would add the option "Add Remote Samples" to your workspace menu. If you click on that menu item,  you will get a window to browse your Lab Archives root: 

anyone with Lab Archives accounts - please let us know how you find this new feature. Anyone without a Lab Archives acct, you can contact them about it. 

maciej simm

 

FlowJo 7 and CSV export

This question just came up on PUCL, so I probably should mention the few ways of handling CSV data in FlowJo 7. This pretty much applies for the version 9 as well.

 

The simplest use case is you have a bunch of FCS files and want to convert all of them into CSV files:

- load files into FJ workspace
- select all the files (ctrl+a)
- click worskpace menu, Export/Concat
- switch type to CSV, and let it rip - this will generate a folderfull of CSV's at your specified destination.

 

Another common one is "I only want a specific gate exported". this is also batch-able in FJ - assuming your gate is group-owned by many samples, you can repeat the above steps with slight variation - instead of selecting all the root nodes (CTRL+A), click once on the gate you want to export, then click Edit -> Select Equivalent Nodes. Then click workspace->export/concat as above.

 

I should also mention the opposite direction, going from some kind of CSV file into FSC so it can be read into FlowJo is also possible. For this, we have a free stand-alone java (all platforms) conversion utility I wrote with Seth and Jay posted a few entries back .

minor changes to Table Editor in next 7.x beta

I'm working on some added functionality to CSV saving/batching from the table editor.  

 

the panel above is found in Edit -> Preferences -> Tables. 

The first one is self explanatory.. if unchecked, you won't get the "mean/average" rows added to your csv output. 

The second one - normally if you batch your table output to "c:\table.csv" and the file already exists, flowjo will create an output file called "table-1.csv" in that same path. 

If the "append" preference is checked, FlowJo will instead append the current table to the existing file's contents (adds after current content, ie at the bottom of the existing file. )  

Anything else here while I'm at it? 

vertex editing rant

semi-tutorial, semi-rant - this is to address removal of vertices from a polygon gate, and resulting performance differences, in FlowJo mac/java. Spoiler alert: fewer vertices = faster. 

Download Vertex tutorial [MOV, 6.4MB]

Autobatching & Command Line tutorial

Download autobatch tutorial [MOV, 18.2MB]

Summary: 

Auto-Batching is a feature which can be utilized when running flowjo in Command-Line Mode (CLI). CLI was developed to give the set of features necessitated by large volumes of data set that may need automated pre-processing prior to 'attended' analysis - pre-gating, compensation calculations,  exporting gates to smaller FCS files, are some examples of tasks that fall into that category. 

AutoBatch is best initiated from an existing template which has things like: 

- dynamic live groups with group-owned-gates
- tables defined in the Table Editor
- layouts defined in the Layout Editor

AutoBatch in Table Editor: 

- open an existing Table in your Table Editor. Click Edit menu >> Include in Batches. 
- define an output for your table that gets written to disk. Click Batch button, then set the option for "File", pick a flavor & output destination, and hit "Save" and "Apply" buttons. This commits the autobatch settings to this particular table.
- notice the tables which were autobatch enabled, are listed in BOLD typeface.

AutoBatch in Layout Editor: 

- open an existing layout, click Layout menu >> Include in Batches. 
- Click Batch button and define some sort of File output (PDF, PPT etc.) and destination. Remember to hit "Save" and "Apply". 
- notice autobatched layouts are listed in BOLD typeface. 

AutoBatching file export/concatenation:

- this is initiated slightly differently from the above two, since there is no "batch" button. 
- to autobatch the export or concatenation of UNGATED FCS files to csv/fcs data on disk, right-click the group definition at the top level, click Export/Concat. In the Export/Concat dialog select your destination (currently ignored, defaults to template location) and file flavor, as well as whether or not to concatenate the data. Lastly, check the "save in template" checkbox - this commits your actions to the template's memory. 
- to autobatch the export or concatenation of GATED data (ex. "merge all my T cell gates into one CSV/FCS file"), right-click the group definition at that gate's level, then click Export/Concat. After this point the directions match the "ungated" export/concat. 

Executing "AutoBatch" templates: 

this requires either access to the command prompt or ability to generate and execute .bat (batch) files. 
the command syntax follows: 

c:\path\flowjo.exe c:\path\template.wspt c:\path\experiment1 -save -quit

where the flowjo exectuable (path + exe name)  is called with four arguments: the input template (path + document name), the input data (path), and two optional arguments to save the executed work into a workspace file. The "-quit" argument is optional but recommended to free up any runaway RAM consumed by each batch :) 

performance issues

FlowJo V7 (Java) performance is an ongoing issue in our support and PUCL. This post is a Cliffnotes of existing DD posts on the subject. Following these steps usually resolves 3/4 of our users' gripes on performance, while the remaining quarter use gigantic (hundreds MB) files and deeply nested (10+ levels) gating trees, at which admittedly the pure-Mac version scales better per core.

1. FlowJo installers ship with conservative defaults. You may wish to change these settings: 

• Max Heap Size in Flowjo.lax file (link)
• sample cache size in flowjo prefs: Edit -> Preferences -> Performance. Default value is 5, try 25 and 50. I recommend against increasing this if your individual FCS file sizes exceed 100MB, it may cause "out of memory" exceptions. 

 

2. FlowJo performance can be scaled on computers with 4+ processors and 8+ GB RAM. 

• open FlowJo, click Edit menu -> Performance, and enable "engine dispatcher". This spawns additional computational processes. I recommend using N+1, where N is your number of cores. 

 

3. Network/CD/DVD/zipdrive data access = slower performance. Always analyze files on Local Volumes. Exception to this are low latency high throughput external hard drives (eSATA.) 

Also in response on someone's request on an update on "news" in FJ Java performance. We've been optimizing the FlowJo 10 prototype engine quite a bit, both for "single" workstations and "clouds". I'll post specifics (numbers) when I get around to doing some thorough benchmarks, but for now 7.x release is my primary focus.  

converting keyword $Vol into absolute cell counts in gates (Accuri C6 data)

This is a tutorial which revolves around the keyword $Vol found in Accuri data. Someone asked, how they can use it in FlowJo to convert gate stats into absolute counts of cells.

The background on this is that $Vol is a keyword found in some Accuri data which contains the volume of the sample acquired in that file, in the units of nL. However, the accuracy handling of the C6 is to the uL, so you may want to adjust your accuracy expectations by three places :P

The best and easiest way to get cell concentration from a gate is in the table editor. The "starting blocks" for this are:

- $Vol (total volume acquired in sample)
- $Tot (total amount of events acquired)
- your gate frequency (relative to $Tot, not parent)

And the formula for this would be..

$Tot (cells) / $Vol (nL) * 1000 (nL/uL) * freq  = cells / uL 

or as it would appear in table editor..

<Column "$TOT">/<Column "$VOL"> * 1000 * <Column "Freq. of my gate">

To implement this in FJ's table editor, drag the keywords $Vol, $TOT in first & label them (give them custom column names). Then drag in your gate to get the frequency of it.

Now when batched, you'll have a new column in your table that calculates the cells/uL values for all tubes :)

couple of bugs/workarounds for 7.6.1's DNA platform

There are two things I've noticed we're doing wrong in the DNA platform, but the good news is there are workarounds (TM) :D

First - if you have offscale events on your DNA parameter, they will be included when the Y axis scale is calculated. Unlike the graph window, the DNA platform does not allow Y axis scale adjustment, so you're stuck with this:

 

The workaround that usually works best is actually part of the due process- if you do exclusion gating on width/height of the DNA signal, this shouldn't be a problem. However, if that is not an option - make a small gate to exclude the last few bins of data, in the above example, that would be the events on max bin of FL2-A parameter, invert it (uncheck "include events inside gate")  and then do your fits on that.

Second: the DNA platform may not update the contents of the plot, nor the fit, if you change the axis. Workaround: close/reopen the DNA window - that forces an update.

Missing Menus in Flowjo 7.6 and 7.6.1.may be related to PC Appearance/Theme Setting.

A few people have experienced missing menus in Graph window, Cell Cycle, and Kinetics on PCs running Vista, XP, and Win7.   This behavior does not occur on all computers and seems to be related to the Appearance/Theme setting of your computer.

We are working to investigate the cause and fix it in a future version of FlowJo. Switching back to the default Theme through the Control Panel will correct this behavior in most instances.  

Graph Window Statistics Menu

Cell Cycle Models Menu

Kinetics Options Menu