The Daily Dongle

Tom writes...

http://www.flowjo.com/home/videotutorials.html - we've posted some more flowjo training videos.

7.5 beta 37

It's not finished, but it's testable. First, some disclaimers:

• Note: this beta is intended to gather early feedback about some changes we've made, aimed at flowjo enthusiasts with idle cycles to spare. This version of flowjo will use a different preferences file (flowjo75.prefs, rather than flowjo7.prefs) to avoid conflicts with existing settings for version 7.2.x. This installer comes with a private java 1.6 VM.

• Supported Operating Systems: Win XP, Vista, 32 bit editions, ONLY (Win2K support was dropped with 7.2.5). If you cannot run this version on winXP, install .net 2.0 from here first.

How to get it:

1. whether or not you already own version 7.2.x license, we really would like you to use our beta-site license for this testing. You could use your existing 7.x license, but we're hoping to fine-tune the site license processing and that's the only way to do it. Email me your MAC address (AKA hardware address, ethernet ID, etc.) and I'll send you back the license. I will be collecting usage stats as with normal site licenses (only how much time you spent using FJ, which is needed by site license admins to do billing). No information collected in this process will ever be shared, with anyone outside treestar. Internet connection will be required to run this beta in site license mode

2. Get the program here - FlowJo_7.5.exe (current build from July 15th)

New Goodies which need Testing:

Correlation plots in table editor. To do this, generate a table with at least two columns. Right click on the column header of one, and choose the other column from the right click menu.

Preferences (I worked on the V5 style panel, click Edit/preferences to start. Known bug - text prefs no worky yet.)

Export/Concatenate (find this in Workspace menu. Works much better than 7.2.x.)

Proliferation/CFSE (now with 100% more fit!)

Derived Parameters

Graph Window T button menus (for lin/log scaling, and biexponential transformation)

Table Editor has a new heatmap option. (Documentation needed, but briefly - double-click on any column definitions in Editor view to open the dialog, and check the "heatmap" option there. )

Accuri files (partially fixed)

Actually, all data which was not working in 7.2 should be supported in 7.5. If not, email us some files.

Compensation Editor.

Multithreading is enabled in this version but kept off with a preference. If you're really curious about this i can send you more info. Short version - enabling it makes the program slower, but more memory-stable.

What not to test:

old workspaces may not load layouts correctly; things may be generally broken in layouts; boolean gates.

State of the Dongle Address

I got a good bunch of questions from a user, which prompted this post. The context here is analyzing a large amounts of 1M+ FCS3 files, ie. high-throughput analysis.

1) Do you think the PC version will sometimes catch up to the Mac version?

Yes. 7.2 is 80% there if you compare performance on single-CPU machines. Mac's big advantage is multi-threading, which we have recently implemented in a 7.5 beta version.

2) When do you plan to release PC version 8.x?

Odd numbers are used for PC, even - for mac. The next version for PC is 7.5, and after that, we start working on 9.0 & FlowDX (clinical FlowJo.)

In Version 7.5 we hope to bridge the gap by introducing support for multiple processors. This is not to say V7.5 will be "as fast" as V8, but it's going to be one step closer. Interestingly, the method we picked for scaling threads in V7 is superior to what we used on the Mac, and will support more than 4 processors as well as distributed processing across network sockets. But you, the user, won't see that difference kick in for at least several more months, since it needs fine-tuning and optimization.

3) What kind of PC hardware do you recommend for smooth operation of FlowJo?

at least 3GB RAM, for your data size. FlowJo will use up to 1.7GB, but that amount gets mirrored by the FlowJo engine's memory allocation. Since 32 bit windows doesn't use more than 3 - 3.5GB RAM, going higher won't change anything.

In therms of CPU, V7.2 series are single-threaded, so it does not matter if you have dual- or quad- core CPU, but V 7.5 will scale linearly with 2- or 4-core processors. I'm not sure what the scaling will be like with 8- or 16- threads, since disk access becomes a bottleneck.

In terms of data bandwidth, use SATA-2 based disks. Don't access big files over network.

4) Can FlowJo run with 64bit versions of Windows (XP/Vista)?

Yes, but it's not supported. That means, if you find bugs in 64 bit windows which I can't reproduce in my 32 bit development environment, I won't be able to fix them ;( But I occasionally test the software in winXP 64 bit and it seems to work OK for version beta 7.5.

Version 7.2x seems to have a problem with vista 64 bit in that it does not find the hardware address of the computer. It's not likely we'll have "official" 64 bit support in the V7 series.

5) Can FlowJo adress more than 4 GB RAM?

In 7.5, since each "thread" gets it's own PID, it will be able to allocate it's own 32bit-compatible amount of RAM (~1.7GB). So if you ran 7.5 on a 64 bit computer which has 8GB and 4 processors, with only 4 threads you should be hitting pretty close to 80% RAM utilization. This is hypothetical, I don't have a windows computer on which I can test this claim.

6) Is RAM more important or the processor's performance?

Yes. This will be true in future versions as well regardless of processor scaling. RAM is ~1000 faster than disk, on average. When computer gets low on RAM, it uses the disk to swap chunks of calculations that don't fit into . Feeding data from disk to the processor imposes a 1000 fold performance penalty. At that point, it doesn't matter how fast the processor is!

7) Do you think it's possible to reach comparable performances with the PC version at all, or should I go and get a Mac (in case, which one would you recommend?) regardless the non-existing support here?

over the next year, while we fine-tune the new multithreading in PC, you should use a Mac for the most demanding analysis.

Apple Care plan (~300 bucks) gets you full support from the source, but if your mac goes down you have to fedex it out - so it's usually down for a week. out of the 15 or so macs I use in testing, some of which have been online for 5+ years, I've only had to repair two, and they were both laptops.

For a new desktop, the best "value" is iMac. the best performance is the quad-core mac pro (the flagship model is 8-core, so the 4-core should be much cheaper by now!)

of interest.. cfr11 part 21

we're getting more and more questions about compliance to this. I intend to grow my own understanding of these regulations and will use this post (static link) to keep track of that.

Most interesting fact about this, which captures some of the definition of what it means to be "compliant":

Q: Can a vendor guarantee compliant software for Part 11?

A: It is not possible, nor correct, for any vendor to offer a turnkey 'part 11 compliant system', as this would involve both the vendor AND the user to comply with procedural controls.

more of such facts here - http://www.21cfrpart11.com/pages/faq/index.htm

also fun to read FDA's warning letters - http://www.fda.gov/foi/warning_letters/s6847c.htm

and wiki entry on this regulation which suggests it's getting a lot of challenges and resistance from the industry:

As of 2007, broad sections of the regulation have been challenged as excessive, and the FDA has stated in guidance that it will exercise enforcement discretion on many parts of the rule. This has led to confusion on exactly what is required, and the rule is being revised. In practice, the requirements on access controls are the only part routinely enforced. The "predicate rules" which required the records to be kept in the first place are still in effect. If electronic records are illegible, inaccessible, or corrupted the manufacturers are still subject to those requirements.

8.7.3 and recalculation frequency

So, you miss the old "immediate" recalculations from 8.7.1 version? From an email I sent to one of our Users earlier today asking about this -

we changed this behavior to address certain flowjo crashes. The most stability we've achieved in the program, is in this version, because of this change. The crashes caused by instant live recalc affected all speeds and models of macs, not just the older ones. If you haven't hit this, you're one of our more careful users who probably waits until "tasks" are finished before you queue up more gate movements, batches, etc.. ;)

To get the old behavior of recalc with a keystroke, click on All Samples and hit [cmd]+[=], or batch, or make a table, all values will be recalculated as needed. We cannot post version 8.7.1 and previous because of fairly major stability concerns on all Macintosh platforms (and various other bugfixes.)

well good morning to you too :)

I'm glad Ryan gave me a gigabit link to our email server. Or am I... ;)

 

3D plots teaser

quicktime, 14MB demo - clicky here

I'm working with Mr. Dominic Blais to bring JMonkeyEngine swiftness into the long sought-after 3D display platform for flow. I can't share the application with you as it's too raw, but here's an early peak at current performance on a 20,000 event DIVA file.

for your iphones

:) a couple of movies in the iphone format. Drag them to your itunes movies, then you can synch them from the iphone tab. 

strategic planning meeting 

make your iphone look busy with this batch

quick showoff of the compwizard

DiVa spike

We frequently get asked about the "spike" in digital FCS3 files. Most of the people who ask about this are DiVa users of various denominations (LSR's, Canto's, etc.)

A subset of these users wish to bring the data back into a legacy program like CellQuest Pro. To do this, the fancy 18 bit data has to first be converted into FCS2 format. However, this conversion is lossy. One of the pieces of info lost in the conversion is baseline subtraction. 

baseline subtraction: the instrument keeps track of background fluorescence as events go through, and the average fluorescence in between events is subtracted from each event. Events without fluorescence will sometimes be recorded while the surrounding background measurement is > 0, causing these events to be assigned a negative fluorescence value even before compensation correction. 

As an example, imagine a nice gaussian distribution from channel -100 to +100. If you throw away baseline subtraction, half of it will now be in the first available bin on the log scale, causing a "spike" like appearance.

(todo: explain why we see a spike in FCS3 format in FlowJo like the screenshot above.. is that width basis compression? )

how the new concatenation works

The goal of this exercise will be to combine 8 tubes of an antibody titration experiment into one file and then display it as Fluorescence vs. tube #. Our Basic Tutorial data set (google "flowjo basic tutorial") will be used.

First, I create some sort of gate delimiting my cells of interest. I don't want to concatenate all the events, just the interesting ones. I called my gate "Lymphocytes" and applied it to the group:

The next thing we will need is a keyword which we will fill with some values in order to identify the tubes. I'll be using "tube number" filled with integer series {1,2,...8}. Other ideas include dilution factors, or dilution concentration of your antibody, etc. - as long as the values are numerical.

Note: if you need a new keyword, click Workspace menu, new keyword. Otherwise use one of your pre-existing ones. You can access those via Workspace/edit columns.

So here are my tubes lined up with tube numbers in the keyword column:

Note: you can quickly make value series. Double-click in the cell which corresponds to the first editable keyword value (could be blank, or filled. ). Then option-click (hold down option) the workspace menu, and choose "create val series". The dialog is self explanatory and helps you create any sort of numerical series you might need for this task.

Next, click any of the "Lymphocytes" gates and click Workspace/Concatenate. Most of the options you'll see in the Concatenate dialog are self-explanatory. The ones to pay attention to:

- Create Additional Parameters - click Choose and pick the keyword which stores values you wish to make into a parameter in your concat output.

- Batch Concatenation - be sure to select the right option and number here! In my example, I picked "concatenate successive files together" and N=8.

Then let it rip.. it'll ask you where to put the output file. Make a new workspace, and drag in your concatenated file. You'll notice the parameter list will now include the tube number parameter, or whatever, so you can graph vs. that.

 

Note: in case of FCS3 data, flowjo will try to recognize if your keyword's values create a patter better seen in log and automatically adopt log scaling. In this example, we're using a series of numbers which are linear.

DD Reader Ben has sent in some of his own usage examples in titration: