The Daily Dongle

FlowJo and Windows 7

One of our testers, Adam, has been pounding away at the new windows 7 beta. We've found no problems with it so far- however- if you run flowjo in either of the "compatibility" modes (xp or vista) it will fail. So run it natively. 

Our testers found that on the same hardware, windows 7 is faster. I for one am looking forward to this upgrade just for that reason - if the hype is true, maybe my PC will finally boot under a minute, like my mac has been for the past 8 years or so. 

:)

Ratiometric parameter question from Purdue

You might have seen this post on Purdue CML.. I thought I'd paste my answer here for future ref. 

Just curious if anyone knows much about creating ratiometric parameters on FlowJo. I did look through the FlowJo manual and saw how to 'derive' a parameter, but it doesn't look like it can do what we want it to.

FlowJo 7.5's performance preferences

I've put together all the preferences which affect performance and put them in the "performance" tab of the preferences (which you can invoke from Edit menu in the Workspace window.)

Feel free to experiment and give me feedback. Be sure to first check out the documentation.

more on threading

well, I ran a 5.6GB dataset on 7.5 beta today, and with just simple gates, not very complex analyses, just to test engine robustness.

I compared the old threading (socket engine, multiple processes, multiple simultaneous IO's) vs the new one (single engine, single process, linear IO's) and found some interesting results.

The socket engine scales better for 4-8 processor machines, but relies heavily on disk IO and needs a server which can handle 3-4 concurrent requests each at sustained 30ish MB/s to saturate all cores.

The socket engine crashes 100% of the time right now, regardless of how many I'm running. We're debugging this right now.

The newer threading engine has a problem with saturating all 4 cores on my test computer, but it definitely uses more than 3. It does not crash, and handles the 5.6GB dataset without problems.

this is all available to you in 7.5 beta 68 (scroll down a few posts for the link.)

on the thread of threading

Threading has been a topic of interest for me. Personally I enjoy a lot of digital arts, which require heavy CPU utilization - so I am used to software which scales nearly linearly in performance with number of processors. So whenever I sit down to benchmark FlowJo, I scratch my head - what are we doing wrong?

the performance gains of going from 1 to 2 to 4 processors are disappointingly nonlinear, except for a very specific task of computing compensation values and transforming the events according to the biexponential function. Long story short, most of flowjo is not very threaded.

About a year ago, on the PC platform, we tested successfully the ability to delegate computation requests over a network to a socked-connected calculation engine running on the same, or different, computer. Each engine is it's own 32 bit system process, capable of allocating up to 1.7GB RAM. So in terms of high memory utilization on high-end servers, problem solved.

Meanwhile, if you take a 4 core PC side by side by a 4 core mac pro, running v8, you'll find that V8 smokes the java version like one of them wacky tobaccy cigarettes. Or more accurately, it used to, until today.

The current divide between V8 and shipping version 7.2.5 is pretty much this:

  on a 4 core mac pro
same mac pro via VMware

the test is 16 x 0.5M event files being read in, compensated, transformed, and gated. Lower is better.

however, our senior engineer Jay has figured out how to make our individual engine processes threaded themselves, meaning multiple CPU usage by a single engine process.

the middle green bar above indicates performance of yesterday's developmental version 7.5b63. Slightly faster than the shipping version (blue bar).

The top green bar for Vista is also VMware, which as my fellow hackers will know, suffers from some IO problems, and only supports 2CPU's.

So I executed 7.5 threaded version in OSX directly via Java 6 on a 4 core mac pro, and those results are shown as the last green bar. Either way, this is still a work in progress, but there is a good chance 7.5 will be able to allocate the most RAM and CPU's of any flow analysis package out there.        

State of the Dongle Address

I got a good bunch of questions from a user, which prompted this post. The context here is analyzing a large amounts of 1M+ FCS3 files, ie. high-throughput analysis.

1) Do you think the PC version will sometimes catch up to the Mac version?

Yes. 7.2 is 80% there if you compare performance on single-CPU machines. Mac's big advantage is multi-threading, which we have recently implemented in a 7.5 beta version.

2) When do you plan to release PC version 8.x?

Odd numbers are used for PC, even - for mac. The next version for PC is 7.5, and after that, we start working on 9.0 & FlowDX (clinical FlowJo.)

In Version 7.5 we hope to bridge the gap by introducing support for multiple processors. This is not to say V7.5 will be "as fast" as V8, but it's going to be one step closer. Interestingly, the method we picked for scaling threads in V7 is superior to what we used on the Mac, and will support more than 4 processors as well as distributed processing across network sockets. But you, the user, won't see that difference kick in for at least several more months, since it needs fine-tuning and optimization.

3) What kind of PC hardware do you recommend for smooth operation of FlowJo?

at least 3GB RAM, for your data size. FlowJo will use up to 1.7GB, but that amount gets mirrored by the FlowJo engine's memory allocation. Since 32 bit windows doesn't use more than 3 - 3.5GB RAM, going higher won't change anything.

In therms of CPU, V7.2 series are single-threaded, so it does not matter if you have dual- or quad- core CPU, but V 7.5 will scale linearly with 2- or 4-core processors. I'm not sure what the scaling will be like with 8- or 16- threads, since disk access becomes a bottleneck.

In terms of data bandwidth, use SATA-2 based disks. Don't access big files over network.

4) Can FlowJo run with 64bit versions of Windows (XP/Vista)?

Yes, but it's not supported. That means, if you find bugs in 64 bit windows which I can't reproduce in my 32 bit development environment, I won't be able to fix them ;( But I occasionally test the software in winXP 64 bit and it seems to work OK for version beta 7.5.

Version 7.2x seems to have a problem with vista 64 bit in that it does not find the hardware address of the computer. It's not likely we'll have "official" 64 bit support in the V7 series.

5) Can FlowJo adress more than 4 GB RAM?

In 7.5, since each "thread" gets it's own PID, it will be able to allocate it's own 32bit-compatible amount of RAM (~1.7GB). So if you ran 7.5 on a 64 bit computer which has 8GB and 4 processors, with only 4 threads you should be hitting pretty close to 80% RAM utilization. This is hypothetical, I don't have a windows computer on which I can test this claim.

6) Is RAM more important or the processor's performance?

Yes. This will be true in future versions as well regardless of processor scaling. RAM is ~1000 faster than disk, on average. When computer gets low on RAM, it uses the disk to swap chunks of calculations that don't fit into . Feeding data from disk to the processor imposes a 1000 fold performance penalty. At that point, it doesn't matter how fast the processor is!

7) Do you think it's possible to reach comparable performances with the PC version at all, or should I go and get a Mac (in case, which one would you recommend?) regardless the non-existing support here?

over the next year, while we fine-tune the new multithreading in PC, you should use a Mac for the most demanding analysis.

Apple Care plan (~300 bucks) gets you full support from the source, but if your mac goes down you have to fedex it out - so it's usually down for a week. out of the 15 or so macs I use in testing, some of which have been online for 5+ years, I've only had to repair two, and they were both laptops.

For a new desktop, the best "value" is iMac. the best performance is the quad-core mac pro (the flagship model is 8-core, so the 4-core should be much cheaper by now!)

new engine fixes kinetics bug

Download FlowJoEngine.DLL

This file fixes a bug in Kinetics where some files would show up as very straight lines. This bug is due to a non-standard Time parameter annotation in this type of data, and our fix unfortunately makes the assumption of constant event rate throughout the entire range.

Just download and plop the file into any 7.2.x installation folder in c:\program files\flowjo7, replacing the old engine.dll.

7.2.5 windows release, last version to support win2k

we're very close to putting 7.2.5 out the door. Most notably, we fixed:

- printing from workspace window
- using dongles while other USB devices are plugged in
- "out of memory" errors
- better filtering of LAN devices (nortel, etc..)

download 7.2.5 candidate build here - http://mail.treestar.com/bin/7.2.5.exe

We are also dropping the support for windows 2000 after this version - 7.3 will be xp and vista only!!

funky backgrounds

Adam writes that with one line of code, we can make backgrounds and gradients:

But what do you (the users) think - is this something you'd wanna see in the next version?

Upgrades not going so smooth

FlowJo 8.7 and 7.2.4 releases haven't worked so well outside this building. We're trying to figure out what went wrong. The tough part is we can't reproduce the errors we're receiving from our users.

 

Seven Mac V8.7 users have complained that preferences stopped working, or have not transferred, since an upgrade to V8.7  If you are experiencing this, try resetting your preferences (as outlined here.)

If resetting prefs is not an option, and you're convinced that going back to aprevious version, you can download version 8.6.3 here .

I repeated my QA testing of preferences carrying over from 8.6.3 to 8.7 on G4, G5 and intel based macs spanning OSX 10.3 thru 10.5 and it always worked on all 8 test machines.. Not sure what's going on, but we're collecting more info, working on it.

Windows 7.2.4 upgrade wasn't so great either, most users were unable to start the software after webstart did the update (~40 reports I'm aware of..). So far, we've found that resetting java webstart's cache and reinstalling the program will fix this defect. Again, worked for all our test computers. Hmm.

If for some reason clearing the cache does not fix the problem, we have made version 7.2.2 available here - 7.2.2 webstart