Figure featured in PRA Kaleidoscopes

3d matplotlib image

Q-function figure (without labels) as presented in PRA Kaleidoscope

This figure, generated from our data and visualized using matplotlib, was selected to be featured in PRA’s Kaleidoscope listings. Figures are selected for aesthetics so we are naturally pleased to have this recognition.

For more on how this figure was created, see my earlier post on publication-ready 3D figures from Matplotlib. Finally, the source for this figure is available as an Ipython notebook in our github repository:

Multimode quantum state tomography – in PRA

The measured Q-function for a weak coherent state.

The measured Q-function for a weak coherent state.

The latest paper from my lab at Pacific appears this week in Physical Review A. Multimode quantum state tomography using unbalanced array detection describes a technique we have developed to measure the quantum state of multiple optical modes simultaneously. We are currently working to apply this technique to slow and stored light systems where it will help us characterize the fidelity of optical memory devices.

Abstract — We measure the joint Q function of a multispatial-mode field using a charge-coupled device array detector in an unbalanced heterodyne configuration. The intensity pattern formed by interference between a weak signal field and a strong local oscillator is resolved using Fourier analysis and used to reconstruct quadrature amplitude statistics for 22 spatial modes simultaneously. The local oscillator and signal propagate at an angle of 12 mrad, thus shifting the classical noise to modes that do not overlap with the signal. In this configuration, balanced detection is not necessary.

How do top students study?

This is excellent advice for students, especially the comment about starting homework alone. You don't have to do it all on your own, but if you start it alone (and early!) you will find what you know and what you don't know.

Answer by David Koh:

I did pretty well at MIT, and there's a few things that I did there that haven't been mentioned yet.

The most important thing is to keep track of how good your understanding is of the material.  I saw a lot of people thought that they understood what was going on in a class, but could easily get tripped up by the basics even after the class had moved onto more advanced topics.  Having a thorough understanding of the basics is particularly key, because most advanced material is really just an extension of the basics.

For most mathy, computer science topics, there's really two parts to understanding.  The first is intuition, which for me always meant being able to build a mental model of whatever I was studying in my head.  Usually, working out examples helps with the intuition.  The second is knowing how to formally reason about the topic, which usually involves being able to work through examples step by step using logic and/or algebra, and is especially key for proofs in math and algorithms.  Most good students learn the rules for formal manipulation fairly well, but without a good intuition it becomes harder to deal with more complex problems, where the correct path is not immediately clear.

Another important thing, especially at top schools, is time management.  If one class just clicks for you and you get it easily, and another class is really hard, you should be able to spend less time on the first in order to spend more on the second.  Good students study smart, not hard (great students usually do both).

Here's some tips broken down by part of the class:


  • I find that in lecture, one of three things happens for me: (1) Everything goes too fast and I get lost. (2) I can follow what the professor is doing and I understand what they're talking about.  (3) The professor is going kind of slow, and I can usually get a good idea of where they're going before they say it.
  • In case (1), it usually means that I'm going to need to do extra work beforehand and afterwards in order to get anything out of lecture.  This usually means doing the readings beforehand, and going over the lecture notes afterwards until I feel confident I know what the professor was talking about.  Office hours and recitation are also great places to ask to have something explained again, and hopefully this time in a way you can understand.
  • In case (2), I usually don't worry too much about doing readings beforehand, but I will go over the lecture notes, since usually there are some things I didn't realize I missed.  The key thing to know here is that I never really feel like I understand the material yet at this stage, since I usually know the 'how' but not the 'why'.  The 'why' typically comes with problem sets.
  • In case (3), I may skim the lecture notes, but otherwise I'm usually good.  Case (3) usually only happens when I've seen some of the material before anyway, so I usually have a pretty good idea of how well I know it.
  • One last tip that I think is really key for lecture is to try to understand how the professor thinks and where they're coming from.  For instance, my math professor for Real Analysis always thought of the concepts he was teaching in terms of metaphors and visuals, and he would always start with that, and then tie it into the rigorous math.  From that, I knew what the structure of the lecture would be and also that I would get good intuition from the professor (but he might leave out some of the rigor).  In contrast, I had another professor in Probability who did everything through equations, which meant I got most of my intuition about topics from the book.

Problem Sets

  • Problem sets are usually the best place to find out how well you actually understand the material.  Always start them alone, because if someone else gives you the key intuition for a problem right at the beginning, they've just prevented you from finding a hole in your own understanding you didn't know was there.
  • Here's how I typically do problem sets.  First, read the whole thing, and then start on the easiest looking question.  Work on it until you've either solved it, or you're stuck and not sure what to do next.  Then move to the next easiest question, and so on, until you've tried all of them.  Which questions you can do and can't do right off the bat are good indicators of how well you understand a certain topic.
  • Once you've done a first pass of the problem set, take a short break, and then double down again on the problems, one at a time.  If you get really stuck on a problem, skip it again, but I'll usually only do this if I spend more than half an hour without getting anywhere.  You may try lots of things and hit lots of dead ends while working on problems this way, but that is good, because you are learning what doesn't work, which is something you can't usually learn in lecture (and is very important on tests). 
  • Finally, once you've given all the problems a good, honest try, go and find help.  Friends and classmates, TAs and professors are all good options.  Hopefully, they can help you get unstuck on the problems you're still stuck on.  At this point, you should be able to understand the solution they gave you, and if not, keep bugging them until you can.

Studying for Tests

  • First thing I always did to get ready for a test: Find a practice exam, and do it cold, before you study at all, no notes or helpers.  This will give you a very good idea of how well you know the material, and is likely to be an hour or two well spent.  Make sure you skip things that haven't been covered yet, if the test is from an older version of the class.  If you can't solve a problem, note that down and move on.
  • Review the test and see what you didn't know.  These are the things that are most important to study.  The other key thing to do is see what topics weren't covered on the test, and make sure you study those as well.  If you can find another test that did have a problem on that topic, I suggest doing that problem as well, to test your understanding.
  • Another good technique is to make a list of all the topics on the test, and see how much the class has covered those topics.  This gives you a good idea of how hard the problems on those topics will be.  If you've had a lot of problems or lectures on a topic, the test will probably have more advanced problems about it, but probably fewer.  Conversely, if you've only covered a topic a couple times, the test will probably give you easier problems, but there might be more of them.  This is just a rule of thumb and is sometimes completely off, but it is often helpful to guide studying.
  • Non-trivial problems – I consider test problems that require an extra insight which was not necessarily taught in class to be a special class of problems.  There's not one good way to prepare for them, since you don't really know what's coming, but having a really solid intuition of the basics is usually really helpful.  Also, a good understanding of general related topics can be invaluable, which means just being a good all around student will probably help you on the hardest tests.
  • Help other people study!  Explaining concepts you think you understand is a great way to find out where the gaps in your knowledge are.  If you don't know anyone who needs help with the class, find someone who already knows the material and "teach" it to them.  They will be able to tell you what you didn't talk about, or things that you got wrong, and they may be able to ask questions you don't know the answers to (and then answer them for you)

Taking Tests

  • You can probably find tons of test taking tips elsewhere, so I'll stick to the basics here.  Read the whole test beforehand, start with the easiest problems first, followed by the ones that are worth the most points (if you know how much they're worth).
  • If you get stuck on a problem, make sure you show work up to where you are, and go to a different one.  Partial credit on everything is good.
  • If you get stuck on a non-trivial problem which is really hard (and you're done with everything else), trying something that you're not sure if it works usually doesn't hurt, and sometimes the professor might give you partial credit if you were kind of close.  If it feels good to your intuition but you can't justify it formally, you're probably not far off.

Finally, one last note: If you feel like you're starting to get overwhelmed by a class, don't lose hope!  Go back to the basics, and learn them again.  If the class is moving too fast for you, you have to go back, put in the time to learn it at your own speed, and build up your understanding piece by piece.  Often, advanced concepts are built up one simple step at a time from the basic concepts, and if you missed some of the steps in the middle, it's easy to feel like nothing makes any sense anymore.

How do top students study?

Publication-ready 3D figures from Matplotlib

Screen Shot 2014-06-27 at 9.23.25 AMI’m a big fan of scripting my plot creation. I also use python for all of my data acquisition and analysis. This naturally put me in a spot to dive into matplotlib when it came time to create figures for a paper I’m working on. It took a bit of digging, but I worked through the kinks and put together a 3D surface plot (with contours) that is PDF and publication ready. I addressed the several issues by overriding the defaults. I want to clarify that I like the defaults for on-screen display, presentations, and posters, but they weren’t quite right for a tiny two-pane figure in a two-column manuscript. Things I “fixed” include:

  • Too many ticks and labels
  • Small fonts (when reduced to publication dimensions)
  • Tick labels misaligned relative to ticks (problem seems to come from using larger fonts to fix the previous issue)
  • Odd axes label alignment, rotation, and placement
  • Busy background (removed the gray panes with gridlines)

Some of the solutions were easy, some are hackish and only one uses the lightly-documented _axinfo dict. I’ll highlight a few snippets that were useful in this process (full code linked below).
Continue reading

Great linux tips (from Quora)

This is a repost of an answer on Quora ( by Joshua Levy:


  • Learn basic Bash. Actually, read the whole bash man page; it’s pretty easy to follow and not that long. Alternate shells can be nice, but bash is powerful and always available (learning mainly zsh or tcsh restricts you in many situations).
  • Learn vim. There’s really no competition for random Linux editing (even if you use Emacs or Eclipse most of the time).
  • Know ssh, and the basics of passwordless authentication, via ssh-agent, ssh-add, etc.
  • Be familiar with bash job management: &, Ctrl-Z, Ctrl-C, jobs, fg, bg, kill, etc.
  • Basic file management: ls and ls -l (in particular, learn what every column in “ls -l” means), less, head, tail and tail -f, ln and ln -s (learn the differences and advantages of hard versus soft links), chown, chmod, du (for a quick summary of disk usage: du -sk *), df, mount.
  • Basic network management: ip or ifconfig, dig.
  • Know regular expressions well, and the various flags to grep/egrep. The -o, -A, and -B options are worth knowing.
  • Learn to use apt-get or yum (depending on distro) to find and install packages.

Everyday use

  • In bash, use Ctrl-R to search through command history.
  • In bash, use Ctrl-W to kill the last word, and Ctrl-U to kill the line. See man readline for default keybindings in bash. There are a lot. For example Alt-. cycles through prevous arguments, and Alt-* expands a glob.
  • To go back to the previous working directory: cd –
  • If you are halfway through typing a command but change your mind, hit Alt-# to add a # at the beginning and enter it as a comment (or use Ctrl-A, #, enter). You can then return to it later via command history.
  • Use xargs (or parallel). It’s very powerful. Note you can control how many items execute per line (-L) as well as parallelism (-P). If you’re not sure if it’ll do the right thing, use xargs echo first. Also, -I{} is handy. Examples:

find . -name \*.py | xargs grep some_function

cat hosts | xargs -I{} ssh root@{} hostname

  • pstree -p is a helpful display of the process tree.
  • Use pgrep and pkill to find or signal processes by name (-f is helpful).
  • Know the various signals you can send processes. For example, to suspend a process, use kill -STOP [pid].  For the full list, see man 7 signal
  • Use nohup or disown if you want a background process to keep running forever.
  • Check what processes are listening via netstat -lntp. See also lsof.
  • In bash scripts, use set -x for debugging output. Use set -e to abort on errors. Consider using set -o pipefail as well, to be strict about errors (though this topic is a bit subtle). For more involved scripts, also use trap.
  • In bash scripts, subshells (written with parentheses) are convenient ways to group commands. A common example is to temporarily move to a different working directory, e.g.

# do something in current dir

(cd /some/other/dir; other-command)

# continue in original dir

  • In bash, note there are lots of kinds of variable expansion. Checking a variable exists: ${name:?error message}. For example, if a bash script requires a single argument, just write input_file=${1:?usage: $0 input_file}. Arithmetic expansion: i=$(( (i + 1) % 5 )). Sequences: {1..10}. Trimming of strings: ${var%suffix} and ${var#prefix}. For example if var=foo.pdf, then echo ${var%.pdf}.txt prints “foo.txt”.
  • The output of a command can be treated like a file via <(some command). For example, compare local /etc/hosts with a remote one: diff /etc/hosts <(ssh somehost cat /etc/hosts)
  • Know about “here documents” in bash, as in cat <<EOF ….
  • In bash, redirect both standard output and standard error via: some-command >logfile 2>&1. Often, to ensure a command does not leave an open file handle to standard input, tying it to the terminal you are in, it is also good practice to add “</dev/null”.
  • Use man ascii for a good ASCII table, with hex and decimal values.
  • On remote ssh sessions, use screen or dtach to save your session, in case it is interrupted.
  • In ssh, knowing how to port tunnel with -L or -D (and occasionally -R) is useful, e.g. to access web sites from a remote server.
  • It can be useful to make a few optimizations to your ssh configuration; for example, this .ssh/config contains settings to avoid dropped connections in certain network environments, not require confirmation connecting to new hosts, forward authentication, and use compression (which is helpful with scp over low-bandwidth connections):







  • To get the permissions on a file in octal form, which is useful for system configuration but not available in “ls” and easy to bungle, use something like

stat -c ‘%A %a %n’ /etc/timezone

Data processing

  • To convert HTML to text: lynx -dump -stdin
  • If you must handle XML, xmlstarlet is good.
  • For Amazon S3, s3cmd is convenient (albeit immature, with occasional misfeatures).
  • Know about sort and uniq (including uniq’s -u and -d options).
  • Know about cut, paste, and join to manipulate text files. Many people use cut but forget about join.
  • It is remarkably helpful sometimes that you can do set intersection, union, and difference of text files via sort/uniq. Suppose a and b are text files that are already uniqued. This is fast, and works on files of arbitrary size, up to many gigabytes. (Sort is not limited by memory, though you may need to use the -T option if /tmp is on a small root partition.)

cat a b | sort | uniq > c   # c is a union b

cat a b | sort | uniq -d > c   # c is a intersect b

cat a b b | sort | uniq -u > c   # c is set difference a – b

  • Know that locale affects a lot of command line tools, including sorting order and performance. Most Linux installations will set LANG or other locale variables to a local setting like US English. This can make sort or other commands run many times slower. (Note that even if you use UTF-8 text, you can safely sort by ASCII order for many purposes.) To disable slow i18n routines and use traditional byte-based sort order, use export LC_ALL=C (in fact, consider putting this in your .bashrc).
  • Know basic awk and sed for simple data munging. For example, summing all numbers in the third column of a text file: awk ‘{ x += $3 } END { print x }’. This is probably 3X faster and 3X shorter than equivalent Python.
  • To replace all occurrences of a string in place, in files:

perl -pi.bak -e ‘s/old-string/new-string/g’ my-files-*.txt

  • Use shuf to shuffle or select random lines from a file.
  • Know sort’s options. Know how keys work (-t and -k). In particular, watch out that you need to write -k1,1 to sort by only the first field; -k1 means sort according to the whole line.
  • Stable sort (sort -s) can be useful. For example, to sort first by field 2, then secondarily by field 1, you can use sort -k1,1 | sort -s -k2,2
  • If you ever need to write a tab literal in a command line in bash (e.g. for the -t argument to sort), press Ctrl-V <tab> or write $’\t’ (the latter is better as you can copy/paste it).
  • For binary files, use hd for simple hex dumps and bvi for binary editing.
  • Also for binary files, strings (plus grep, etc.) lets you find bits of text.
  • To convert text encodings, try iconv. Or uconv for more advanced use; it supports some advanced Unicode things. For example, this command lowercases and removes all accents (by expanding and dropping them):

uconv -f utf-8 -t utf-8 -x ‘::Any-Lower; ::Any-NFD; [:Nonspacing Mark:] >; ::Any-NFC; ‘ < input.txt > output.txt

  • To split files into pieces, see split (to split by size) and csplit (to split by a pattern).

System debugging

  • For web debugging, curl and curl -I are handy, and/or their wget equivalents.
  • To know disk/cpu/network status, use iostat, netstat, top (or the better htop), and (especially) dstat. Good for getting a quick idea of what’s happening on a system.
  • To know memory status, run and understand the output of free and vmstat. In particular, be aware the “cached” value is memory held by the Linux kernel as file cache, so effectively counts toward the “free” value.
  • Java system debugging is a different kettle of fish, but a simple trick on Sun’s and some other JVMs is that you can run kill -3 <pid> and a full stack trace and heap summary (including generational garbage collection details, which can be highly informative) will be dumped to stderr/logs.
  • Use mtr as a better traceroute, to identify network issues.
  • For looking at why a disk is full, ncdu saves time over the usual commands like “du -sk *”.
  • To find which socket or process is using bandwidth, try iftop or nethogs.
  • The ab tool (comes with Apache) is helpful for quick-and-dirty checking of web server performance. For more complex load testing, try siege.
  • For more serious network debugging, wireshark or tshark.
  • Know strace and ltrace. These can be helpful if a program is failing, hanging, or crashing, and you don’t know why, or if you want to get a general idea of performance. Note the profiling option (-c), and the ability to attach to a running process (-p).
  • Know about ldd to check shared libraries etc.
  • Know how to connect to a running process with gdb and get its stack traces.
  • Use /proc. It’s amazingly helpful sometimes when debugging live problems. Examples: /proc/cpuinfo, /proc/xxx/cwd, /proc/xxx/exe, /proc/xxx/fd/, /proc/xxx/smaps.
  • When debugging why something went wrong in the past, sar can be very helpful. It shows historic statistics on CPU, memory, network, etc.
  • For deeper systems and performance analyses, look at stap (systemtap) and perf.
  • Confirm what Linux distribution you’re using (works on most distros): “lsb_release -a”
  • Use dmesg whenever something’s acting really funny (it could be hardware or driver issues).