Orion in the Infrared and Millimeter

This composite image of the Orion A Giant Molecular Cloud star-forming complex shows infrared emission from the WISE and MSX missions in 4 micron (red), 12 micron (blue), and 22 micron (green) emission with Bolocam Galactic Plane Survey 1.1mm emission overlaid in yellow/orange. The Orion A region is frequently featured in astronomical images:
http://apod.nasa.gov/apod/ap110917.html
http://apod.nasa.gov/apod/ap120212.html
http://apod.nasa.gov/apod/ap120206.html
http://www.eso.org/public/images/eso1219c/
http://www.eso.org/public/news/eso1209/
but its tail tends to be ignored. This quiescent region is the source of the next generation of stars, although the relatively small mass concentrations imply that no massive stars like the bright Theta 1C that powers the Orion Nebula will form.

The infrared colors show all sorts of stars including protostars. The infrared can pierce through the dust and find young stars still forming. The green and blue bands also see diffuse clouds of dust being illuminated by the central stars of the Orion nebula.
The yellow 1.1 mm dust emission shows the coldest dust that is shielded from external radiation. These cold clumps contain enough mass to form new stars...

Sync a fork with the original repository on Git

A seemingly simple operation that I just can't seem to get right.

This page gives me useless information:

$ git checkout -b upstream/master
$ git remote add upstream git://github.com/upstream_maintainer/master.git
$ git pull upstream remote
$ git checkout master
$ git merge upstream/master


If you do that, it frankly doesn't work. Why? "upstream/master" etc. all have special meanings.

Here's the real process & explanation (thanks to Erik Tollerud for some help):

git checkout master # (assuming you have a local branch named master - otherwise, pick whatever branch you want synced)
git remote add original git@github.com:thing/thing.git # "upstream" = "original" = "remote" - the place you're trying to sync from
git fetch original # 'original' being the name YOU gave for the "remote/original" repository
git merge master original/master # now merge the "original/master" branch (they should have a branch named "master" too, otherwise you have to figure out which branch to get) into your master
git push # push your now-merged stuff back to github

# I was instructed to use these commands. They didn't work.
# git reset original # the word "original" here matches the word "original" on the previous line
# git reset --hard original # this will overwrite local changes


NaN-friendly convolution

NaN-friendly convolution is important for, e.g., masked data sets in which you want to interpolate across the masked region.

Astropy has gained this functionality with pull request 155:
https://github.com/astropy/astropy/pull/155
but this is a "direct" convolution parallel to IDL's 'convol' routine.

My FFT-based version now works in N dimensions and is a little cleaner:
http://code.google.com/p/agpy/source/browse/trunk/AG_fft_tools/convolve_nd.py

I'm still working on writing unit tests, and I'm really not sure what the "correct" behavior at the edges is for the different cases... right now, it seems counterintuitive to me, but the code is doing what I expect it to.

Also, Boxcar kernels always result in shifts for me... they're never supposed to. This is a bug.

Currently, other links to these codes:
http://stackoverflow.com/questions/1100100/fft-based-2d-convolution-and-correlation-in-python/8454010#8454010

Compiling vim...

Figured I had to post this...


I've been trying to compile command-line vim 7.3 on Mac OS X 10.7. I have the latest `hg clone`d version of vim. I'm stuck on ncurses.

If I `./configure` with no options, I get the following error:

checking --with-tlib argument... empty: automatic terminal library selection
checking for tgetent in -ltinfo... no
checking for tgetent in -lncurses... no
checking for tgetent in -ltermlib... no
checking for tgetent in -ltermcap... no
checking for tgetent in -lcurses... no
no terminal library found
checking for tgetent()... configure: error: NOT FOUND!
You need to install a terminal library; for example ncurses.
Or specify the name of the library with --with-tlib.

If instead I try `./configure --with-tlib=ncurses`

checking --with-tlib argument... ncurses
checking for linking with ncurses library... configure: error: FAILED

I have Xcode 4.1. As far as I can tell, ncurses is available:

$ file /usr/lib/libncurses.*
/usr/lib/libncurses.5.4.dylib: Mach-O universal binary with 2 architectures
/usr/lib/libncurses.5.4.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
/usr/lib/libncurses.5.4.dylib (for architecture i386): Mach-O dynamically linked shared library i386
/usr/lib/libncurses.5.dylib: Mach-O dynamically linked shared library i386
/usr/lib/libncurses.dylib: Mach-O universal binary with 2 architectures
/usr/lib/libncurses.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
/usr/lib/libncurses.dylib (for architecture i386): Mach-O dynamically linked shared library i386


Then I changed my PATH from /usr/local/bin... to /usr/bin.....

The problem was trying to use my /usr/local/bin/gcc instead of the mac default /usr/bin/gcc. Something about my locally installed gcc (4.6.1) caused major problems.


I also eventually had to do this command:

LDFLAGS=-L/usr/lib CFLAGS='-arch i386 -arch x86_64' CCFLAGS='-arch i386 -arch x86_64' CXXFLAGS='-arch i386 -arch x86_64' ./configure --enable-perlinterp --enable-pythoninterp --enable-cscope --with-features=huge

and then had to make sure my default python was NOT pointing to enthought!

LaTeX: VIM + Skim

macvim-skim-install.sh is my install script for using MacVim.app with Skim.app.

The agpy wiki page has instructions that are probably more clear; I don't really like the colorscheme / layout of this blog.

You can use synctex to make an editor and viewer work together, but it is far from easy and far harder than it should be. Forward-search is pretty easy, but the latex-suite \ls only works intermittently and is not easily customizable.

I had to do the following:

For VIM->Skim.app (Skim.app is necessary for any of this to work), add these commands to .vimrc:

" Activate skim
map ,v :w<CR>:silent !/Applications/Skim.app/Contents/SharedSupport/displayline -r <C-r>=line('.')<CR> %<.pdf %<CR><CR>
map ,p :w<CR>:silent !pdflatex -synctex=1 --interaction=nonstopmode %:p <CR>:silent !/Applications/Skim.app/Contents/SharedSupport/displayline -r <C-r>=line('.')<CR> %<.pdf %<CR><CR>
map ,m :w<CR>:silent !make <CR>:silent !/Applications/Skim.app/Contents/SharedSupport/displayline -r <C-r>=line('.')<CR> %<.pdf %<CR><CR>
" Reactivate VIM
map ,r :w<CR>:silent !/Applications/Skim.app/Contents/SharedSupport/displayline -r <C-r>=line('.')<CR> %<.pdf %<CR>:silent !osascript -e "tell application \"MacVim\" to activate" <CR><CR>
map ,t :w<CR>:silent !pdflatex -synctex=1 --interaction=nonstopmode %:p <CR>:silent !/Applications/Skim.app/Contents/SharedSupport/displayline -r <C-r>=line('.')<CR> %<.pdf %<CR>:silent !osascript -e "tell application \"MacVim\" to activate" <CR><CR>

The ,m command will reload the file and put your cursor where the text is. ,t will return VIM to the front afterwards.


Going the other way (reverse-search / inverse-search) was MUCH more challenging. The code that does this is on agpy . Reproduced here for posterity (I hope to update the agpy version to deal with tabs). [A few hours later, I HAVE replaced the code. Below are the old applescript version, then the new, vim-based version



#!/bin/bash

file="$1"
line="$2"

[ "${file:0:1}" == "/" ] || file="${PWD}/$file"

# Use Applescript to activate VIM, find file, and load it
# the 'delay' command is needed to prevent command/control/shift from sticking when this
# is activated (e.g., from Skim, where the command is command-shift-click)
#
# key code 53 is "escape" to escape to command mode in VIM
exec osascript \
-e "delay 0.2" \
-e "tell application \"MacVim\" to activate" \
-e "tell application \"System Events\"" \
-e " tell process \"MacVim\"" \
-e " key code 53 "\
-e " keystroke \":set hidden\" & return " \
-e " keystroke \":if bufexists(bufname('$file'))\" & return " \
-e " keystroke \":exe \\\":buffer \\\" . bufnr(bufname('$file'))\" & return " \
-e " keystroke \":else \" & return " \
-e " keystroke \":echo \\\"Could not load file\\\" \" & return " \
-e " keystroke \":endif\" & return " \
-e " keystroke \":$line\" & return " \
-e " end tell" \
-e "end tell"


New code: download link

#!/bin/bash

# Install directions:
# Put this file somewhere in your path and make it executable
# To set up in Skim, go to Preferences:Sync
# Change Preset: to Custom
# Change Command: to macvim-load-line
# Change Arguments: to "%file" %line

file="$1"
line="$2"
debug="$3"

echo file: $file
echo line: $line
echo debug: $debug

for server in `mvim --serverlist` 
do
    foundfile=`mvim --servername $server --remote-expr "WhichTab('$file')"`
    if [[ $foundfile > 0 ]]
    then
        mvim --servername $server --remote-expr "foreground()" 
        if [[ $debug ]] ; then echo mvim --servername $server --remote-send ":exec \"tabnext $foundfile\" "; fi
        mvim --servername $server --remote-send ":exec \"tabnext $foundfile\" "
        if [[ $debug ]] ; then echo mvim --servername $server --remote-send ":$line "; fi
        mvim --servername $server --remote-send ":$line "
    fi
done

Save that as an executable in your default path (e.g., /usr/local/bin/macvim-load-line) and open Skim.app, go to Preferences:Sync and make the command look like this:

You need to have mvim on your path. mvim comes with MacVim.app, but is NOT installed by default. Install it by doing something like:

cp /Users/adam/Downloads/MacVim-7_3-53/mvim /usr/local/bin/mvim

You'll also need to install WhichTab.vim in your ~/.vim/plugins/ directory. It's available here ( download link ). Here's the source:

function! WhichTab(filename)
    " Try to determine whether file is open in any tab.  
    " Return number of tab it's open in
    let buffername = bufname(a:filename)
    if buffername == ""
        return 0
    endif
    let buffernumber = bufnr(buffername)

    " tabdo will loop through pages and leave you on the last one;
    " this is to make sure we don't leave the current page
    let currenttab = tabpagenr()
    let tab_arr = []
    tabdo let tab_arr += tabpagebuflist()

    " return to current page
    exec "tabnext ".currenttab

    " Start checking tab numbers for matches
    let i = 0
    for tnum in tab_arr
        let i += 1
        echo "tnum: ".tnum." buff: ".buffernumber." i: ".i
        if tnum == buffernumber
            return i
        endif
    endfor

endfunction

function! WhichWindow(filename)
    " Try to determine whether the file is open in any GVIM *window*
    let serverlist = split(serverlist(),"\n")

    "let currentserver = ????
    for server in serverlist
        let remotetabnum = remote_expr(server, 
            \"WhichTab('".a:filename."')")
        if remotetabnum != 0
            return server
        endif
    endfor

endfunction

API documentation on agpy

I finally processed agpy through sphinx and made some nice html documentation.

http://agpy.googlecode.com/svn/trunk/doc/html/agpy.html

latex: producing a bibliography and paper independently

AG

If you want citations to work, but you don't want your bibliography to show up, try the following:

latex file.tex
bibtex file

comment out \biblography{} line

latex file.tex


If you latex again, it will screw up.

To make an independent bibliography, remove all text and replace all citep/citet/cite commands with \nocite{} in a different document. Remember to usepackage{natbib} etc. You may have to copy over the .bbl file.

mercurial merge

AG

My most hated behavior of mercurial:

searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 3 changes to 3 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
remote: 1 changesets found
running hook post-pull: hg up
abort: crosses branches (merge branches or use --clean to discard changes)
warning: post-pull hook exited with status 255
$ hg merge
abort: outstanding uncommitted changes (use 'hg status' to list changes)
$ hg commit
nothing changed

Solution:
hg merge --force
Hopefully there are other solutions that I'll eventually add to this.

Offset symbols?

AG

my scipy install...

As far as I was able to reconstruct, my scipy install looked like this when it went well:

mkdir scipy-bin
cp ../scipy-svn/site.cfg .
export PATH=/Users/adam/repos/scipy.git/scipy-bin:$PATH
ln -s /usr/bin/g++-4.0 scipy-bin/g++-4.0
ln -s /usr/bin/g++-4.0 scipy-bin/c++
export CC=/usr/bin/gcc-4.0
ln -s /usr/bin/gcc-4.0 scipy-bin/
ln -s /usr/bin/gcc-4.0 scipy-bin/gcc
ln -s /usr/local/bin/gfortran-4.0 scipy-bin/gfortran-4.0
ln -s /usr/local/bin/gfortran-4.0 scipy-bin/gfortran
ln -s /usr/local/bin/g95 scipy-bin/g95
ln -s /usr/local/bin/i686-apple-darwin8-gfortran-4.2 scipy-bin/
python2.7 setup.py build
python2.7 setup.py install


However, site.cfg included pointers to AMD and UMFPACK that were installed via the incredibly complicated series of steps listed here: http://blog.hyperjeff.net/?p=160

AG

P Cygni

P Cygni is a neglected LBV. You search for Eta Carinae on google and get hundreds of epic images of the fella, but nothing of P Cyg! Not even my Fe II image, which is now reproduced here:

finally got matplotlib to install...

the key is reading the readme, not just the make.osx file.

These commands Just Work:

make -f make.osx PYVERSION=2.6 PREFIX=/Users/adam/repos/mpl_dependencies/ fetch deps mpl_install_std
make -f make.osx PYVERSION=2.7 PREFIX=/Users/adam/repos/mpl_dependencies/ fetch deps mpl_install_std

while, e.g., this one:
make -f make.osx PYVERSION=2.6 PREFIX=/Users/adam/repos/mpl_dependencies/ fetch deps mpl_install
didn't. I guess because that one doesn't actually install anything.

matplotlib 64 bit installs never work

a clue as to why:
functional ft2font.so:

$ otool -L /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/ft2font.so
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/matplotlib/ft2font.so:
/usr/local/lib/libfreetype.6.dylib (compatibility version 10.0.0, current version 10.22.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
/usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.14.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)


nonfunctional ft2font.so:

$ otool -L /Users/adam/repos/yt/yt-i386/lib/python2.7/site-packages/matplotlib/ft2font.so
/Users/adam/repos/yt/yt-i386/lib/python2.7/site-packages/matplotlib/ft2font.so:
/Users/adam/repos/yt/yt-i386/lib/libfreetype.6.dylib (compatibility version 13.0.0, current version 13.2.0)
/Users/adam/repos/yt/yt-i386//lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
/usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.14.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)
/usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)


The culprit is the difference between those two (maybe?):

$ file /usr/local/lib/libgcc_s.1.dylib
/usr/local/lib/libgcc_s.1.dylib: Mach-O universal binary with 4 architectures
/usr/local/lib/libgcc_s.1.dylib (for architecture i386): Mach-O dynamically linked shared library i386
/usr/local/lib/libgcc_s.1.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
/usr/local/lib/libgcc_s.1.dylib (for architecture ppc): Mach-O dynamically linked shared library ppc
/usr/local/lib/libgcc_s.1.dylib (for architecture ppc64): Mach-O 64-bit dynamically linked shared library ppc64
$ otool -L /usr/local/lib/libgcc_s.1.dylib
/usr/local/lib/libgcc_s.1.dylib:
/usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.9)


Major mac problems

The errors are, in short:

• Browser stops responding / starts returning "page not found"
  (indicating a failure of mDNSResponder)
  
• killing mDNSResponder sometimes brings browser back, but more often
  leads to a partial system freeze (some windows don't respond, can't
  switch between windows except by clicking)
  
• /var/log/system.log gets flooded with "too many files open" errors.
  
• somewhere in here the Dock fails
  
• killing Google Chrome and/or the Dock fails; the process never halts
  (even kill -9 + kill -s SIGCHLD)
  
• usually one or two crash reports pop up, at least one of which is for
  crash_reporter
  
• system.log stops getting flooded, but the browser and Dock never recover
  

The only message in system.log that gives me any hint about what might be happening is occasionally a freeze was resolved at the same time as this message: Jun 22 19:02:09 eta Quicksilver[93771]: Multiple Scans Attempted but it doesn't seem to change the situation if quicksilver is open or not. Google has nothing on this issue, either, except for the quicksilver source code, so evidently it has not caused problems for other people system.log was also being flooded with this message:

Jun 23 08:57:19 eta postfix/master[99954]: fatal: open /dev/null: Bad file descriptor
Jun 23 08:57:20 eta com.apple.launchd[1] (org.postfix.master[99954]): Exited with exit code: 1
Jun 23 08:57:20 eta com.apple.launchd[1] (org.postfix.master): Throttling respawn: Will start in 9 seconds

so I disabled my postman:

Jun 23 08:57:27 eta sudo[99955]: adam : TTY=ttys006 ; PWD=/Users/adam/proposals/alma ; USER=root ; COMMAND=/bin/launchctl unload -w /System/Library/LaunchDaemons/org.postfix.master.plist

These errors:

Jun 23 09:06:40 eta Dock[99877]: kCGErrorIllegalArgument: CGSSetWindowTransformAtPlacement: Singular matrix [nan 0.000 0.000 nan]
Jun 23 09:06:40 eta com.apple.Dock.agent[99877]: Thu Jun 23 09:06:40 eta.colorado.edu Dock[99877] : kCGErrorIllegalArgument: CGSSetWindowTransformAtPlacement: Singular matrix [nan 0.000 0.000 nan]

are correlated with opening Chrome windows and/or Chrome's crash_inspector

Jun 23 09:06:09 eta [0x0-0x69e69e].com.google.Chrome[99995]: [99995:24579:485131152128125:ERROR:shared_memory_posix.cc(164)] Creating shared memory in /var/folders/ni/ni+DtdqFGMeSMH13AvkNkU+++TI/-Tmp-/.com.google.chrome.sHcu6r failed: Too many open files in system

This is the problem that really gets me... I think it's crash_inspector's fault.

But there's definitely more going on here than just Chrome. Trying to change default browsers (by opening Safari and opening Preferences) led to a partial Dock crash (?!) in which I can alt-tab but can't see the Dock. Not clear at all what's going on.... argh.

Dying Dock

My dock keeps dying. Repeatedly. Over and over.

Only solution so far:

ps -vax | grep -E "Dock|PID"
kill -HUP PID
kill -s SIGCHLD PID


And similarly for problems with Chrome + /usr/sbin/mDNSResponder. They tend to go bad together.... no clues yet from the system logs. Ironically, the crash reporter seems to fail the most often...

specanpy

AG

There are now 2 clones of PySpecKit:
sPecAnPy
Spectroscopic Toolkit (Astronomy)

Why can't numpy do duplicate index assignment

AG

I want to do drizzling with numpy. It should be trivial, but it's impossible (without a for loop, afaik) instead.


In [2]: a = array([1,1,2,2])

In [3]: b = arange(5)

In [4]: b[a] += 1

In [5]: b
Out[5]: array([0, 2, 3, 3, 4])

In [6]: # but b should really be:

In [7]: b[a] += 1

In [8]: b
Out[8]: array([0, 3, 4, 3, 4])


pyspeckit: an astronomical spectroscopic toolkit

Jordan and I have been working on our python-based spectroscopic analysis tool for a while now:
pyspeckit is a pretty awesome, now functional but incomplete (and incompletely documented) tool.

pstopng

Following this thread and my need to convert IDL .ps files to .pngs so that I can view them in Mac Preview without having to go through an (often failed) conversion process led to a few discoveries.

First, it is challenging to get ImageMagick's convert to make an opaque background for the .ps file without seriously degrading the resolution. This can be accomplished by passing both the -alpha Off and the -density 300 simultaneously. This is slow, though, and recommendations to speed it up using the -limit area 4096 -limit memory 4096 tags actually made it slower!

However, the thread pointed out that ghostscript can do the conversion directly:
gs -dBATCH -sDEVICE=png16m -r300 -dEPSCrop -dNOPAUSE -sOutputFile=XXX.png XXX.ps

-sDEVICE sets the output to png, -r300 tag sets the density to be 300 pixels/inch, -dEPSCrop is necessary to get the right sized image out (otherwise it defaults to a portrait 8.5x11), and -dBATCH prevents the gs command line from activating after the command is executed. I'm not sure if -dNOPAUSE is necessary, but apparently if you don't activate it you have to do something after every page is processed.

My code to do batch ps-to-png conversion is available at http://code.google.com/p/agpy/source/browse/trunk/agpy/pstopng.

Timing demonstrations (units are seconds, R is 'real' or clock time, 'U' is user time, and 'S' is system time):

/usr/local/bin/convert -density 300 -alpha Off deline_zero_10hz_timestreams_003.ps deline_zero_10hz_timestreams_003.png
TIMING: R: 3.126 U: 2.948 S: 0.084
/usr/local/bin/convert -limit area 4096 -limit memory 4096 -density 300 -alpha Off deline_zero_10hz_timestreams_003.ps deline_zero_10hz_timestreams_003.png
TIMING: R: 3.800 U: 2.970 S: 0.161
gs -dBATCH -sDEVICE=png16m -r300 -dEPSCrop -dNOPAUSE -sOutputFile=deline_zero_10hz_timestreams_003.png deline_zero_10hz_timestreams_003.ps
TIMING: R: 0.801 U: 0.781 S: 0.017


Converting GILDAS-CLASS data cubes (lmv files) to fits

As usual, CLASS documentation is nearly impossible to navigate. At the end of the CLASS "introduction" (gildas-intro.pdf) there is a subtle and obscure reference to the vector\fits command. The conversion is actually relatively straightforward:

vector\fits outfile.fits from infile.lmv



AG