Nuke: Copy with Expressions

So, here's a thing I made.

I was recently doing some motion graphics work in Nuke, as I do. I had several elements in a shot that I needed to animate-on with the same motion, speed, size, etc.

The graphics were already in their "final" positions, having done a full layout before animating. Now, I just wanted to add an animated Transform to each element. And I wanted to be able to easily adjust them all, together, as I finessed the animation.

But I couldn't just Clone a Transform and paste it into the other branches. The problem being that none of the elements shared the same Anchor Point (Center). If I cloned the Transform, all of the graphics would be scaling from the center of the first element, not their own centers.

I needed a handful of Transforms that were linked by all of their parameters except Center.

So, rather than spending 15 minutes writing and copy/pasting expressions to link all of the various knobs on the Transform nodes, I spent an hour writing a Python tool that will do it with a keyboard shortcut.

def CopyWithExp():
  sourceNode = nuke.selectedNode().name()
  nuke.nodeCopy("%clipboard%")
  nuke.nodePaste("%clipboard%")
  destNode = nuke.selectedNode().name()
  for i in nuke.selectedNodes():
      for j in i.knobs():
          i[j].setExpression( sourceNode + '.' + j )
      i['xpos'].setExpression('')
      i['ypos'].setExpression('')
      i['selected'].setExpression('')
      i['channels'].setExpression( sourceNode + '.channels' )


nuke.menu('Nuke').addCommand('Edit/Copy with Expressions', "CopyWithExp()", "^#C")

(This goes in your Menu.py file in your ./nuke directory.)

Now, when I press ⌥+⌘+C, it will duplicate the selected node, and link every knob with an expression. Essentially a DIY Clone, but with the ability to easily "declone" individual parameters by right clicking on the parameter and selecting Set to default.

What's Up With That Extra Junk In The For Loop?

It wouldn't be a homemade tool if it didn't include some hacky code to fix some unexpected results. As it turns out, when you programmatically link every knob from one node to another, you also end up linking the hidden knobs that are not exposed to the user in the GUI. Which is not always good.

In the for loop above, you'll see that, after linking every knob between the old and new nodes, I'm reverting the parameters for xpos, ypos, and selected. These parameters are the x and y position of the node on the node graph, and whether or not the node has been selected.

For obvious reasons, we'd like the ability to select the nodes individually. And, if we don't unlink the x and y positions of the nodes, the new node will be permanently affixed atop the old node. You won't even be able to see the original node. Not super helpful.

I've also "manually" linked the channels knob. For some reason, it was not expression linking correctly on its own. It would end up linked to the channel knob, which is a different thing entirely. So, rather than figuring out why it wasn't working, I lazily fixed it with an extra line of code.

Does It Work With Nodes That Aren't Transform Nodes?

Yes, it does. But you may discover a node that has a parameter that breaks in the duplication, like the channels knob did in the Transforms. If/when you find a broken knob, you can add it to the "whitelist" of parameters at the end of the for loop and so on, and so on.

If you'd like to do some exploring, you can see a full list of the knobs associated with a node by firing up Nuke's Script Editor and running the following command while the node is selected:

for i in nuke.selectedNode().knobs():
    print i

Ugh. Are There Any Other Limitations?

There totally are.

Being that the new nodes created are linked via expressions, copying the nodes into a new Nuke project will result in the same error message one would see copying any expression linked node into a new script. It will complain that it can't find the source node that the expressions are looking for. Even if you copy the source node with the expression linked nodes, it will throw an error, then, when you dismiss the error message, the nodes will work. Nuke.

Also, the nuke.nodeCopy("%clipboard%") and nuke.nodePaste("%clipboard%") commands in the Python script use the system clipboard to duplicate the node. This isn't different than using the normal system copy and paste tools in Nuke, but some of you out there use weird clipboard utilities that do things I can't predict. So. There's not much I can do for you there.

Anyway

This tools comes from a conversation I had with friend-of-the-blog, and my podcast co-host, Joe Steel. I was complaining about this problem (and others) in our Slack channel, and he mentioned that Katana had a Copy with Expressions command that would do what I was asking.

I'm actually surprised this feature isn't already built in to Nuke, but now, thanks to Joe, all of our Expression-Linked Dreams have come true.

Customizing Native Nuke Nodes with addOnCreate

As much as I enjoy building custom gizmos to make my work in Nuke more enjoyable, they're really no replacement for native nodes that (hopefully, eventually) include the tweaked feature that I want.

In fact, trying to use a custom gizmo as a replacement for a native node can add friction since I end up with two results in my tab+search window rather than one. When trying to use my beloved FrameHold gizmo, I end up adding the old, dumb, native FrameHold node about half of the time. It's partly because there are two FrameHold nodes in my results, and partly because my custom gizmo shows up lower in the search results than than the native node. Sure, I could rename my gizmo to something unique to avoid ambiguity in the search, but that would come at the cost of precious muscle memory.

But, thanks to an email from reader Patrick, I've recently become aware of the addOnCreate command in Nuke's Python API. Essentially, addOnCreate allows you to define a function to be run whenever a given Class of node is added, opening the door to customizing the native Nuke nodes as they're added to your script.

As a quick test, I used addOnCreate to add some TCL code to the labels of my TimeOffset and Tracker nodes.

# Time Offset Label Modification

def OnTimeOffsetCreate():
  nTO = nuke.thisNode()
  if nTO != None:
    label = nTO['label'].value()
    nTO['label'].setValue('[value time_offset]')

# Tracker Label Modification

def OnTrackerCreate():
  nTR = nuke.thisNode()
  if nTR != None:
    label = nTR['label'].value()
    nTR['label'].setValue('[value transform]\n[value reference_frame]')

# Add the custom labels to newly created nodes

nuke.addOnCreate(OnTimeOffsetCreate, nodeClass="TimeOffset")
nuke.addOnCreate(OnTrackerCreate, nodeClass="Tracker4")

(code only)

I've long been a fan of using the label of my TimeOffset nodes to show me how many frames have been offset. Especially handy for my kind of work, where a single animated element can be reused dozens times, offset in time, and randomized to make large animations easier to create and manage. For Tracker nodes, it's important to keep track of both the Reference Frame used, as well as the type of Transform being applied. Now, every time I add a TimeOffset or Tracker node, the additional information is automatically added to my node label.

Nuke default nodes on top. With custom labels below.

As expected, there are limits to what you can modify with the Python API but, henceforth, this method of interface customization is going to be my preference, resorting to creating gizmos as a fall-back when I run into a limitation. The thought of dropping a single Menu.py file into my local .nuke directory, and having all my Nuke customizations show up, is incredibly appealing to me.

Node Sets for Nuke v1.1

Since creating the Node Sets for Nuke toolset back in June, I've been using it like crazy on all of my projects. Which has led to the discovery of one incredibly obnoxious bug.

This little guy is the maxPanels property at the top of the Properties Pane:

This is where you set the maximum number of node properties panels that can be open simultaneously. I usually keep mine set to 3 or 4. When I open a node's properties panel, if I already have the maximum number of panels open, the oldest panel, at the bottom of the list, is closed and the new panel opens on top. Which is great.

Unless, of course, you're trying to simultaneously open an unknown number of properties panels, all at the same time.

When using the Node Sets tool for showing all nodes in a set, I would have to manually set the maxPanels number to a value greater than or equal to the number of nodes I'd already tagged, prior to running the command. Since I usually have no idea how many nodes are in a set, I end up setting the maxPanels property to something I know is way too high, like 35. That way, when the Show Nodes in Set function runs, I won't be left looking at only 3 of my tagged nodes.

But since the Show Nodes in Set command is already searching through all the nodes to see which ones are tagged, wouldn't it be great if it could keep a tally as it searches and automatically update the maxPanels property to match?

Yes. That would be nice.

# Node Sets for Nuke v1.1

# This function opens the control panels of
# all the nodes with "inNodeSet" on their label

def showOnlyChosenNodes():
  names = []
  li = []
  for node in nuke.allNodes():
    if "inNodeSet" in node['label'].value():
      names.extend([node.name()])
      li.extend([node])
  numPan = nuke.toNode('preferences')['maxPanels']
  numPan.setValue(len(names))
  for i in range(len(li)):
    node = li[i]
    node.showControlPanel()

# This function adds "inNodeSet" to a
# new line on the label of all the selected nodes

def labelNodes():
  for node in nuke.selectedNodes():
    label = node['label'].value()
    if 'inNodeSet' not in label:
      node['label'].setValue( label +  '\ninNodeSet')


# and this one clears the label of
# all the selected nodes

def unLabelNodes():
  for node in nuke.selectedNodes():
    label = node['label'].value()
    if 'inNodeSet' in label:
      node['label'].setValue( label.replace('\ninNodeSet','') )


toolbar = nuke.menu("Nodes")
nsets = toolbar.addMenu("Node Sets")
nsets.addCommand('Node Set: Show Nodes', 'showOnlyChosenNodes()')
nsets.addCommand('Node Set: Add Selected', 'labelNodes()')
nsets.addCommand('Node Set: Remove Selected', 'unLabelNodes()')

In addition to updating the showOnlyChosenNodes() function, I've also renamed the actual menu commands to all start with Node Set:. This way, I can start a tab+search with the same three letters, nod, and quickly narrow results to the only 3 tools that fit that criteria; my Node Set tools.

I love using Node Sets in Nuke and I'm glad to finally be rid of this annoying workaround.

iOS Spinner Gizmo for Nuke

Throughout the course of my work, I end up rebuilding and animating a lot of app interfaces. One of the most common pieces of the iOS interfaces I create is the "thinking" spinner.

It's one of those things that I almost always need, and typically forget about until I need one. Which usually results in me grabbing a screenshot of a spinner from the app and animating it to rotate. Which is not actually what it does [1].

Since I like to use my free time between projects to automate repetitive tasks, I built a Nuke Gizmo to take care of this in the future.

Default Spinner

Dark Mode Spinner

In order to make it useful in almost any situation, I made the spinner gigantic. It's built on a 1024x1024 Constant. It's got alpha and it's premultiplied, ready to go. It makes a full revolution in exactly 1 second [2].

The exposed controls are exceedingly basic. It has Transform, Scale, and Dark Mode controls. The default spinner is white on a semi-transparent black round-rect. Dark Mode switches it to a black spinner with no round-rect backdrop; typically used on mostly-white interface images.

I'm going to get a ton of use out of this little thing and, if you think you will too, hit the download link below. Drop the gizmo file into your .nuke directory and add the following to the bottom of your Menu.py file:

toolbar = nuke.menu("Nodes")
gzmos = toolbar.addMenu("Gizmos")
gzmos.addCommand("iOS Spinner", "nuke.createNode('iOS-spinner')")

Download: iOS-spinner.gizmo

---

1. I animate them with an expression, not keyframes. I’m not an animal.  ↩

2. Not the absurdly slow speed your browser is displaying the GIFs.  ↩

Automated Viewer Frame Handles in Nuke

When working on a VFX shot, more often than not, the plate's image sequence will include frame handles; an additional 8 or 12 frames at the start and end of the shot. The extra half or third of a second at the ends of the shot give us the flexibility to adjust in and out points later without having to come back to our compositing application and re-render the shot with a new frame range. Which is great.

But, when working on a shot in, say, Nuke, you'll often want to play back the shot without viewing the currently-extraneous frames. Lucky for us, Nuke has a these fun little red triangles on its timeline that you can drag around to set custom in and out points as you see fit. It's a great feature for focusing on a smaller range of frames for a specific task.

View fullsize

But, if you're trying to see eactly the frames that are currently in the edit, and only those frames, dragging around tiny red triangles isn't terribly precise. And the other option, typing in the first frame plus the handle, a hyphen, then the last frame minus the handle, is slow and requires math (yuck).

This being a well-defined, menial, and repetitive task, it's a perfect candidate for automation. With help, again, from my good friend Jake at The Foundry Support, I've added two new items to the bottom of my Viewer menu.

View fullsize

Now, all it takes is a single click (or keyboard shortcut, if you're so inclined) to quickly add 12 or 8 frame handles to my Viewer Timeline Range.

Here's the code to add to your Menu.py file in your .nuke folder:

def newViewerRange12():
  # Get the node that is the current viewer
  v = nuke.activeViewer().node()
  # Get the first and last frames from the project settings
  firstFrame = nuke.Root()['first_frame'].value()
  lastFrame = nuke.Root()['last_frame'].value()
  # get a string for the new range and set this on the viewer
  newRange = str(int(firstFrame)+12) + '-' + str(int(lastFrame) - 12)
  v['frame_range_lock'].setValue(True)
  v['frame_range'].setValue(newRange)


def newViewerRange8():
  # Get the node that is the current viewer
  v = nuke.activeViewer().node()
  # Get the first and last frames from the project settings
  firstFrame = nuke.Root()['first_frame'].value()
  lastFrame = nuke.Root()['last_frame'].value()
  # get a string for the new range and set this on the viewer
  newRange = str(int(firstFrame)+8) + '-' + str(int(lastFrame) - 8)
  v['frame_range_lock'].setValue(True)
  v['frame_range'].setValue(newRange)

# Add the commands to the Viewer Menu
nuke.menu('Nuke').addCommand('Viewer/Viewer Handles - 12f',
"newViewerRange12()")
nuke.menu('Nuke').addCommand('Viewer/Viewer Handles - 8f',
"newViewerRange8()")

Naturally, if you only ever deal with one duration for your frame handles, you can add just one of the functions, but I like having the flexibility of both options.

That Jake, he sure is good with the Python code, isn't he?

Update - 2014-10-25

The release of Nuke 9 is just around the corner and its revamped Viewer interface brings with it a small bug in my Viewer Frame Handles code (allegedly).

Design decisions in Nuke's new big brother, Nuke Studio, have made their way into the standalone app (reportedly) and now require the Frame Range Lock to be set before the new Frame Range is defined (or so I hear). Keen observers will notice that the last 2 lines of the newViewerRange functions above have been swapped to reflect this change.

According to a friend who's not me, the old code will still work in Nuke 9, but requires you to have existing In and Out points prior to running the command or run the command twice.

If you're planning on using this tool with Nuke 9 and Nuke Studio, you should update your Menu.py file now. The updated code will run properly in both Nuke 8 and Nuke 9 (so the story goes).

Node Sets in Nuke

UPDATE: A newer version of this plugin exists here.

The story goes like this. It may sound familiar.

You're working on an animation in your favorite node-based compositing application, and you want to make a timing change. The first half of the animation is perfect, but it should hold a little longer before it finishes, to better match the background plate.

Problem is, you've got animated nodes all over your script, and all of their keyframes need to move in sync. Transform nodes, Grade nodes, GridWarp nodes.

You zoom in and move around your script, looking for nodes with the little red "A" in the upper right corner.

No, not that node. That one's for that other asset and it doesn't need to move.

Okay, got 'em all open? Now switch the the Dope Sheet, grab everything after frame 75 and slide it to the right a few frames. Done?

Let's watch the new timing.

Shit. Forgot one.

Which one?

Oh, here it is. Wait. How many frames did the other 6 nodes move?

Sigh.

CMD+Z. CMD+Z. CMD+Z. CMD+Z. CMD+Z. CMD+Z.

Okay, are they all open this time? Good. Now slide them all together.

Done? Let's watch it.

Better.

10 Minutes And 20 Additional Nodes Later.

Well...now I need a little less time between frames 30 and 42.

Dammit.

Feature Request

This is the annoying scenario I found myself repeating about a dozen times on a recent project, so I sent an email to The Foundry's support team, requesting the addition of a feature I described as "Node Sets".

A Node Set is an arbitrary collection of nodes that can be opened all at once with a single command. New nodes can be added as the script grows, or removed if they're no longer needed.

Along with my feature request, I provided these two screenshots to help explain:

View fullsize

View fullsize

What I received back from Jake, my new best friend at The Foundry Support, was the following script:

# This function opens the control panels of
# all the nodes with "inNodeSet" on their label

def showOnlyChosenNodes():
  for node in nuke.allNodes():
    if "inNodeSet" in node['label'].value():
      print node.name()
      node.showControlPanel()
    else:
      node.hideControlPanel()

# This function adds "inNodeSet" to a
# new line on the label of all the selected nodes

def labelNodes():
  for node in nuke.selectedNodes():
    label = node['label'].value()
    if 'inNodeSet' not in label:
      node['label'].setValue( label +  '\ninNodeSet')

# This function clears the label of
# all the selected nodes

def unLabelNodes():
  for node in nuke.selectedNodes():
    label = node['label'].value()
    if 'inNodeSet' in label:
      node['label'].setValue( label.replace('\ninNodeSet','') )

# These commands create a new menu item with
# entries for the functions above

nuke.menu('Nuke').addCommand('Node Sets/Show Nodes in Set', "showOnlyChosenNodes()")
nuke.menu('Nuke').addCommand('Node Sets/Add Selected Nodes to Set', "labelNodes()")
nuke.menu('Nuke').addCommand('Node Sets/Remove Selected Nodes from Set', "unLabelNodes()")

For those of you who don't speak Python, allow me explain what's happening here. Once added to your Menu.py file, the script creates 3 tools in a new menu within Nuke.

View fullsize

Just as I requested, I have the ability to add or remove selected nodes from the group, then, when I need to make a change, open all of those nodes with a single command.

Magic.

Not Magic

What the script is actually doing is tagging the nodes. No, Nuke did not suddenly or secretly gain the ability to add tags to things, it's cleverly using the label section in the Node tab to hold the inNodeSet text. The Show Nodes in Set command simply scans all the nodes in your script for nodes with inNodeSet in their labels, and opens them. Simple. Smart.

View fullsize

As a result, yes, you can add the inNodeSet text to the label field manually, rather than using the new menu command, and the Show Nodes in Set command will find it, but who would want to do such a barbarous thing?

Customization

As with all commands in Nuke, a keyboard shortcut can be added to these commands to make the process even quicker. But, since I don't particularly enjoy cluttering up my menu bar with unnecessary menus, nor do I enjoy having more keyboard shortcuts than I can remember (I totally already do), I opted to move the commands into the Nodes menu. This is easily done by swapping the last 3 lines of Jake's script with these lines:

toolbar = nuke.menu("Nodes")
nsets = toolbar.addMenu("Node Sets")
nsets.addCommand('Show Nodes in Set', 'showOnlyChosenNodes()')
nsets.addCommand('Add Selected Nodes to Set', 'labelNodes()')
nsets.addCommand('Remove Selected Nodes from Set', 'unLabelNodes()')

Here's where my tools now live.

View fullsize

I do this for one major reason; having these tools available in the Tab + Search tool. For those unfamiliar, Nuke has a built in tool similar to Spotlight or LaunchBar that allows you to press Tab then type the name of the tool you're looking for, avoiding the need to have keyboard shortcuts for every type of node.

View fullsize

Current Limitations

This being a bit of a hack, there are naturally a few limitations. First and foremost, using this tool will delete anything you already had in the label field of a node. I doesn't support the ability to add a tag to the text in the label field. The tag has to be the only thing in the label field.

Secondly, once you realize how useful this is, you may want to have more than one Node Set at your disposal. The good news about this current limitation is that you can very easily create as many node sets as you want by duplicating the code and changing the inNodeSet tag to something like inNodeSet2.

Of course, with multiple node sets, it'd be ideal if you could include a given node in multiple sets at the same time, but like I mentioned, this is not a real tagging system. If real tagging ever makes its way into the application, I imagine such a thing will then be possible.

Update - 2014-06-25

I emailed my pal Jake again, telling him how much I appreciate his work on this script, and you'll never guess what he did. He sent me an updated version of the script that adds the tag to the node label without overwriting the current text in the field.

Not only is this great for general usability, it means we can add a node to multiple Node Sets at the same time. We now have a real tagging system built into Nuke. How great is Jake? Seriously.

One thing I will note; if you are planning on using multiple Node Sets, you'll want to change the default tag to inNodeSet1. If you leave it as inNodeSet, it will also show up in results for other tags, like inNodeSet2.

Attribution

If it wasn't clear before, all credit for this script goes to The Foundry and their awesome support team. They continue to be one of my favorite companies, specifically because they offer great support in addition to their great products.

I'm incredibly happy to have this annoyance removed from my workflow, and I hope you are too.

Creating Unique Footnotes with MultiMarkdown Metadata

Footnotes. Writers love ’em. But if you’re not paying proper attention when creating them, you can quickly make a mess of your website or blog. In fact, until very recently, this site had broken footnote links all over the place. I’m going to pretend like none of you noticed, and tell you about how I fixed the problem, gently glossing over my past stupidity.

Each post on this site starts as a MultiMarkdown document in Sublime Text 2. When it’s complete, I preview it in Marked, then copy the HTML code and paste it into a new Squarespace post.

The Problem

Thing is, since I’m creating the post locally on my Mac, with no reference to the site as a whole, Marked has no way of knowing which footnote IDs have already been used, and which have not. Therefore each post labels its first footnote fn:1 and subsequent footnotes increase by 1, as you’d expect. The problem occurs when viewing the home page that shows 10 posts on a single page, each with their own fn:1. When you click on the [1] link in the body text, where do you think it’s going to take you? That’s right, to the first footnote it finds with the label fn:1, regardless of which post it was intended to link to [1].

Now, Marked has a setting in its preference pane called Use Random Footnote IDs which is used “to avoid conflicts when multiple documents are displayed on the web.” So this is already a solved problem, right? Probably, but there’s a part of my brain that feels uneasy about using randomly generated IDs. I’m sure using this feature would solve my problem and everything would be fine, but since I’m a nut, I want to control exactly how these unique footnotes are created.

Enter MultiMarkdown. MultiMarkdown includes a number of enhancements to the original Markdown syntax [2], including support for customizable metadata fields [3]. So, I created a custom metadata key called footnote:. Here’s what my boilerplate MultiMarkdown header looks like [4] :

Title:            
Author:         Dan Sturm      
Date:           %snippet:ddate%  
Marked Style:   ds_doc  
footnote:         

Now, whatever keyword I choose to add after the footnote: label will show up in the HTML header as [5].

Make With The Magic

The next thing I needed was a script to scan the HTML for the metadata key, and add it to all the footnote IDs, turning fn:1 into fn:my-footnote-key1.

import sys  
import re  
import subprocess


#   Open the file and convert it to searchable text  

fileText = open (sys.argv[1], "r+")  
searchText = fileText.read()


#   Pull the MultiMarkdown metadata key from the header  

mmdkey = re.search("(?<=\"footnote\" content=\")(.*)(?=\"/>)", searchText, re   .I)  
mmdkey = mmdkey.group()


#   Create the new footnote IDs  

fnnew = ("fn:", mmdkey)  
fnrefnew = ("fnref:", mmdkey)  

fnnew = "".join(fnnew)  
fnrefnew = "".join(fnrefnew)


#   Swap the footnote IDs for the new ones  

fnfix = re.sub("(fn:)", fnnew, searchText)  
fnreffix = re.sub("(fnref:)", fnrefnew, fnfix)


#   Strip HTML Header and Body tags. Copy the result to Clipboard  

def setClipboardData(data):  
  p = subprocess.Popen(['pbcopy'], stdin=subprocess.PIPE)  
  p.stdin.write(data)  
  p.stdin.close()  
  retcode = p.wait()  

fixStripped = re.sub("(?s).*(\n)\n", "", fnreffix)  
fixStripped = re.sub("(?s)\n\n\n().*", "", fixStripped)  
setClipboardData(fixStripped)


#   Write the updated code to the same file  

# fileText.seek(0)  
# fileText.write(fnreffix)  
# fileText.truncate()  
# fileText.close()  

Click here to download the script.

Now that we have that out of the way, let’s talk about how to actually use this code. I’m using a Hazel [6] rule pointed at a folder called Create Unique Footnotes. It looks like this:

Simply put, it watches the folder for a new file and, when it sees one, it runs the script, displays a confirmation message, then trashes the now-unnecessary file, leaving the cleaned up, new code on the clipboard for easy pasting into your CMS of choice. So, when I like what I see in Marked and I’m ready to post, I hit CMD+S and save the HTML file to my Create Unique Footnotes folder, head over to my site, and hit CMD+V into a new post.

A Small Rabbit Hole

There are a few specific things I want to point out, just to make sure we’re clear.

The new HTML code is copied to the clipboard by the Python script, not by the Hazel rule. My workflow involves pasting HTML code into Squarespace, not uploading an HTML file and, since my MultiMarkdown file is the master copy of the post, that’s why I’m deleting the HTML file after running the script.

The HTML file being used as input for the script is a complete HTML document with a header, body tags, the whole shebang. I don’t need those extra bits when I’m pasting the code into Squarespace, so the Python script removes them before placing the code on the clipboard. In fact, the code on the clipboard looks exactly like what you see when you toggle the Switch to HTML Source View button in Marked [7] while previewing your MultiMarkdown document.

Another important note; keen observers will notice the last four lines of the Python script are commented out. That code is there for people who actually want a fully processed HTML document with fixed footnote IDs. Un-commenting-out [8] those four lines will write the fixed code over the original HTML document, while maintaining the header, et cetera. If you plan on using this workflow, you’ll want to remove the step in the Hazel rule that throws away the HTML document. I’d suggest you change that part of the rule to move the new HTML file to the Desktop as a sort of visual confirmation that the file has been updated.

I Believe Some Recognition is in Order

When I initially conceived of this idea I hadn’t the slightest idea how to best go about tackling it. I am not a programmer in any sense of the word. A Twitter conversation with my pal Sid O’Neill revealed that REGEX is the method by which I could find and replace items in the code. I don’t know how to do that, so…

Patterns is a Mac app for creating Regular Expressions that I’ve heard a number of people compliment in the past. It lets you see a live preview of the current pattern matches and, when you’ve got the selection you want, will generate the appropriate code for you; Python in my case. Completely indispensable and only $2.99.

Next, of course I have to thank Stack Overflow from whence I acquired some REGEX knowledge and code. Patterns comes with a great Reference Sheet for REGEX commands, but some of the descriptions still left me befuddled. Stack Overflow also linked me to…

This article by Gabe at Macdrifter, where I got the code that places the HTML on the clipboard. It was exactly what I needed, so I took it. And I would have gotten away with it if it wasn’t for you pesky kids and…nevermind.

In any case, high-fives all around. Go team.

---

1. Some of what I’m describing is obfuscated by the use of Bigfoot for my footnotes, specifically the numeral for each footnote, but the problem remains. Let’s move on.  ↩

2. Like footnotes, for one.  ↩

3. You can learn all about MultiMarkdown Metadata here.  ↩

4. Always invoked via TextExpander.  ↩

5. Obviously, I need to choose a unique key for each post or this whole exercise is pointless.  ↩

6. You do use Hazel, don’t you? Good.  ↩

7. It’s in the upper right corner of your document, next to the fullscreen button.  ↩

8. That’s so not a word, is it?  ↩

Open Last QuickTime File Macro

Yesterday, Dr. Drang had a nice post about creating a keyboard shortcut for the Open Recent menu item in BBEdit using Keyboard Maestro. I purchased Keyboard Maestro a few months ago and have been meaning to find some time to play with it. I’ve also been meaning to find a way to open the most recently viewed video in QuickTime with a global keyboard shortcut. Let’s skip right to the punchline.

Throughout the course of a project, I spend a lot of time referring to the most recent cut of a video alongside notes from the client. Once I know the next shot or note I’m going to address, I close the cut, my notes, and get to it. Since my notes live in nvALT, they disappear and reappear with a quick ⌃Z. Now, no matter what I’m doing, I can quickly summon the video file with a keyboard shortcut as well.

A simple tool that removes way more friction that you’d probably guess. It’s a good thing.

Managing Disk Cache (with a Hammer)

Throughout the course of a given project, I use a handful of applications to complete my work. At the moment, I edit in Avid MediaComposer and Final Cut Pro 7. I do my VFX work in NukeX. I review shots and conform sequences in Hiero. And I do final color correction in AfterEffects with Red Giant Colorista II. There’s an obvious advantage to using the right tool for the task at hand, but there’s a caveat that I almost never remember until it smacks me right in the face: disk space.

I’m not talking about the space required for digital negatives, plates, renders, or project files. No, the piece I always manage to forget is the massive disk cache each application in my workflow creates on my startup disk [1]. With single frames of a sequence ranging between 5 and 30Mb, and individual shot versions reaching well into double digits, disk cache can take up a ton of space. And since cache files are created whenever an element is changed and viewed, it’s not an easy task to estimate how much space will be used by a project ahead of time.

Most applications, including every application I use, have built in preferences designed to limit the size of the disk cache. Most also feature a big button labeled “Clear Disk Cache”. These preferences are great if you spend all your time in a single application, but are of little consolation when you’re halfway through previewing a shot in Nuke and OS X pops up telling you your startup disk is full. Once you’re in that sad, embarrassing moment, good luck getting AfterEffects to open so you can hit that “Clear Cache” button.

“To the Finder,” you say? Even if the Finder were responsive when your startup disk had less than 50Mb of free space, are you the kind of person that keeps a sticky note on your monitor listing the paths to all the buried, hidden cache folders for each application? Neither am I.

Historically, I’ve gone straight to my project’s Renders folder and deleted my 2 oldest exports, giving me about 6Gb of breathing room to go hunt down the various disk hogs. Not a great solution. What would be ideal is the ability to hit one keyboard shortcut, see a list of which applications are taking up space and, more importantly, how much space, then quickly purging the unnecessary files.

View fullsize

Normally, the sizes for AfterEffects, Nuke, and Hiero are in the double digits of Gigabytes. This screenshot was taken after using the script as intended.

The Script

cache.command:

#!/bin/sh  

clear  
echo Current Cache Size:  
echo      
du -c -h -s "/Users/dansturm/Library/Preferences/Adobe/After Effects/11.0/Adobe After Effects Disk Cache - Dan’s MacBook Pro.noindex/" "/var/tmp/nuke-u501/" "/var/tmp/hiero/" "/Avid MediaFiles/MXF/" "/Users/dansturm/Documents/Final Cut Pro Documents"  

echo      
read -p "Purge Cache files?(y/n) " -n 1 -r  
echo      
if [[ $REPLY =~ ^[Yy]$ ]]  
then
    rm -r "/Users/dansturm/Library/Preferences/Adobe/After Effects/11.0/Adobe After Effects Disk Cache - Dan’s MacBook Pro.noindex/"*
    rm -r "/var/tmp/nuke-u501/"*
    rm -r "/var/tmp/hiero/"*
    echo    
    echo Done!
    echo      
fi  

The script displays the space taken up by each application, the path to that cache folder, and a total of space used. It then prompts for a y/n input to delete the cache files.

Since managing Avid and Final Cut Pro media requires a bit more attention and nuance than a dumb hammer like this can provide, their disk usage is listed, but their files are not removed. If you modify the script to delete Avid of FCP media, allow me to preemptively say “I told you so” when your project becomes corrupted.

Details

I run the script with a FastScripts keyboard shortcut. I wanted to shortcut to open a new Terminal window to display the information, so the keybaord shortcut actually calls a second short script called cache.sh which takes care of that part:

#!/bin/sh  

chmod +x /Path/To/cache.command; open /Path/To/cache.command  

The paths for the cache folders are hard coded into the main script and, as I mentioned, not all items listed are deleted. I like to see as much information as possible, but manage the deletion list more carefully.

For me, the main offenders of disk cache consumption are AfterEffects and Nuke which, together, average nearly 200Gb of disk usage. Once I’ve dealt with those 2 applications, I don’t usually need to go hunting for more free space.

Next time I accidentally inevitably fill up my startup disk with cache files, it’ll take seconds to rectify, rather than a half hour of excruciatingly slow, manual effort.

Hooray for automation!

---

1. While it’s true that, in almost every application, you can re-map the disk cache folder to any disk of your choice, the entire point of a disk cache is to have the fastest read/write speeds possible, and no disk I own is faster than my rMBP’s internal SSD.  ↩

Experiments in iSight Scripting

A week or two ago, a conversation with my girlfriend reminded me of this old video from the Defcon 18 Conference, in which a hacker by the name of Zoz recounts the tale of how he located and recovered his recently-stolen computer by way of some fancy Internet skills. In addition to being a great reminder about data safety and security, it’s a pretty damn funny story.

Re-watching the video reminded me I’ve always wanted to experiment with scripting the iSight camera on my MacBook Pro. Not necessarily for the purpose of having photo evidence in the event of a theft, but more as a fun exercise in scripting and automation. Maybe I’m easily entertained (rhetorical), but I found the process and results throughly enjoyable.

Command Line Tools

The first thing I needed was a command line interface to the iSight camera. Despite my best Googling efforts, I wasn’t able to find any native OS tools to fire the camera (I’m still not sure if there is one). But lucky for me, nearly every search I did pointed me to a free, no-longer-supported-but-still-functional CLI tool called iSightCapture. Installation is as simple as tossing it into /usr/local/bin.

Regarding syntax, here’s the pertinent information from the iSightCapture readme file:

isightcapture [-v] [-d] [-n frame-no] [-w width] [-h height] [-t jpg|png|tiff|bmp] output-filename

Options
-v output version information and exit
-d enable debugging messages. Off by default
-n capture nth-frame
-w output file pixel width. Defaults to 640 pixels.
-h output file pixel height. Defaults to 480 pixels.
-t output format - one of jpg, png, tiff or bmp. Defaults to JPEG.

Examples
$ ./isightcapture image.jpg
will output a 640x480 image in JPEG format

$ ./isightcapture -w 320 -h 240 -t png image.png
will output a scaled 320x240 image in PNG format

Triggering

I wanted the ability to remotely trigger the iSight camera from my phone, and have it run for a predefined interval of time. To do this, I’d use a watch folder and a trigger file, uploaded from my phone, to initiate the script. This would theoretically be the method I’d use to “gather photographic evidence” in the event my laptop is stolen.

As usual, to set this up, I turned to my trusty friend Dropbox. To keep things tidy, I needed a couple of folders. The directory structure looks like this:

• iSight_backup
  • scripts
  • trigger

The photos taken by the script will be placed in the top level directory, iSight_backup. The script itself would be placed in the scripts folder, and the trigger folder remains empty, awaiting a text file to be uploaded via Dropbox on my phone.

The Script

Here’s the script, called isbackup_t.bash, and saved into my iSight_backup/scripts/ folder:

#!/bin/bash  

#   path to iSightCapture CLI tool  
APPP="/usr/local/bin/isightcapture"  

#   path to Dropbox folder to receive photos  
FPATH="/Path/To/Dropbox/iSight_backup/" 

#   path to trigger file  
TPATH="/Path/To/Dropbox/iSight_backup/trigger/"  

#   select photo filetype; jpg, png, tiff, or bmp  
XTN=".jpg"  

#   number of photos to take  
PNUM="24"  

#   interval between photos (seconds)  
PINT="300"  

for i in `seq 1 $PNUM`;  
    do
        DTS=$(date -u +"%F--%H-%M-%S")  
        $APPP "$FPATH$DTS$XTN"  
        sleep $PINT  
    done  

rm $TPATH*  

The Details

Each photo gets a date and time stamp added to the file name for easy sorting. Since the FPATH destination is set to a folder within my Dropbox, and each file is a 640x480 jpeg, approximately 25kb in size, I can see the resulting photos on my phone just seconds after their taken.

By setting PNUM to 24, and PINT to 300, the script will take a photo every 5 minutes, for the next 2 hours, then stop until I upload another trigger file. However, since the script itself lives in Dropbox, I can change the interval or duration variables from my iOS text editor at any time.

Tying The Room Together

All that’s left to complete our little project is to create a launchd task to keep an eye on our trigger folder, and fire up the script if anything lands inside. Since creating, modifying, or using launchd tasks via any method other than a GUI is currently beyond me, I turned to Lingon.

Here’s the resulting launchd task:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>isbackup triggered</string>
    <key>ProgramArguments</key>
    <array>
        <string>bash</string>
        <string>/Path/To/Dropbox/iSight_backup/scripts/isbackup_t.bash</string>
    </array>
    <key>QueueDirectories</key>
    <array>
        <string>/Path/To/Dropbox/iSight_backup/trigger</string>
    </array>
</dict>
</plist>

And just like that, we’ve got a setup that will monitor our trigger folder, fire up our script when it sees anything inside, and save each file to our Dropbox.

Extra Credit

During my research for this project, I found a number of people using iSightCapture for things other than a makeshift security tool. Some, for example, used it to take a self portrait at login every day and auto-upload it to an online photo diary.

I thought that was a fun idea, so I wrote a second, shorter version of my script to automatically take a shot every 2 hours without the need for a trigger. For my version, however, I was most definitely not interested in auto-uploading the photos.

The script:

#!/bin/bash

APPP="/usr/local/bin/isightcapture"
FPATH="/Path/To/Dropbox/iSight_backup/"
DTS=$(date -u +"%F--%H-%M-%S")
XTN=".jpg"

$APPP "$FPATH$DTS$XTN"

And the launchd task:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>KeepAlive</key>
    <false/>
    <key>Label</key>
    <string>IS Secuirty</string>
    <key>ProgramArguments</key>
    <array>
        <string>bash</string>
        <string>/Path/To/Dropbox/iSight_backup/scripts/isbackup.bash</string>
    </array>
    <key>QueueDirectories</key>
    <array/>
    <key>RunAtLoad</key>
    <true/>
    <key>StartInterval</key>
    <integer>7200</integer>
</dict>
</plist>

Since I typically use a multiple monitor setup when working, I almost never see the green light come on when the script runs and when I check the photo directory, usually the beginning of the next day, I’m treated to at least a couple hilarious images. Like these, for example:

View fullsize

Hi Merlin!

View fullsize

I may be asleep in this photo. I can't tell.

View fullsize

No, that's not my poster. My girlfriend got to decorate that side of the room.

Yes, It makes Skype calls awkward. Then again, so does that hair.

Cleanup

Despite the photos being almost inconsequentially small in file size, I really don’t need to be keeping a long-running historical record of my disheveled appearance. To keep things from getting unwieldy, I created a Hazel rule that deletes files older than 2 weeks. I see no reason I’d need to keep them any longer than that.

Why?

I don’t know if it’s because this little project was so different from my day to day work, or I thought it would be a genuinely useful security tool, or I’m a narcissist that loves seeing pictures of himself (again, rhetorical), but I had a blast playing around with this little project.

As with nearly everything on this blog, I learned some things, failed a few times, and came out the other side with a fun little script I can demo to people, making it absolutely certain they’ll never ask me anything about computers again.

Running Scripts with TextExpander

I love that the Internet is full of smart people making and sharing awesome things. Like Dr. Drang and his Tidying Markdown Reference Links script. Seth Brown’s formd is another great tool I don’t know how I ever lived without. But, being the amateur that I am, I always struggle to figure out just how to use the scripts these amazing programmers have been kind enough to provide.

Lucky for me (and you), I’m able to play along with the help of everyone’s favorite writing tool, TextExpander. In the documentation for formd, Seth was gracious enough to spell it out for us laypersons[1].

To run formd with TextExpander, simply place the Markdown text to convert onto the clipboard and issue the appropriate TextExpander shortcut (I use fr or fi for referenced or inline, respectively).

It took longer than I’d like to admit, but eventually I realized this snippet could be tweaked to run any script I happen to download from a coder more skilled than myself. Additionally, since most of the scripts I want to use are built for processing text, the copy/paste activation generally works great.

#!/bin/bash

pbpaste | python /Path/To/Your/Script/script.py

With this short snippet, I can copy my text to be processed, invoke the script snippet of my choosing, and have the results immediately pasted back into my text editor, email, or wherever.

This particular snippet assumes you’re running a python script, but you can just as easily swap python for ruby or perl. Or you can omit the python call if you’re just running a standard Bash command. As long as it’s a valid command that would run in the Terminal[2], you can automate it this way. And, just as with formd, to use the snippet, you simply Copy the text to be processed, and invoke the snippet.

Boom. Done. Life is grand.

While none of this is new or revolutionary, connecting the dots between TextExpander and the Terminal is something I wish I’d have discovered long ago and, therefore, may be of interest to you.

Of course none of this would be possible without the amazing minds that write the scripts in the first place. So, along with this post, I offer a sincere Thank You to Dr. Drang and his site And now it’s all this, as well as Seth Brown and his site Dr. Bunsen.

---

1. Layfolk? Laypeople? Missing the point?  ↩

2. And frankly, I just start trying things in the Terminal until I find the correct command.  ↩

Launching Marked from Sublime Text 2

I do pretty much all my MultiMarkdown writing anymore in Sublime Text 2. I’ve come to rely on Marked for previewing my files with a variety of custom CSS files, depending on the type of project I’m writing.

At the moment, the most annoying part of this workflow is the time it takes to open Marked and locate the MultiMarkdown file I’m currently working on in Sublime Text. I think I’ve been spoiled by the speed of using Sublime Text’s Goto Anything feature (Command + P) for opening files.

To speed things along, I wrote a new build system for Sublime Text that launches the active document in Marked. Now previewing my active file is as easy as invoking the build command which, for me, is still Command + B.

The (insanely simple) code [1]:

{  
"shell": "true",  
"cmd": ["open -a marked \"$file\""]  
}  

More non-rocket-science, but a big time saver in my world.

---

1. This build system is only for OS X.  ↩

Automated FTP upload with Hazel via Bash

A couple weeks ago Macdrifter had a nice post about automating FTP uploads with Hazel, Dropbox, and Python. It’s a similar idea to the setup I’ve been using to automate my MultiMarkdown workflow, but the main reason it grabbed my interest was this line:

I really like Transmit for FTP, but it seemed a little heavy handed for Hazel automation.

He’s right. Using Transmit for the FTP portion of my process was a poor decision. It just happened to be the only way I knew how, and scripting the FTP upload didn’t even occur to me at the time (there’s a reason for the name of this site).

After reading his post, I decided to swap out the slow and clunky Transmit portion of my Hazel rule with some fancy code. One problem. I don’t know anything about Python[1], and the 3 hours I spent trying to get up to speed were futile and fruitless.

Since I’ve dabbled a bit with Bash, I decided to see if there was an equivalent way to accomplish the same tasks. After a bit of searching, and a lot of trial and error (again, read the name of the site), I came up with this:

HOST=ftp.DOMAIN.com  
USER=myUserName  
PASS=myPassword  

JNAME= \ basename $1\  

ftp -inv $HOST << EOF  

user $USER $PASS  

put "$1" "$JNAME"  

EOF  

echo "http://dansturm.com/$JNAME" >> /Users/PATH/Dropbox/PATH/UPLOADS_LOG_FILE.txt

It’s faster than the Transmit method. It works more often than the Transmit method. And it even records the URL of the uploaded file to a text file in my Dropbox (my favorite idea from the Macdrifter post). I also have the Hazel rule change the file name to all lowercase and swap spaces for underscores, something is already performed in my Text to HTML conversion rule, but now the FTP rule can be used stand-alone as well as in conjunction with the MultiMarkdown process.

It has no error reporting, and you can’t even really tell it’s running (save for the evidence in the log file), but it’s way better than what I was using. Many thanks to Macdrifter for this one.

But…

I’m happy I was able to improve my exiting tools, and learn a few things in the process, but now that I’ve migrated this blog to Squarespace, I’m not sure how much I’ll actually use them considering Squarespace doesn’t support FTP access, and my home site doesn’t need updating very often.

---

1. Okay, I know some Python, but only enough to customize the interface in Nuke and build some basic gizmos and comp tools.  ↩

SP-MMD Cheaters

Since Brett Terpstra gave us Cheaters, I’ve been populating the app with the various tools I use frequently.

I’ve finally gotten around to creating a cheat sheet for my MultiMarkdown Screenplay syntax. I expect it to be used by precisely one person. Me.

However I’m posting it here to serve as a shorter explanation of the way I write, to spare you from reading the whole back story. It uses the same modified CSS file I created for the Fountain cheet sheet.

Here’s the SP-MMD Cheaters page.

And here’s the cheat sheet in a web-friendly view for curious passers-by.

My Hazel CMS

It should be no surprise to anyone familiar with the app, that Noodlesoft’s Hazel is amazing. Today I setup an automated system using Dropbox, Automator, and Hazel to process MultiMarkdown documents into HTML, give them web-ready filenames, and upload them to my website.

Everything on this site starts as a MultiMarkdown text file. I preview the page in Marked, and when its ready to go live, I save a copy of the file in a folder called _1_ready_, which has 3 Hazel rules applied.

Ready

First things first, the text file needs to be converted to an HTML document. I’ve used the Run Shell Script action to call the mmd command.

Pretty straight forward. To add some flexibility, the second Hazel rule for the _1_ready_ folder checks for changes to preexisting text files, and processes them using the same bash script. That saves me from having to delete and re-copy a file to make an update.

The last rule for the _1_ready_ folder renames the HTML file, making it entirely lowercase, replaces the spaces with underscores, and moves the new file to a folder called _2_go_.

It’s important to make sure the name element in the with patern: section is set to lowercase, and the replace text dialog is used to swap the spaces for underscores.

Go

The _2_go_ folder automatically uploads any file it finds to the root of my site. As with _1_ready_, the final files are left in the folder to more easily make changes later. Additionally, I keep a copy of my Blog index page and rss XML in the _2_go_ folder so I can quickly update the main page with links to the new posts.

To upload the files, the Hazel rule calls an Automator Workflow that uses the Transmit [1] upload action to log into my site (stored as a favorite), and drop everything in the root folder, overwriting files if necessary.

Dropbox

The best part about this whole workflow is Dropbox. Both the _1_ready_ and _2_go_ folders are in my Dropbox, giving me the ability to drop in files from my iPhone, iPad, etc. With apps like TextExpander and Nebulous Notes, there’s no reason I can’t create, and post entirely from an iOS device. Obviously I’ll need a Mac, running and online, but the flexibility of this workflow is well worth the cost of a dedicated system.

Needless to say, I’m incredibly excited to have this new capability, and I can’t wait to see what other workflow magic I can create with Hazel

---

1. Many thanks to Macdrifter for recommending Transmit. I love this app.  ↩

Update: I've since revised my upload method to use a Bash script, rather than Transmit. It's much faster and more efficient, so if this idea interests you, you should definitely check it out.