Jekyll2020-04-21T18:36:13-04:00/feed.xmlOpen the Black BoxPaul Gazzilloblog@pgazz.comPlaying and Creating MIDI Songs in Debian2020-04-21T02:23:43-04:002020-04-21T02:23:43-04:00/2020/04/21/playing-and-creating%20midi-songs-in-debian<h2 id="handling-midi-in-debian">Handling midi in Debian</h2>
<p>Playing midi can be done with <code class="highlighter-rouge">timidity</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt install timidity # install it
timidity twelvetone1.mid # play a midi file
</code></pre></div></div>
<p>Select between various midi synthesizers cards with <code class="highlighter-rouge">--module=n</code> (see <code class="highlighter-rouge">man timidity</code> for more), e.g.,</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>timidity --module=32 twelvetone1.mid # emulate a sound blaster live
</code></pre></div></div>
<p>Convert to a wav file and then an mp3 (with ffpmeg)<sup id="fnref:stream"><a href="#fn:stream" class="footnote">1</a></sup>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>timidity -Ow -o twelvetone1.wav twelvetone1.mid
ffmpeg -i twelvetone1.wav twelvetone1.mp3
rm twelvetone1.wav # delete the intermediate file
</code></pre></div></div>
<h2 id="noteworthy-composer-in-wine">Noteworthy Composer in Wine</h2>
<p>Noteworthy Composer is a classic Windows program for writing in musical notation. Noteworthy will play these compositions in midi according to instruments selected for each staff. Download the <a href="https://noteworthycomposer.com/nwc2/viewer.htm">free viewer</a> or buy the <a href="https://noteworthycomposer.com/order/buy/?formid=nwc2">paid version</a>.</p>
<p>Noteworthy can be run with the <a href="https://www.winehq.org/">Wine windows emulator</a> along with timidity providing midi output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># debian needs i386 packages for wine32
sudo dpkg --add-architecture i386
sudo apt update
sudo apt install wine32
# install wine
sudo apt install wine wine32 winetricks cabextract
# install midi package and alsa for providing sound output
apt-get install timidity
apt-get install alsa-oss
# create a separate prefix for noteworthy if you like
export WINEPREFIX=~/.nwc
# prepare the wine config
WINEARCH=win32 winecfg
# free viewer
wine setup_nwc275a_viewer.exe
# paid version
wine setup_nwc275a.exe
</code></pre></div></div>
<p>Wine should provide a desktop shortcut accessible from the application search in Gnome (you may need to check the box for the quick launcher near the end of installation). Otherwise, run the program with the following:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>timidity -iA -Os # setup the midi output
# free viewer
WINEPREFIX=~/.nwc wine ~/.wine/drive_c/Program\ Files/Noteworthy\ Software/NoteWorthy\ Composer\ 2\ Viewer/NWC2View.exe
# paid version
WINEPREFIX=~/.nwc wine ~/.wine/drive_c/Program\ Files/Noteworthy\ Software/NoteWorthy\ Composer\ 2/NWC2.exe
</code></pre></div></div>
<p>Select the midi output in Tools > Options > MIDI as discussed <a href="https://appdb.winehq.org/objectManager.php?sClass=version&iId=14355&iTestingId=39970">here</a>.</p>
<div class="footnotes">
<ol>
<li id="fn:stream">
<p>I’m sure there’s a more clever way to do this where timidity outputs a stream and ffmpeg reads the stream, avoiding the intermediate file. <a href="#fnref:stream" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Paul Gazzilloblog@pgazz.comHandling midi in DebianComputing Days Between Dates with bash2019-05-29T15:04:00-04:002019-05-29T15:04:00-04:00/2019/05/29/computing-days-between-dates-with-bash<p>This script will compute the days between dates.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># adapted from https://www.linuxforfreshers.com/2019/01/how-to-count-days-since-specific-date.html
if [ $# -ne 2 ]; then
echo "USAGE: $(basename $0) start_date end_date";
exit 1;
else
start=$1;
end=$2;
echo "$((($(date +%s --date "$end") - $(date +%s --date "$start"))/(3600*24))) days between $start and $end";
fi
</code></pre></div></div>Paul Gazzilloblog@pgazz.comThis script will compute the days between dates.VNC via SSH Tunnel2019-01-20T09:25:00-05:002019-01-20T09:25:00-05:00/2019/01/20/vnc-via-ssh-tunnel<p>While I typically use a remote shell, occasionally I’ll need use a GUI remotely. On the remote machine, create the remote server:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vncserver :1 -geometry 1920x1080 -depth 16 -localhost
</code></pre></div></div>
<p>This creates a new VNC server for X11 display <code class="highlighter-rouge">:1</code> (<code class="highlighter-rouge">:0</code> is the one created on boot, if not a headless machine). <code class="highlighter-rouge">-localhost</code> ensures that the server does not accept remote connections. Instead, I use SSH tunneling to provide an encrypted channel to the server:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh -NL localhost:5901:localhost:5901 SERVERNAME &
</code></pre></div></div>
<p>Using <code class="highlighter-rouge">-L</code>, we bind local port 5901 to port 5901 on the remote server, which is the default VNC port for display <code class="highlighter-rouge">:1</code>.</p>
<p>Tear down the server when finished:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vncserver -kill :1
</code></pre></div></div>
<h3 id="using-an-existing-desktop-session">Using an Existing Desktop Session</h3>
<p>To mirror an existing X11 session instead, use <code class="highlighter-rouge">x11vnc</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh -L 5900:localhost:5900 SERVERNAME 'x11vnc -usepw -localhost -forever -display :0;'
</code></pre></div></div>Paul Gazzilloblog@pgazz.comWhile I typically use a remote shell, occasionally I’ll need use a GUI remotely. On the remote machine, create the remote server:PDF from Markdown2019-01-18T10:45:00-05:002019-01-18T10:45:00-05:00/2019/01/18/pdf-from-markdown<p>I like to use markdown for my course <a href="https://github.com/cop3402spring19/syllabus">syllabus</a>, since it’s quick and easy, and allows easy use of git to make and track updates.</p>
<p>My department requires a PDF version of the syllabus, so I discovered this handy way to convert from github-flavored markdown (at least tables are supported):</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pandoc -t latex -o README.pdf README.md
</code></pre></div></div>
<p>It first converts the markdown to latex, then produces the PDF.</p>
<p>(Update 2020-04-05) This will change the font (to something in <code class="highlighter-rouge">/usr/local/texlive/*/texmf-dist/source/fonts/</code>) and set the margins to be smaller.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pandoc -o README.pdf README.md --variable=fontfamily:arev -V geometry:margin=1in
</code></pre></div></div>Paul Gazzilloblog@pgazz.comI like to use markdown for my course syllabus, since it’s quick and easy, and allows easy use of git to make and track updates.Notes on the C Preprocessor: Token Pasting2018-02-26T10:43:00-05:002018-02-26T10:43:00-05:00/2018/02/26/notes-on-the-c-preprocessor-token-pasting<p>A feature of the preprocessor little known outside of hardcore C programmers is <em>token pasting</em> (also called <em>concatenation</em>). As the name suggests, token pasting lets the programmer take any two tokens in the source code and turn them into a brand new token. Developers can and do use token pasting in all sorts of clever ways. The <a href="https://gcc.gnu.org/onlinedocs/gcc-7.3.0/cpp/Concatenation.html#Concatenation">manual</a> for the preprocessor provides an example of using token pasting to generate code from boilerplate:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: #define COMMAND(NAME) { #NAME, NAME ## _command }
2: struct command commands[] =
3: {
4: COMMAND (quit), // expands to { "quit", quit_command }
5: COMMAND (help), // expands to { "help", help_command }
6: …
7: };
</code></pre></div></div>
<p>The <code class="highlighter-rouge">COMMAND</code> macro uses the token paste operator <code class="highlighter-rouge">##</code> to construct identifiers with the <code class="highlighter-rouge">_command</code> suffix, helping to eliminate repetitive code. This pattern is used to generate all sorts of <a href="https://elixir.bootlin.com/linux/v4.15/source/arch/x86/entry/syscall_64.c#L17">things</a>, even entire <a href="https://elixir.bootlin.com/linux/v4.15/source/block/mq-deadline.c#L521">function</a> <a href="https://elixir.bootlin.com/linux/v4.15/source/sound/aoa/fabrics/layout.c#L641">definitions</a>.</p>
<p>Token pasting can be a serious pain for program analyses, code browsers, and really any tool that depends on the syntax tree of a program, because it alters the stream of tokens that the parser sees. A tool can opt to resolve token pasting first, but this requires evaluating at least some macros, resulting in code that might be dramatically different from the source code the developer is writing. Even worse, the resulting tokens can depend on configuration options when used with <code class="highlighter-rouge">#ifdef</code>s. Let’s look at a few examples, and see why this is so hard.</p>
<p>Developers have found very creative uses for token pasting. This [example]((https://elixir.bootlin.com/linux/v2.6.33/source/include/linux/compiler-gcc.h), from a past version of the Linux kernel source, includes a header file whose name depends on the version of GCC being used to compile the source code:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: #define __gcc_header(x) #x
2: #define _gcc_header(x) __gcc_header(linux/compiler-gcc##x.h)
3: #define gcc_header(x) _gcc_header(x)
4: #include gcc_header(__GNUC__)
</code></pre></div></div>
<p>Line 4 is the actual include, which calls a macro to generate a file name from <code class="highlighter-rouge">__GNUC__</code>, a built-in macro that returns the hard-coded version number of the compiler. Line 3 defines the header file generation macros, using an extra macro definition with a single preceding underscore (line 2). The extra macro definition is necessary to invoke token pasting on the value of the parameter x (rather than the symbol x itself). The macro on line 1 then <code class="highlighter-rouge">stringifies</code> (with operator <code class="highlighter-rouge">#</code>) the resulting header file name, turning C tokens into a C string, yet another feature of the preprocessor.</p>
<p>If this example wasn’t confusing enough, token pasting can even muck with types, since any C token can be constructed by pasting, including type names. The following [example]((https://elixir.bootlin.com/linux/v2.6.33/source/fs/udf/balloc.c#L41), also from the Linux kernel, constructs different type names depending on whether we want to build for a 32-bit or 64-bit architecture.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1: #define uintBPL_t uint(BITS_PER_LONG)
2: #define uint(x) xuint(x)
3: #define xuint(x) __le ## x
4: uintBPL_t *p = ... ;
</code></pre></div></div>
<p><code class="highlighter-rouge">BITS_PER_LONG</code> is either <code class="highlighter-rouge">64</code> or <code class="highlighter-rouge">32</code> depending on another configuration option. Trying to reason about this piece of code is tricky, particularly for an automated tool that, for instance, looks for bugs or does refactoring.</p>Paul Gazzilloblog@pgazz.comA feature of the preprocessor little known outside of hardcore C programmers is token pasting (also called concatenation). As the name suggests, token pasting lets the programmer take any two tokens in the source code and turn them into a brand new token. Developers can and do use token pasting in all sorts of clever ways. The manual for the preprocessor provides an example of using token pasting to generate code from boilerplate:Notes on the C Preprocessor: Problems with Parentheses2017-08-30T22:42:00-04:002017-08-30T22:42:00-04:00/2017/08/30/notes-on-the-c-preprocessor-problems-with-parentheses<p>Use of preprocessor macros is full of pitfalls.
In fact, its user manual includes a section called “<a href="https://gcc.gnu.org/onlinedocs/gcc-6.4.0/cpp/Macro-Pitfalls.html#Macro-Pitfalls">Macro Pitfalls</a>”.
One of these pitfalls revolves around <a href="https://gcc.gnu.org/onlinedocs/gcc-6.4.0/cpp/Operator-Precedence-Problems.html#Operator-Precedence-Problems">operator precedence</a>.
Operator precedence problems happen when the expectations about pass-by-value, the default for C functions, are applied to function-like macros.
Function-like macros are pass-by-name; the arguments are textually substituted in the macro body.
This is especially confusing because both C and preprocessor functions have the same invocation syntax.</p>
<p>Here is an example of what can go wrong:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #define square(a) a * a
int z = square(x + y) * 2
</code></pre></div></div>
<p>This square root method looks perfectly sensible and would be if <code class="highlighter-rouge">square</code> were a typical C function.
Let’s see what happens when we preprocess this example.
We get the following:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>int z = x + y * x + y * 2
</code></pre></div></div>
<p>Notice this will not compute the square of <code class="highlighter-rouge">x + y</code>.
The preprocessor could care less about operator precedence or any other aspect of the C grammar.
It’s just a text-replacement tool.
What we want this macro to give us is the following:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>int z = ((x + y) * (x + y)) * 2
</code></pre></div></div>
<p>So the manual advises macro writers to always wrap argument uses in parentheses to avoid this misnesting.
Here is the updated example that gives the behavior we want by wrapping the argument <code class="highlighter-rouge">a</code>, as well as the entire macro definition, in parentheses:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #define square(a) ((a) * (a))
int z = square(x + y) * 2
</code></pre></div></div>
<p>A more subtle consequence of function-like macro behavior is when they piece together other function-like macros.
Take this example from the preprocessor user manual:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> #define twice(x) (2*(x))
#define call_with_1(x) x(1)
call_with_1 (twice)
==> twice(1)
==> (2*(1))
</code></pre></div></div>
<p>The <code class="highlighter-rouge">call_with_1</code> macro takes any function-like macro and passes <code class="highlighter-rouge">1</code> to it.
While this is contrived example, such chicanery does happen in practice<sup id="fnref:contrived"><a href="#fn:contrived" class="footnote">1</a></sup>.</p>
<p>One utterly confusing consequence of text replacement is the possibility of <a href="https://gcc.gnu.org/onlinedocs/gcc-6.4.0/cpp/Misnesting.html#Misnesting">misnesting</a>, where macro definitions may contain unbalanced parentheses.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#define strange(file) fprintf (file, "%s %d",
strange(stderr) p, 35)
==> fprintf (stderr, "%s %d", p, 35)
</code></pre></div></div>
<p>The macro <code class="highlighter-rouge">strange</code> contains the open parenthesis, which is balanced by the close parenthesis on line 2 once the macro is expanded.
Writing unbalanced parentheses in your code may cause others to think <em>you’re</em> unbalanced.</p>
<p>The bottom-line is that function-like macros are not really functions.
The arguments are just textually replaced.
Argument values are not computed before substitution.
They are instead <a href="https://gcc.gnu.org/onlinedocs/gcc-6.3.0/cpp/Argument-Prescan.html#Argument-Prescan">prescanned</a>, which is a whole other can of worms and a topic for a later installment.</p>
<div class="footnotes">
<ol>
<li id="fn:contrived">
<p>See Figure 3 of the paper on <a href="https://www.paulgazzillo.com/papers/pldi12.pdf">SuperC</a>. <a href="#fnref:contrived" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Paul Gazzilloblog@pgazz.comUse of preprocessor macros is full of pitfalls. In fact, its user manual includes a section called “Macro Pitfalls”. One of these pitfalls revolves around operator precedence. Operator precedence problems happen when the expectations about pass-by-value, the default for C functions, are applied to function-like macros. Function-like macros are pass-by-name; the arguments are textually substituted in the macro body. This is especially confusing because both C and preprocessor functions have the same invocation syntax.Notes on the C Preprocessor: Introduction2017-08-03T17:00:00-04:002017-08-03T17:00:00-04:00/2017/08/03/notes-on-the-c-preprocessor-introduction<p>My graduate work on <a href="http://paulgazzillo.com/papers/pldi12.pdf">SuperC</a> made made me way too familiar with the C preprocessor’s ins and outs, more than I ever could have imagined (or wanted).
SuperC’s novel preprocessing and parsing algorithms let you parse a program without having to run the preprocessor first.
Solving this challenge exposed me to interesting quirks of the preprocessor and strange usage patterns that appear in the wild.
I’d like to share these and bring attention to this underrated aspect of compilers, hopefully providing insight for future language development and software tools.</p>
<p>Lurking between the lexer and parser, it can be hard to distinguish the preprocessor from the C language itself.
For instance <code class="highlighter-rouge">#include</code> is not part of the C language, but a preprocessing feature that basically just copies in a given file before compilation.
This and the rest of the preprocessor constructs, macros (<code class="highlighter-rouge">#define</code>) and conditional compilation (<code class="highlighter-rouge">#ifdef</code>), are completely distinct from the C language, sharing only its lexical specification.
This makes for a powerful tool that is used to augment the diminutive C language, even enabling what resemble generics, iterators, modules, and more.</p>
<p>The preprocessor is not without its flaws. For one, it makes program analysis harder.
Software tools for C are easily tripped up by unpreprocessed code.
Yet the preprocessor is used a surprising amount. <a href="https://homes.cs.washington.edu/~mernst/pubs/c-preprocessor-tse2002.pdf">Ernst et al.</a>, looking at a variety of software packages, found that preprocessor directives make up over eight percent of source lines and macros are used on 25% of source lines on average<sup id="fnref:ernst"><a href="#fn:ernst" class="footnote">1</a></sup>.
One easy way for tools to avoid preprocessor directives and macros is to preprocessor first and work on pure C instead. Preprocessing first, however, loses much of the original program.
Library headers are no longer separate, macros are expanded and lose their original meaning, and conditional compilation eliminates whole chunks of the source code.
A preprocessed C file might be unrecognizable to the original developer.</p>
<p>Stroustrup recognized the problems with the preprocessor and designed C++ to minimize the need for it<sup id="fnref:stroustrup"><a href="#fn:stroustrup" class="footnote">2</a></sup>:</p>
<blockquote>
<p>Macros are very important in C but have far fewer uses in C++. The first rule about macros is: don’t use them unless you have to. Almost every macro demonstrates a flaw in the programming language, in the program, or in the programmer. Because they rearrange the program text before the compiler proper sees it, macros are also a major problem for many programming support tools. So when you use macros, you should expect inferior service from tools such as debuggers, cross-reference tools, and profilers.</p>
</blockquote>
<p>The preprocessor should probably be replaced, but doing so is not easy.
For one, there is an enormous amount of existing systems with sprawling C codebases that make extensive use of the preprocessor.
A more subtle challenge is that it’s hard to match the utility of the preprocessor.
While some usages are already subsumed by newer C programming constructs, <code class="highlighter-rouge">const</code> instead of certain macros, others are not trivial to replace, such as poor-man’s generics, without substantial additions to the programming language itself.
Still, there are plenty of things you can do with the preprocessor that are basically absurd and don’t need to be supported in a preprocessor replacement.
Take the following program that uses a macro to redefine the right curly brace:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#include <stdio.h>
#define CURLY }
int main(char **argv, int argc) {
printf("hello, world!\n");
CURLY
</code></pre></div></div>
<p>This code is completely legal and will compile and run, but would anyone ever need to do this? Probably not.</p>
<p>The trick is to give developers those preprocessor features they want and need, while leaving out those that are not useful or make language tools choke.
To replace it entirely likely requires a combination of efforts across language design, tool building, software engineering, and development practices.
This is a testament to the sheer usefulness of the preprocessor.</p>
<p>In this series of posts, we will dig into the preprocessor, how it’s used, and what makes it so hard to analyze.
To get started, the first post in this series will cover a classic preprocessor pitfall and illustrate some preprocessor internals.
From there we’ll look at dynamic includes, highly-configured arrays and structs, free macros in includes, poor-man’s iterators and generics, and more.</p>
<div class="footnotes">
<ol>
<li id="fn:ernst">
<p>Michael D. Ernst, Greg J. Badros, and David Notkin. An Empirical Analysis of C Preprocessor Use. Transactions on Software Engineering, 2002. <a href="#fnref:ernst" class="reversefootnote">↩</a></p>
</li>
<li id="fn:stroustrup">
<p>Bjarne Stroustrup. The C++ Programming Language, Fourth Edition, 2013. Section 12.6 Macros. <a href="#fnref:stroustrup" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Paul Gazzilloblog@pgazz.comMy graduate work on SuperC made made me way too familiar with the C preprocessor’s ins and outs, more than I ever could have imagined (or wanted). SuperC’s novel preprocessing and parsing algorithms let you parse a program without having to run the preprocessor first. Solving this challenge exposed me to interesting quirks of the preprocessor and strange usage patterns that appear in the wild. I’d like to share these and bring attention to this underrated aspect of compilers, hopefully providing insight for future language development and software tools.